On Jan 23, 2014, at 5:55 AM, Xiao, Jianjun (IKET) <[email protected]> wrote:

> Dear Barry,
> 
> There is actually no limitation for me to use the cluster. 
> 
> Somehow, the version Openmpi on our Linux cluster is quite old 1.4.3. Then I 
> updated the Openmpi to 1.6.5.  Now the code works fine without any error. 

   Good

> 
> Ae Jed mentioned, "PETSc does not depend on any specific versions (or 
> implementations) of MPI. 
> http://lists.mcs.anl.gov/pipermail/petsc-users/2011-January/007635.html " 
> 
> I was wondering if you think that the MPI version could be the reason of my 
> errors?

   If there are errors in the MPI implementation 
> 
> Thanks.
> 
> Best regards
> JJ 
> ________________________________________
> From: Barry Smith [[email protected]]
> Sent: Thursday, January 23, 2014 1:28 AM
> To: Xiao, Jianjun (IKET)
> Cc: [email protected]; [email protected]
> Subject: Re: [petsc-users] Error in creation of MPISBAIJ matrix with 
> DMCreateMatrix
> 
> Caught signal number 15 Terminate: Some process (or the batch system) has 
> told this process to end
> 
>   This is a very different problem.  Something on your cluster is deciding to 
> kill your job. Perhaps you are not suppose to be running there? The matrix is 
> very small so it is not likely due to using too much memory. You need to talk 
> directly with the system administrator for that systems and show him/her the 
> message above. It is not a bug in PETSc.
> 
> 
>  Barry
> 
> I ran with 50-60 processes under valgrind and it ran fine for all cases.
> 
> 
> On Jan 22, 2014, at 5:04 PM, Xiao, Jianjun (IKET) <[email protected]> 
> wrote:
> 
>> I changed the code as you suggested, and I updated the nightly tarball.
>> 
>> Unfortunately, I still got the error below with number of processors, say 
>> 22, 55, 61, 62 ... on the Linux cluster.  And I think the number might be 
>> different on your machine. Could you please try other numbers, say 50~63?
>> 
>> It seems the errors could be from my side. Could you please give me some 
>> idea on how to debug such a problem?
>> 
>> Thank you.
>> 
>> [60]PETSC ERROR: 
>> ------------------------------------------------------------------------
>> [60]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the 
>> batch system) has told this process to end
>> [60]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [60]PETSC ERROR: or see 
>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[60]PETSC ERROR: 
>> or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory 
>> corruption errors
>> [60]PETSC ERROR: likely location of problem given in stack below
>> [60]PETSC ERROR: ---------------------  Stack Frames 
>> ------------------------------------
>> [60]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>> [60]PETSC ERROR:       INSTEAD the line number of the start of the function
>> [60]PETSC ERROR:       is given.
>> [60]PETSC ERROR: [60] MatAssemblyBegin_MPISBAIJ line 483 
>> /home/xiao/Local/petsc-dev-debug/src/mat/impls/sbaij/mpi/mpisbaij.c
>> [60]PETSC ERROR: [60] MatAssemblyBegin line 4854 
>> /home/xiao/Local/petsc-dev-debug/src/mat/interface/matrix.c
>> [60]PETSC ERROR: [60] DMCreateMatrix_DA_3d_MPISBAIJ line 1679 
>> /home/xiao/Local/petsc-dev-debug/src/dm/impls/da/fdda.c
>> [60]PETSC ERROR: [60] DMCreateMatrix_DA line 625 
>> /home/xiao/Local/petsc-dev-debug/src/dm/impls/da/fdda.c
>> [60]PETSC ERROR: [60] DMCreateMatrix line 961 
>> /home/xiao/Local/petsc-dev-debug/src/dm/interface/dm.c
>> [60]PETSC ERROR: --------------------- Error Message 
>> ------------------------------------
>> [60]PETSC ERROR: Signal received!
>> [60]PETSC ERROR: 
>> ------------------------------------------------------------------------
>> [60]PETSC ERROR: Petsc Development GIT revision: v3.4.3-2332-g54f71ec  GIT 
>> Date: 2014-01-20 14:12:11 -0700
>> [60]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [60]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [60]PETSC ERROR: See docs/index.html for manual pages.
>> [60]PETSC ERROR: 
>> ------------------------------------------------------------------------
>> [60]PETSC ERROR: ./ex44f on a linux-gnu named cluster08 by xiao Wed Jan 22 
>> 23:51:36 2014
>> [60]PETSC ERROR: Libraries linked from 
>> /home/xiao/Local/petsc-dev-debug/linux-gnu/lib
>> [60]PETSC ERROR: Configure run at Wed Jan 22 23:45:50 2014
>> [60]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 
>> --with-cxx=mpicxx --with-debugging=1
>> [60]PETSC ERROR: 
>> ------------------------------------------------------------------------
>> 
>> JJ
>> _________
>> _______________________________
>> From: Barry Smith [[email protected]]
>> Sent: Wednesday, January 22, 2014 9:29 PM
>> To: Xiao, Jianjun (IKET)
>> Cc: [email protected]; [email protected]
>> Subject: Re: [petsc-users] Error in creation of MPISBAIJ matrix with 
>> DMCreateMatrix
>> 
>> There was an error in your code; because you did not use implicit none in 
>> the main program it did not tell you
>>     CALL DMDACreate3d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,               &
>>                                                          1
>> Error: Symbol 'dmda_boundary_none' at (1) has no IMPLICIT type
>> ex1f.F90:14.26:
>> 
>>    &    DMDA_STENCIL_BOX,-1002,-3,-3,PETSC_DECIDE,PETSC_DECIDE,&
>>                         1
>> Error: Symbol 'dmda_stencil_box' at (1) has no IMPLICIT type
>> 
>> 
>>  I fixed the code by adding use petscdmda and the implicit none and it ran 
>> without error on those number of processes on both a MacOS and a Linux system
>> 
>>    program main   !   Solves the linear system  J x = f
>>     use petscksp; use petscdm; use petscdmda
>>     implicit none
>> #include <finclude/petscdef.h>
>> 
>>  Please let me know if you still have a problem.
>> 
>>  Barry
>> 
>> 
>> On Jan 22, 2014, at 5:15 AM, Xiao, Jianjun (IKET) <[email protected]> 
>> wrote:
>> 
>>> Dear Barry,
>>> 
>>> I modified ex44f.F90, and ran the case on my 64-processors cluster. Please 
>>> find the file in the attachment.
>>> 
>>> I tried various number of processors. It seems that for number of 
>>> processors: 1, 2, 3, 4, 5, 6, 7, 8, 16, 32 and 64 it works fine. For 
>>> numbers such as 20, 30, 31, 60, 61 and 62, I got the error below. This 
>>> time, even for MATMPIBAIJ I got error. I did not try all the numbers 
>>> between 1-64. If you need more information, please let me know.
>>> 
>>> Thank you for your help.
>>> 
>>> Best regards
>>> JJ
>>> 
>>> 
>>> For MATMPISBAIJ, I got the error like this:
>>> 
>>> [55]PETSC ERROR: [55] MatAssemblyBegin_MPISBAIJ line 483 
>>> src/mat/impls/sbaij/mpi/mpisbaij.c
>>> [55]PETSC ERROR: [55] MatAssemblyBegin line 4865 src/mat/interface/matrix.c
>>> [55]PETSC ERROR: [55] DMCreateMatrix_DA_3d_MPISBAIJ line 1694 
>>> src/dm/impls/da/fdda.c
>>> [55]PETSC ERROR: [55] DMCreateMatrix_DA line 626 src/dm/impls/da/fdda.c
>>> [55]PETSC ERROR: [55] DMCreateMatrix line 1002 src/dm/interface/dm.c
>>> [55]PETSC ERROR: --------------------- Error Message 
>>> ------------------------------------
>>> [55]PETSC ERROR: Signal received!
>>> [55]PETSC ERROR: 
>>> ------------------------------------------------------------------------
>>> [55]PETSC ERROR: Petsc Development GIT revision: 
>>> f7404d5510646a3c64be49fff6ce547efef07b3d  GIT Date: 2013-11-27 00:12:54 
>>> +0100
>>> [55]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [55]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [55]PETSC ERROR: See docs/index.html for manual pages.
>>> [55]PETSC ERROR: 
>>> ------------------------------------------------------------------------
>>> [55]PETSC ERROR: ./ex44f on a linux-gnu named cluster07 by xiao Wed Jan 22 
>>> 11:54:32 2014
>>> [55]PETSC ERROR: Libraries linked from 
>>> /home/xiao/Local/petsc-dev-debug/linux-gnu/lib
>>> [55]PETSC ERROR: Configure run at Tue Jan 14 15:27:57 2014
>>> [55]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 
>>> --with-cxx=mpicxx --with-debugging=1
>>> [55]PETSC ERROR: 
>>> ------------------------------------------------------------------------
>>> [55]PETSC ERROR: User provided function() line 0 in  unknown file
>>> 
>>> 
>>> For MATMPIBAIJ, I got the error like this:
>>> 
>>> [44]PETSC ERROR: --------------------- Error Message 
>>> ------------------------------------
>>> [44]PETSC ERROR: Argument out of range!
>>> [44]PETSC ERROR: Trying to set preallocation for row 6678 less than first 
>>> local row 6714!
>>> [44]PETSC ERROR: 
>>> ------------------------------------------------------------------------
>>> [44]PETSC ERROR: Petsc Development GIT revision: 
>>> f7404d5510646a3c64be49fff6ce547efef07b3d  GIT Date: 2013-11-27 00:12:54 
>>> +0100
>>> [44]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [44]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [44]PETSC ERROR: See docs/index.html for manual pages.
>>> [44]PETSC ERROR: 
>>> ------------------------------------------------------------------------
>>> [44]PETSC ERROR: ./ex44f on a linux-gnu named cluster06 by xiao Wed Jan 22 
>>> 11:52:08 2014
>>> [44]PETSC ERROR: Libraries linked from 
>>> /home/xiao/Local/petsc-dev-debug/linux-gnu/lib
>>> [44]PETSC ERROR: Configure run at Tue Jan 14 15:27:57 2014
>>> [44]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 
>>> --with-cxx=mpicxx --with-debugging=1
>>> [44]PETSC ERROR: 
>>> ------------------------------------------------------------------------
>>> [44]PETSC ERROR: DMCreateMatrix_DA_3d_MPIBAIJ() line 1508 in 
>>> src/dm/impls/da/fdda.c
>>> [44]PETSC ERROR: DMCreateMatrix_DA() line 771 in src/dm/impls/da/fdda.c
>>> [44]PETSC ERROR: DMCreateMatrix() line 1007 in src/dm/interface/dm.c
>>> [44]PETSC ERROR: --------------------- Error Message 
>>> ------------------------------------
>>> [44]PETSC ERROR: Corrupt argument:
>>> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>>> [44]PETSC ERROR: Invalid Pointer to Object: Parameter # 1!
>>> [44]PETSC ERROR: 
>>> ------------------------------------------------------------------------
>>> [44]PETSC ERROR: Petsc Development GIT revision: 
>>> f7404d5510646a3c64be49fff6ce547efef07b3d  GIT Date: 2013-11-27 00:12:54 
>>> +0100
>>> [44]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [44]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [44]PETSC ERROR: See docs/index.html for manual pages.
>>> [44]PETSC ERROR: 
>>> ------------------------------------------------------------------------
>>> [44]PETSC ERROR: ./ex44f on a linux-gnu named cluster06 by xiao Wed Jan 22 
>>> 11:52:08 2014
>>> [44]PETSC ERROR: Libraries linked from 
>>> /home/xiao/Local/petsc-dev-debug/linux-gnu/lib
>>> [44]PETSC ERROR: Configure run at Tue Jan 14 15:27:57 2014
>>> [44]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 
>>> --with-cxx=mpicxx --with-debugging=1
>>> [44]PETSC ERROR: 
>>> ------------------------------------------------------------------------
>>> [44]PETSC ERROR: MatDestroy() line 1029 in src/mat/interface/matrix.c
>>> 
>>> ________________________________________
>>> From: Barry Smith [[email protected]]
>>> Sent: Monday, January 20, 2014 8:37 PM
>>> To: Xiao, Jianjun (IKET)
>>> Cc: [email protected]; [email protected]
>>> Subject: Re: [petsc-users] Error in creation of MPISBAIJ matrix with 
>>> DMCreateMatrix
>>> 
>>> Thanks for reporting the problem. This is our error. Could you please send 
>>> us the code that generates the error so we can reproduce the problem, 
>>> determine the cause and fix it.
>>> 
>>> Barry
>>> 
>>> We need to know the exact values of imax,jmax, kmax etc to reproduce the 
>>> problem.
>>> 
>>> 
>>> 
>>> On Jan 20, 2014, at 9:33 AM, Xiao, Jianjun (IKET) <[email protected]> 
>>> wrote:
>>> 
>>>> Dear developers,
>>>> 
>>>> I am using petsc-dev. I tried to create a MPISBAIJ matrix as shown below, 
>>>> and it seems that the matrix creation is sensitive to the number of 
>>>> processors.
>>>> 
>>>>   CALL DMDACreate3d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,            &
>>>>  &    DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,                        &
>>>>  &    DMDA_STENCIL_BOX,-imax,-jmax,-kmax,PETSC_DECIDE,PETSC_DECIDE,&
>>>>  &    PETSC_DECIDE,1,1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,       &
>>>>  &    PETSC_NULL_INTEGER,da,ierr)
>>>> 
>>>>    CALL DMSetMatType(da,MATMPISBAIJ,ierr)
>>>>    CALL DMCreateMatrix(da,mat,ierr)
>>>> 
>>>> A cluster with 64 processors was used for the testing.
>>>> 
>>>> When the number of procssors are 1,2,3,4,5,6,7,8,16,32 and 64, the code 
>>>> always works quite well.
>>>> 
>>>> For some other numbers, the code works not so stable. Sometimes, the 
>>>> matrix was created successfully. Sometimes, it failed.
>>>> 
>>>> When the number of procssors are 20, 33, 63 or some relatively bigger 
>>>> numbers , the code always got the error below.
>>>> 
>>>> 
>>>> mpirun: [25]PETSC ERROR: --------------------- Error Message 
>>>> ------------------------------------
>>>> mpirun: [25]PETSC ERROR: Argument out of range!
>>>> mpirun: [25]PETSC ERROR: New nonzero at (198,7038) caused a malloc!
>>>> mpirun: [25]PETSC ERROR: 
>>>> ------------------------------------------------------------------------
>>>> mpirun: [25]PETSC ERROR: Petsc Development GIT revision: 
>>>> f7404d5510646a3c64be49fff6ce547efef07b3d GIT Date: 2013-11-27 00:12:54 
>>>> +0100
>>>> mpirun: [25]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>> mpirun: [25]PETSC ERROR: See docs/faq.html for hints about trouble 
>>>> shooting.
>>>> mpirun: [25]PETSC ERROR: See docs/index.html for manual pages.
>>>> mpirun: [25]PETSC ERROR: Configure options --with-cc=mpicc 
>>>> --with-fc=mpif90 --with-cxx=mpicxx --with-debugging=1
>>>> mpirun: [25]PETSC ERROR: 
>>>> ------------------------------------------------------------------------
>>>> mpirun: [25]PETSC ERROR: MatSetValuesBlocked_SeqBAIJ() line 1836 in 
>>>> src/mat/impls/baij/seq/baij.c
>>>> mpirun: [25]PETSC ERROR: MatSetValuesBlocked_MPISBAIJ() line 339 in 
>>>> src/mat/impls/sbaij/mpi/mpisbaij.c
>>>> mpirun: [25]PETSC ERROR: MatSetValuesBlocked() line 1658 in 
>>>> src/mat/interface/matrix.c
>>>> mpirun: [25]PETSC ERROR: DMCreateMatrix_DA_3d_MPISBAIJ() line 1780 in 
>>>> src/dm/impls/da/fdda.c
>>>> mpirun: [25]PETSC ERROR: DMCreateMatrix_DA() line 777 in 
>>>> src/dm/impls/da/fdda.c
>>>> mpirun: [25]PETSC ERROR: DMCreateMatrix() line 1007 in 
>>>> src/dm/interface/dm.c
>>>> 
>>>> Then I changed the matrix format to MATMPIBAIJ. DMCreateMatrix worked fine 
>>>> for any number of processors.
>>>> 
>>>>    CALL DMSetMatType(da,MATMPIBAIJ,ierr)
>>>>    CALL DMCreateMatrix(da,gfmat,ierr)
>>>> 
>>>> Could you please let me know how can I fix this problem? If you need more 
>>>> information, please let me know. Thank you.
>>>> 
>>>> JJ
>>> 
>>> <ex44f.F90>
>> 
>> <ex44f.F90>
> 

Reply via email to