There is a problem with the 3.3 series when reinitializing PETSc. Can you switch to PETSc 3.4 which properly supports calling PetscInitialize() several times.
Barry On Sep 3, 2013, at 1:58 PM, Suraj Prabhakaran <[email protected]> wrote: > Hello, > > Again, this relates to finalizing and initializing petsc in my application. I > basically want to finalize petsc and initialize it again with a different MPI > communicator. A rough sketch of my quite huge application is as follows. > > > main() > { > PetscInitialize() > ... > ... > AOCreate().. > AO operations.. > AODestroy().. > ... > ... > PetscFinalize() > PETSC_COMM_WORLD = A new communicator (which includes more processes) > PetscInitialize() > ... > ... > AOCreate().. > AO operations.. > AODestroy() > ... > ... > PetscFinalize() > } > > So after finalizing and initializing it back, I always get an error with one > of the AO calls. Given below is the complete error I get when running with 8 > procs. Can anyone give me a hint what the problem could be? I was told > earlier, that finalizing and initializing petsc should not be a problem! > > Best, > Suraj > > -------------------------ERROR starts here------------------------------- > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: > or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] PetscStrcmp line 414 > /root/petsc/petsc-3.3-p7/src/sys/utils/str.c > [0]PETSC ERROR: [0] PetscFListAdd line 202 > /root/petsc/petsc-3.3-p7/src/sys/dll/reg.c > [0]PETSC ERROR: [0] AORegister line 99 > /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoreg.c > [0]PETSC ERROR: [0] AORegisterAll line 28 > /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoregall.c > [0]PETSC ERROR: [0] AOInitializePackage line 49 > /root/petsc/petsc-3.3-p7/src/dm/ao/interface/dlregisdm.c > [0]PETSC ERROR: [0] AOCreate line 530 > /root/petsc/petsc-3.3-p7/src/dm/ao/interface/ao.c > [0]PETSC ERROR: [0] AOCreateBasicIS line 381 > /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c > [0]PETSC ERROR: [0] AOCreateBasic line 336 > /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT > 2013 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep > 3 20:56:21 2013 > [0]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > [0]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 > --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 > --with-debugging > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown > file > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 59. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the > batch system) has told this process to end > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [1]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: > or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [1]PETSC ERROR: likely location of problem given in stack below > [1]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [1]PETSC ERROR: INSTEAD the line number of the start of the function > [1]PETSC ERROR: is given. > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Signal received! > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT > 2013 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep > 3 20:56:21 2013 > [1]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > [1]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > [1]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 > --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 > --with-debugging > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown > file > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the > batch system) has told this process to end > [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [2]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSC ERROR: > or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [2]PETSC ERROR: likely location of problem given in stack below > [2]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [2]PETSC ERROR: INSTEAD the line number of the start of the function > [2]PETSC ERROR: is given. > [2]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [2]PETSC ERROR: Signal received! > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT > 2013 > [2]PETSC ERROR: See docs/changes/index.html for recent updates. > [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [2]PETSC ERROR: See docs/index.html for manual pages. > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep > 3 20:56:21 2013 > [2]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > [2]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > [2]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 > --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 > --with-debugging > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: User provided function() line 0 in unknown directory unknown > file > [3]PETSC ERROR: > ------------------------------------------------------------------------ > [3]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the > batch system) has told this process to end > [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [3]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[3]PETSC ERROR: > or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [3]PETSC ERROR: likely location of problem given in stack below > [3]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [3]PETSC ERROR: INSTEAD the line number of the start of the function > [3]PETSC ERROR: is given. > [3]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [3]PETSC ERROR: Signal received! > [3]PETSC ERROR: > ------------------------------------------------------------------------ > [3]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT > 2013 > [3]PETSC ERROR: See docs/changes/index.html for recent updates. > [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [3]PETSC ERROR: See docs/index.html for manual pages. > [3]PETSC ERROR: > ------------------------------------------------------------------------ > [3]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep > 3 20:56:21 2013 > [3]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > [3]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > [3]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 > --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 > --with-debugging > [3]PETSC ERROR: > ------------------------------------------------------------------------ > [3]PETSC ERROR: User provided function() line 0 in unknown directory unknown > file > [4]PETSC ERROR: > ------------------------------------------------------------------------ > [4]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the > batch system) has told this process to end > [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [4]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[4]PETSC ERROR: > or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [4]PETSC ERROR: likely location of problem given in stack below > [4]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [4]PETSC ERROR: INSTEAD the line number of the start of the function > [4]PETSC ERROR: is given. > [4]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [4]PETSC ERROR: Signal received! > [4]PETSC ERROR: > ------------------------------------------------------------------------ > [4]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT > 2013 > [4]PETSC ERROR: See docs/changes/index.html for recent updates. > [4]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [4]PETSC ERROR: See docs/index.html for manual pages. > [4]PETSC ERROR: > ------------------------------------------------------------------------ > [4]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep > 3 20:56:21 2013 > [4]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > [4]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > [4]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 > --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 > --with-debugging > [4]PETSC ERROR: > ------------------------------------------------------------------------ > [4]PETSC ERROR: User provided function() line 0 in unknown directory unknown > file > [5]PETSC ERROR: > ------------------------------------------------------------------------ > [5]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the > batch system) has told this process to end > [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [5]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[5]PETSC ERROR: > or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [5]PETSC ERROR: likely location of problem given in stack below > [5]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [5]PETSC ERROR: INSTEAD the line number of the start of the function > [5]PETSC ERROR: is given. > [5]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [5]PETSC ERROR: Signal received! > [5]PETSC ERROR: > ------------------------------------------------------------------------ > [5]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT > 2013 > [5]PETSC ERROR: See docs/changes/index.html for recent updates. > [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [5]PETSC ERROR: See docs/index.html for manual pages. > [5]PETSC ERROR: > ------------------------------------------------------------------------ > [5]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep > 3 20:56:21 2013 > [5]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > [5]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > [5]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 > --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 > --with-debugging > [5]PETSC ERROR: > ------------------------------------------------------------------------ > [5]PETSC ERROR: User provided function() line 0 in unknown directory unknown > file > [6]PETSC ERROR: > ------------------------------------------------------------------------ > [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the > batch system) has told this process to end > [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [6]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC ERROR: > or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [6]PETSC ERROR: likely location of problem given in stack below > [6]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [6]PETSC ERROR: INSTEAD the line number of the start of the function > [6]PETSC ERROR: is given. > [6]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [6]PETSC ERROR: Signal received! > [6]PETSC ERROR: > ------------------------------------------------------------------------ > [6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT > 2013 > [6]PETSC ERROR: See docs/changes/index.html for recent updates. > [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [6]PETSC ERROR: See docs/index.html for manual pages. > [6]PETSC ERROR: > ------------------------------------------------------------------------ > [6]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep > 3 20:56:21 2013 > [6]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > [6]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > [6]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 > --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 > --with-debugging > [6]PETSC ERROR: > ------------------------------------------------------------------------ > [6]PETSC ERROR: User provided function() line 0 in unknown directory unknown > file > [7]PETSC ERROR: > ------------------------------------------------------------------------ > [7]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the > batch system) has told this process to end > [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [7]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[7]PETSC ERROR: > or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [7]PETSC ERROR: likely location of problem given in stack below > [7]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [7]PETSC ERROR: INSTEAD the line number of the start of the function > [7]PETSC ERROR: is given. > [7]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [7]PETSC ERROR: Signal received! > [7]PETSC ERROR: > ------------------------------------------------------------------------ > [7]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT > 2013 > [7]PETSC ERROR: See docs/changes/index.html for recent updates. > [7]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [7]PETSC ERROR: See docs/index.html for manual pages. > [7]PETSC ERROR: > ------------------------------------------------------------------------ > [7]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep > 3 20:56:21 2013 > [7]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > [7]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > [7]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 > --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 > --with-debugging > [7]PETSC ERROR: > ------------------------------------------------------------------------ > [7]PETSC ERROR: User provided function() line 0 in unknown directory unknown > file > -------------------------------------------------------------------------- > mpiexec has exited due to process rank 0 with PID 2670 on > node grsacc17 exiting improperly. There are two reasons this could occur: > > 1. this process did not call "init" before exiting, but others in > the job did. This can cause a job to hang indefinitely while it waits > for all processes to call "init". By rule, if one process calls "init", > then ALL processes must call "init" prior to termination. > > 2. this process called "init", but exited without calling "finalize". > By rule, all processes that call "init" MUST call "finalize" prior to > exiting or it will be considered an "abnormal termination" > > This may have caused other processes in the application to be > terminated by signals sent by mpiexec (as reported here).
