Dear Barry and Satish, Thank you for your responses and help. I got it all cleared now. If it will be interesting for you, what I am trying to do is spawn (using MPI_COMM_SPAWN) new processes and re-initialize petsc along with the parents and children in one communicator. At the moment, after a couple of tests, it seems to work fine with Petsc 3.4.0.
Best, Suraj On Sep 4, 2013, at 6:18 PM, Satish Balay wrote: > On Wed, 4 Sep 2013, Barry Smith wrote: > >> >>>> In file included from /opt/petsc-3.4.2/include/petsc.h:8, >>>> from Main/time_integration_dir/compute_tau_w_mean.c:91: >>>> /opt/petsc-3.4.2/include/petscbt.h: In function ‘compute_tau_w_mean’: >>>> /opt/petsc-3.4.2/include/petscbt.h:33: error: invalid storage class for >>>> function ‘PetscBTLength’ >>>> /opt/petsc-3.4.2/include/petscbt.h:38: error: invalid storage class for >>>> function ‘PetscBTMemzero’ >>>> /opt/petsc-3.4.2/include/petscbt.h:43: error: invalid storage class for >>>> function ‘PetscBTDestroy’ >> >> >> Ok, nothing wrong with the static inline or build; There must be something >> wonky in compute_tau_w_mean.c or a file it includes that is causing the >> error. >> >> I would start debugging by removing stuff from compute_tau_w_mean.c before >> the #include "petsc.h" until the error goes away and that will show the >> culprit. > > Also you have a huge list of -I dirs. Make sure a wrong petscconf.h isn't > getting > included from one of these locations. > > I would do a 'make CFLAGS=-E' compute_tau_w_mean.o > > and look at the preprocessed 'PetscBTLength' reference in there. > > Also 'grep petscconf.h compute_tau_w_mean.o' > > Satish > > >> >> Barry >> >> >> >> On Sep 3, 2013, at 10:21 PM, Suraj Prabhakaran <[email protected]> >> wrote: >> >>> Hi Barry, >>> >>> Thanks a lot! Here are the files attached. >>> >>> Suraj >>> >>> <configure.log><make.log> >>> >>> On Sep 4, 2013, at 5:13 AM, Barry Smith wrote: >>> >>>> >>>> Something is going wrong with the PETSC_STATIC_INLINE definition. Please >>>> send configure.log and make.log >>>> >>>> Barry >>>> >>>> On Sep 3, 2013, at 10:01 PM, Suraj Prabhakaran >>>> <[email protected]> wrote: >>>> >>>>> Thanks a lot for the confirmation!! >>>>> I am now compiling it with 3.4.2, but I get a strange error and could be >>>>> great if someone can just point out whats wrong. (Might be something >>>>> stupid, but just in case...) >>>>> >>>>> Thanks in advance! >>>>> >>>>> Suraj >>>>> >>>>> -------------------------------ERROR----------------------------- >>>>> >>>>> In file included from /opt/petsc-3.4.2/include/petsc.h:8, >>>>> from Main/time_integration_dir/compute_tau_w_mean.c:91: >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ‘compute_tau_w_mean’: >>>>> /opt/petsc-3.4.2/include/petscbt.h:33: error: invalid storage class for >>>>> function ‘PetscBTLength’ >>>>> /opt/petsc-3.4.2/include/petscbt.h:38: error: invalid storage class for >>>>> function ‘PetscBTMemzero’ >>>>> /opt/petsc-3.4.2/include/petscbt.h:43: error: invalid storage class for >>>>> function ‘PetscBTDestroy’ >>>>> /opt/petsc-3.4.2/include/petscbt.h:48: error: invalid storage class for >>>>> function ‘PetscBTLookup’ >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ‘PetscBTLookup’: >>>>> /opt/petsc-3.4.2/include/petscbt.h:47: warning: declaration of ‘index’ >>>>> shadows a global declaration >>>>> /usr/include/string.h:489: warning: shadowed declaration is here >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ‘compute_tau_w_mean’: >>>>> /opt/petsc-3.4.2/include/petscbt.h:59: error: invalid storage class for >>>>> function ‘PetscBTView’ >>>>> /opt/petsc-3.4.2/include/petscbt.h:74: error: invalid storage class for >>>>> function ‘PetscBTCreate’ >>>>> /opt/petsc-3.4.2/include/petscbt.h:79: error: invalid storage class for >>>>> function ‘PetscBTLookupSet’ >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ‘PetscBTLookupSet’: >>>>> /opt/petsc-3.4.2/include/petscbt.h:78: warning: declaration of ‘index’ >>>>> shadows a global declaration >>>>> /usr/include/string.h:489: warning: shadowed declaration is here >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ‘compute_tau_w_mean’: >>>>> /opt/petsc-3.4.2/include/petscbt.h:91: error: invalid storage class for >>>>> function ‘PetscBTSet’ >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ‘PetscBTSet’: >>>>> /opt/petsc-3.4.2/include/petscbt.h:90: warning: declaration of ‘index’ >>>>> shadows a global declaration >>>>> /usr/include/string.h:489: warning: shadowed declaration is here >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ‘compute_tau_w_mean’: >>>>> /opt/petsc-3.4.2/include/petscbt.h:103: error: invalid storage class for >>>>> function ‘PetscBTLookupClear’ >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ‘PetscBTLookupClear’: >>>>> /opt/petsc-3.4.2/include/petscbt.h:102: warning: declaration of ‘index’ >>>>> shadows a global declaration >>>>> /usr/include/string.h:489: warning: shadowed declaration is here >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ‘compute_tau_w_mean’: >>>>> /opt/petsc-3.4.2/include/petscbt.h:115: error: invalid storage class for >>>>> function ‘PetscBTClear’ >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ‘PetscBTClear’: >>>>> /opt/petsc-3.4.2/include/petscbt.h:114: warning: declaration of ‘index’ >>>>> shadows a global declaration >>>>> /usr/include/string.h:489: warning: shadowed declaration is here >>>>> In file included from /opt/petsc-3.4.2/include/petsc.h:9, >>>>> from Main/time_integration_dir/compute_tau_w_mean.c:91: >>>>> /opt/petsc-3.4.2/include/petscctable.h: In function ‘compute_tau_w_mean’: >>>>> /opt/petsc-3.4.2/include/petscctable.h:20: error: invalid storage class >>>>> for function ‘PetscHash’ >>>>> /opt/petsc-3.4.2/include/petscctable.h:40: error: invalid storage class >>>>> for function ‘PetscTableAdd’ >>>>> In file included from /opt/petsc-3.4.2/include/petsc.h:9, >>>>> from Main/time_integration_dir/compute_tau_w_mean.c:91: >>>>> /opt/petsc-3.4.2/include/petscctable.h:83: error: invalid storage class >>>>> for function ‘PetscTableAddCount’ >>>>> /opt/petsc-3.4.2/include/petscctable.h:120: error: invalid storage class >>>>> for function ‘PetscTableFind’ >>>>> In file included from /opt/petsc-3.4.2/include/petscsnesfas.h:3, >>>>> from /opt/petsc-3.4.2/include/petsc.h:27, >>>>> from Main/time_integration_dir/compute_tau_w_mean.c:91: >>>>> /opt/petsc-3.4.2/include/petscsnes.h:585: error: invalid storage class >>>>> for function ‘SNESGetSNESLineSearch’ >>>>> /opt/petsc-3.4.2/include/petscsnes.h:586: error: invalid storage class >>>>> for function ‘SNESSetSNESLineSearch’ >>>>> Main/time_integration_dir/compute_tau_w_mean.c:223: warning: implicit >>>>> declaration of function ‘wall_thermoquants’ >>>>> Main/time_integration_dir/compute_tau_w_mean.c:155: warning: unused >>>>> variable ‘iface_plus_one’ >>>>> Main/time_integration_dir/compute_tau_w_mean.c:330: warning: value >>>>> computed is not used >>>>> Main/time_integration_dir/compute_tau_w_mean.c:116: warning: unused >>>>> variable ‘ierr’ >>>>> Main/time_integration_dir/compute_tau_w_mean.c:111: warning: unused >>>>> variable ‘icell’ >>>>> Main/time_integration_dir/compute_tau_w_mean.c:105: warning: unused >>>>> variable ‘mach’ >>>>> /opt/petsc-3.4.2/include/petscts.h:328: warning: unused variable >>>>> ‘TSRegisterAllCalled’ >>>>> /opt/petsc-3.4.2/include/petscts.h:327: warning: unused variable ‘TSList’ >>>>> /opt/petsc-3.4.2/include/petscts.h:168: warning: unused variable >>>>> ‘DMTS_CLASSID’ >>>>> /opt/petsc-3.4.2/include/petscts.h:167: warning: unused variable >>>>> ‘TS_CLASSID’ >>>>> /opt/petsc-3.4.2/include/petscts.h:163: warning: unused variable >>>>> ‘TSExactFinalTimeOptions’ >>>>> /opt/petsc-3.4.2/include/petscts.h:101: warning: unused variable >>>>> ‘TSConvergedReasons’ >>>>> /opt/petsc-3.4.2/include/petscts.h:80: warning: unused variable >>>>> ‘TSEquationTypes’ >>>>> /opt/petsc-3.4.2/include/petscsnesfas.h:19: warning: unused variable >>>>> ‘SNESFASTypes’ >>>>> /opt/petsc-3.4.2/include/petscsnes.h:679: warning: unused variable >>>>> ‘SNESQNRestartTypes’ >>>>> /opt/petsc-3.4.2/include/petscsnes.h:675: warning: unused variable >>>>> ‘SNESQNScaleTypes’ >>>>> /opt/petsc-3.4.2/include/petscsnes.h:667: warning: unused variable >>>>> ‘SNESNCGTypes’ >>>>> /opt/petsc-3.4.2/include/petscsnes.h:654: warning: unused variable >>>>> ‘SNESNGMRESSelectTypes’ >>>>> /opt/petsc-3.4.2/include/petscsnes.h:648: warning: unused variable >>>>> ‘SNESNGMRESRestartTypes’ >>>>> /opt/petsc-3.4.2/include/petscsnes.h:474: warning: unused variable >>>>> ‘SNESLineSearch_Apply’ >>>>> /opt/petsc-3.4.2/include/petscsnes.h:473: warning: unused variable >>>>> ‘SNESLineSearchList’ >>>>> /opt/petsc-3.4.2/include/petscsnes.h:472: warning: unused variable >>>>> ‘SNESLineSearchRegisterAllCalled’ >>>>> /opt/petsc-3.4.2/include/petscsnes.h:471: warning: unused variable >>>>> ‘SNESLINESEARCH_CLASSID’ >>>>> /opt/petsc-3.4.2/include/petscsnes.h:470: warning: unused variable >>>>> ‘SNESList’ >>>>> /opt/petsc-3.4.2/include/petscsnes.h:469: warning: unused variable >>>>> ‘SNESRegisterAllCalled’ >>>>> /opt/petsc-3.4.2/include/petscsnes.h:356: warning: unused variable >>>>> ‘SNESNormTypes’ >>>>> /opt/petsc-3.4.2/include/petscsnes.h:217: warning: unused variable >>>>> ‘SNESConvergedReasons’ >>>>> /opt/petsc-3.4.2/include/petscsnes.h:50: warning: unused variable >>>>> ‘DMSNES_CLASSID’ >>>>> /opt/petsc-3.4.2/include/petscsnes.h:49: warning: unused variable >>>>> ‘SNES_CLASSID’ >>>>> /opt/petsc-3.4.2/include/petsccharacteristic.h:44: warning: unused >>>>> variable ‘CharacteristicList’ >>>>> /opt/petsc-3.4.2/include/petsccharacteristic.h:43: warning: unused >>>>> variable ‘CharacteristicRegisterAllCalled’ >>>>> /opt/petsc-3.4.2/include/petscdmda.h:27: warning: unused variable >>>>> ‘DMDABoundaryTypes’ >>>>> /opt/petsc-3.4.2/include/petscpf.h:50: warning: unused variable >>>>> ‘PFRegisterAllCalled’ >>>>> /opt/petsc-3.4.2/include/petscpf.h:40: warning: unused variable >>>>> ‘PF_CLASSID’ >>>>> /opt/petsc-3.4.2/include/petscpf.h:12: warning: unused variable ‘PFList’ >>>>> /opt/petsc-3.4.2/include/petscdm.h:34: warning: unused variable >>>>> ‘DMRegisterAllCalled’ >>>>> /opt/petsc-3.4.2/include/petscdm.h:33: warning: unused variable ‘DMList’ >>>>> /opt/petsc-3.4.2/include/petscdm.h:11: warning: unused variable >>>>> ‘DM_CLASSID’ >>>>> /opt/petsc-3.4.2/include/petscdraw.h:239: warning: unused variable >>>>> ‘PETSC_DRAWHG_CLASSID’ >>>>> /opt/petsc-3.4.2/include/petscdraw.h:225: warning: unused variable >>>>> ‘PETSC_DRAWSP_CLASSID’ >>>>> /opt/petsc-3.4.2/include/petscdraw.h:206: warning: unused variable >>>>> ‘PETSC_DRAWLG_CLASSID’ >>>>> /opt/petsc-3.4.2/include/petscdraw.h:195: warning: unused variable >>>>> ‘PETSC_DRAWAXIS_CLASSID’ >>>>> /opt/petsc-3.4.2/include/petscdraw.h:26: warning: unused variable >>>>> ‘PetscDrawList’ >>>>> /opt/petsc-3.4.2/include/petscdraw.h:9: warning: unused variable >>>>> ‘PETSC_DRAW_CLASSID’ >>>>> /opt/petsc-3.4.2/include/petscmatlab.h:8: warning: unused variable >>>>> ‘MATLABENGINE_CLASSID’ >>>>> /opt/petsc-3.4.2/include/petscthreadcomm.h:35: warning: unused variable >>>>> ‘PetscThreadCommReductionOps’ >>>>> /opt/petsc-3.4.2/include/petscthreadcomm.h:32: warning: unused variable >>>>> ‘PetscThreadCommList’ >>>>> >>>>> >>>>> >>>>> On Sep 3, 2013, at 9:03 PM, Satish Balay wrote: >>>>> >>>>>> Also you would have to call MPI_Init() before the first >>>>>> PetscInitialize() [and >>>>>> similarly MPI_Finalize() after the last call to PetscFinalize()] >>>>>> >>>>>> Satish >>>>>> >>>>>> On Tue, 3 Sep 2013, Barry Smith wrote: >>>>>> >>>>>>> >>>>>>> There is a problem with the 3.3 series when reinitializing PETSc. Can >>>>>>> you switch to PETSc 3.4 which properly supports calling >>>>>>> PetscInitialize() several times. >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> On Sep 3, 2013, at 1:58 PM, Suraj Prabhakaran >>>>>>> <[email protected]> wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> Again, this relates to finalizing and initializing petsc in my >>>>>>>> application. I basically want to finalize petsc and initialize it >>>>>>>> again with a different MPI communicator. A rough sketch of my quite >>>>>>>> huge application is as follows. >>>>>>>> >>>>>>>> >>>>>>>> main() >>>>>>>> { >>>>>>>> PetscInitialize() >>>>>>>> ... >>>>>>>> ... >>>>>>>> AOCreate().. >>>>>>>> AO operations.. >>>>>>>> AODestroy().. >>>>>>>> ... >>>>>>>> ... >>>>>>>> PetscFinalize() >>>>>>>> PETSC_COMM_WORLD = A new communicator (which includes more processes) >>>>>>>> PetscInitialize() >>>>>>>> ... >>>>>>>> ... >>>>>>>> AOCreate().. >>>>>>>> AO operations.. >>>>>>>> AODestroy() >>>>>>>> ... >>>>>>>> ... >>>>>>>> PetscFinalize() >>>>>>>> } >>>>>>>> >>>>>>>> So after finalizing and initializing it back, I always get an error >>>>>>>> with one of the AO calls. Given below is the complete error I get when >>>>>>>> running with 8 procs. Can anyone give me a hint what the problem could >>>>>>>> be? I was told earlier, that finalizing and initializing petsc should >>>>>>>> not be a problem! >>>>>>>> >>>>>>>> Best, >>>>>>>> Suraj >>>>>>>> >>>>>>>> -------------------------ERROR starts >>>>>>>> here------------------------------- >>>>>>>> [0]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >>>>>>>> probably memory access out of range >>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or >>>>>>>> -on_error_attach_debugger >>>>>>>> [0]PETSC ERROR: or see >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC >>>>>>>> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to >>>>>>>> find memory corruption errors >>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>>>> ------------------------------------ >>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>>>>>> available, >>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the >>>>>>>> function >>>>>>>> [0]PETSC ERROR: is given. >>>>>>>> [0]PETSC ERROR: [0] PetscStrcmp line 414 >>>>>>>> /root/petsc/petsc-3.3-p7/src/sys/utils/str.c >>>>>>>> [0]PETSC ERROR: [0] PetscFListAdd line 202 >>>>>>>> /root/petsc/petsc-3.3-p7/src/sys/dll/reg.c >>>>>>>> [0]PETSC ERROR: [0] AORegister line 99 >>>>>>>> /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoreg.c >>>>>>>> [0]PETSC ERROR: [0] AORegisterAll line 28 >>>>>>>> /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoregall.c >>>>>>>> [0]PETSC ERROR: [0] AOInitializePackage line 49 >>>>>>>> /root/petsc/petsc-3.3-p7/src/dm/ao/interface/dlregisdm.c >>>>>>>> [0]PETSC ERROR: [0] AOCreate line 530 >>>>>>>> /root/petsc/petsc-3.3-p7/src/dm/ao/interface/ao.c >>>>>>>> [0]PETSC ERROR: [0] AOCreateBasicIS line 381 >>>>>>>> /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c >>>>>>>> [0]PETSC ERROR: [0] AOCreateBasic line 336 >>>>>>>> /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c >>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>> ------------------------------------ >>>>>>>> [0]PETSC ERROR: Signal received! >>>>>>>> [0]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 >>>>>>>> 22:15:24 CDT 2013 >>>>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>> [0]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [0]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh >>>>>>>> Tue Sep 3 20:56:21 2013 >>>>>>>> [0]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>>>> [0]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>>>> [0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 >>>>>>>> --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 >>>>>>>> --with-debugging >>>>>>>> [0]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [0]PETSC ERROR: User provided function() line 0 in unknown directory >>>>>>>> unknown file >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>>>>>> with errorcode 59. >>>>>>>> >>>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>>>>> You may or may not see output from other processes, depending on >>>>>>>> exactly when Open MPI kills them. >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> [1]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [1]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or >>>>>>>> the batch system) has told this process to end >>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger or >>>>>>>> -on_error_attach_debugger >>>>>>>> [1]PETSC ERROR: or see >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC >>>>>>>> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to >>>>>>>> find memory corruption errors >>>>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames >>>>>>>> ------------------------------------ >>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>>>>>> available, >>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the >>>>>>>> function >>>>>>>> [1]PETSC ERROR: is given. >>>>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>>>> ------------------------------------ >>>>>>>> [1]PETSC ERROR: Signal received! >>>>>>>> [1]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 >>>>>>>> 22:15:24 CDT 2013 >>>>>>>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>> [1]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>> [1]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [1]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh >>>>>>>> Tue Sep 3 20:56:21 2013 >>>>>>>> [1]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>>>> [1]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>>>> [1]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 >>>>>>>> --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 >>>>>>>> --with-debugging >>>>>>>> [1]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [1]PETSC ERROR: User provided function() line 0 in unknown directory >>>>>>>> unknown file >>>>>>>> [2]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [2]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or >>>>>>>> the batch system) has told this process to end >>>>>>>> [2]PETSC ERROR: Try option -start_in_debugger or >>>>>>>> -on_error_attach_debugger >>>>>>>> [2]PETSC ERROR: or see >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSC >>>>>>>> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to >>>>>>>> find memory corruption errors >>>>>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [2]PETSC ERROR: --------------------- Stack Frames >>>>>>>> ------------------------------------ >>>>>>>> [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>>>>>> available, >>>>>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the >>>>>>>> function >>>>>>>> [2]PETSC ERROR: is given. >>>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>>> ------------------------------------ >>>>>>>> [2]PETSC ERROR: Signal received! >>>>>>>> [2]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 >>>>>>>> 22:15:24 CDT 2013 >>>>>>>> [2]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>> [2]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>> [2]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [2]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh >>>>>>>> Tue Sep 3 20:56:21 2013 >>>>>>>> [2]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>>>> [2]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>>>> [2]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 >>>>>>>> --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 >>>>>>>> --with-debugging >>>>>>>> [2]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [2]PETSC ERROR: User provided function() line 0 in unknown directory >>>>>>>> unknown file >>>>>>>> [3]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [3]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or >>>>>>>> the batch system) has told this process to end >>>>>>>> [3]PETSC ERROR: Try option -start_in_debugger or >>>>>>>> -on_error_attach_debugger >>>>>>>> [3]PETSC ERROR: or see >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[3]PETSC >>>>>>>> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to >>>>>>>> find memory corruption errors >>>>>>>> [3]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [3]PETSC ERROR: --------------------- Stack Frames >>>>>>>> ------------------------------------ >>>>>>>> [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>>>>>> available, >>>>>>>> [3]PETSC ERROR: INSTEAD the line number of the start of the >>>>>>>> function >>>>>>>> [3]PETSC ERROR: is given. >>>>>>>> [3]PETSC ERROR: --------------------- Error Message >>>>>>>> ------------------------------------ >>>>>>>> [3]PETSC ERROR: Signal received! >>>>>>>> [3]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [3]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 >>>>>>>> 22:15:24 CDT 2013 >>>>>>>> [3]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>> [3]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>> [3]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [3]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh >>>>>>>> Tue Sep 3 20:56:21 2013 >>>>>>>> [3]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>>>> [3]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>>>> [3]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 >>>>>>>> --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 >>>>>>>> --with-debugging >>>>>>>> [3]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [3]PETSC ERROR: User provided function() line 0 in unknown directory >>>>>>>> unknown file >>>>>>>> [4]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [4]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or >>>>>>>> the batch system) has told this process to end >>>>>>>> [4]PETSC ERROR: Try option -start_in_debugger or >>>>>>>> -on_error_attach_debugger >>>>>>>> [4]PETSC ERROR: or see >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[4]PETSC >>>>>>>> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to >>>>>>>> find memory corruption errors >>>>>>>> [4]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [4]PETSC ERROR: --------------------- Stack Frames >>>>>>>> ------------------------------------ >>>>>>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>>>>>> available, >>>>>>>> [4]PETSC ERROR: INSTEAD the line number of the start of the >>>>>>>> function >>>>>>>> [4]PETSC ERROR: is given. >>>>>>>> [4]PETSC ERROR: --------------------- Error Message >>>>>>>> ------------------------------------ >>>>>>>> [4]PETSC ERROR: Signal received! >>>>>>>> [4]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [4]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 >>>>>>>> 22:15:24 CDT 2013 >>>>>>>> [4]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>> [4]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>> [4]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>> [4]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [4]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh >>>>>>>> Tue Sep 3 20:56:21 2013 >>>>>>>> [4]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>>>> [4]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>>>> [4]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 >>>>>>>> --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 >>>>>>>> --with-debugging >>>>>>>> [4]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [4]PETSC ERROR: User provided function() line 0 in unknown directory >>>>>>>> unknown file >>>>>>>> [5]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [5]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or >>>>>>>> the batch system) has told this process to end >>>>>>>> [5]PETSC ERROR: Try option -start_in_debugger or >>>>>>>> -on_error_attach_debugger >>>>>>>> [5]PETSC ERROR: or see >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[5]PETSC >>>>>>>> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to >>>>>>>> find memory corruption errors >>>>>>>> [5]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [5]PETSC ERROR: --------------------- Stack Frames >>>>>>>> ------------------------------------ >>>>>>>> [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>>>>>> available, >>>>>>>> [5]PETSC ERROR: INSTEAD the line number of the start of the >>>>>>>> function >>>>>>>> [5]PETSC ERROR: is given. >>>>>>>> [5]PETSC ERROR: --------------------- Error Message >>>>>>>> ------------------------------------ >>>>>>>> [5]PETSC ERROR: Signal received! >>>>>>>> [5]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [5]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 >>>>>>>> 22:15:24 CDT 2013 >>>>>>>> [5]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>> [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>> [5]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>> [5]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [5]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh >>>>>>>> Tue Sep 3 20:56:21 2013 >>>>>>>> [5]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>>>> [5]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>>>> [5]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 >>>>>>>> --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 >>>>>>>> --with-debugging >>>>>>>> [5]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [5]PETSC ERROR: User provided function() line 0 in unknown directory >>>>>>>> unknown file >>>>>>>> [6]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or >>>>>>>> the batch system) has told this process to end >>>>>>>> [6]PETSC ERROR: Try option -start_in_debugger or >>>>>>>> -on_error_attach_debugger >>>>>>>> [6]PETSC ERROR: or see >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC >>>>>>>> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to >>>>>>>> find memory corruption errors >>>>>>>> [6]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [6]PETSC ERROR: --------------------- Stack Frames >>>>>>>> ------------------------------------ >>>>>>>> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>>>>>> available, >>>>>>>> [6]PETSC ERROR: INSTEAD the line number of the start of the >>>>>>>> function >>>>>>>> [6]PETSC ERROR: is given. >>>>>>>> [6]PETSC ERROR: --------------------- Error Message >>>>>>>> ------------------------------------ >>>>>>>> [6]PETSC ERROR: Signal received! >>>>>>>> [6]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 >>>>>>>> 22:15:24 CDT 2013 >>>>>>>> [6]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>> [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>> [6]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>> [6]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [6]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh >>>>>>>> Tue Sep 3 20:56:21 2013 >>>>>>>> [6]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>>>> [6]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>>>> [6]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 >>>>>>>> --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 >>>>>>>> --with-debugging >>>>>>>> [6]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [6]PETSC ERROR: User provided function() line 0 in unknown directory >>>>>>>> unknown file >>>>>>>> [7]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [7]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or >>>>>>>> the batch system) has told this process to end >>>>>>>> [7]PETSC ERROR: Try option -start_in_debugger or >>>>>>>> -on_error_attach_debugger >>>>>>>> [7]PETSC ERROR: or see >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[7]PETSC >>>>>>>> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to >>>>>>>> find memory corruption errors >>>>>>>> [7]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [7]PETSC ERROR: --------------------- Stack Frames >>>>>>>> ------------------------------------ >>>>>>>> [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>>>>>> available, >>>>>>>> [7]PETSC ERROR: INSTEAD the line number of the start of the >>>>>>>> function >>>>>>>> [7]PETSC ERROR: is given. >>>>>>>> [7]PETSC ERROR: --------------------- Error Message >>>>>>>> ------------------------------------ >>>>>>>> [7]PETSC ERROR: Signal received! >>>>>>>> [7]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [7]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 >>>>>>>> 22:15:24 CDT 2013 >>>>>>>> [7]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>> [7]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>> [7]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>> [7]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [7]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh >>>>>>>> Tue Sep 3 20:56:21 2013 >>>>>>>> [7]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>>>> [7]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>>>> [7]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 >>>>>>>> --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 >>>>>>>> --with-debugging >>>>>>>> [7]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> [7]PETSC ERROR: User provided function() line 0 in unknown directory >>>>>>>> unknown file >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> mpiexec has exited due to process rank 0 with PID 2670 on >>>>>>>> node grsacc17 exiting improperly. There are two reasons this could >>>>>>>> occur: >>>>>>>> >>>>>>>> 1. this process did not call "init" before exiting, but others in >>>>>>>> the job did. This can cause a job to hang indefinitely while it waits >>>>>>>> for all processes to call "init". By rule, if one process calls "init", >>>>>>>> then ALL processes must call "init" prior to termination. >>>>>>>> >>>>>>>> 2. this process called "init", but exited without calling "finalize". >>>>>>>> By rule, all processes that call "init" MUST call "finalize" prior to >>>>>>>> exiting or it will be considered an "abnormal termination" >>>>>>>> >>>>>>>> This may have caused other processes in the application to be >>>>>>>> terminated by signals sent by mpiexec (as reported here). >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>
