Hi all, Our research group recently got access to a relatively new HPC cluster (with SLURM scheduler) and I am trying to run some test examples with it on multiple processors. When I run SNES ex12 serially it works. However when i run it on two processes, I get the following error:
[0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] DMLabelMakeInvalid_Private line 81 /home/jchang23/petsc/src/dm/impls/plex/plexlabel.c [0]PETSC ERROR: [0] DMLabelSetValue line 466 /home/jchang23/petsc/src/dm/impls/plex/plexlabel.c [0]PETSC ERROR: [0] DMPlexLabelComplete line 72 /home/jchang23/petsc/src/dm/impls/plex/plexsubmesh.c [0]PETSC ERROR: [0] DMCreateDefaultSection_Plex line 5368 /home/jchang23/petsc/src/dm/impls/plex/plex.c [0]PETSC ERROR: [0] DMGetDefaultSection line 2974 /home/jchang23/petsc/src/dm/interface/dm.c [0]PETSC ERROR: [0] DMGetDefaultGlobalSection line 3129 /home/jchang23/petsc/src/dm/interface/dm.c [0]PETSC ERROR: [0] DMCreateGlobalVector_Section_Private line 12 /home/jchang23/petsc/src/dm/interface/dmi.c [0]PETSC ERROR: [0] DMCreateGlobalVector_Plex line 1152 /home/jchang23/petsc/src/dm/impls/plex/plexcreate.c [0]PETSC ERROR: [0] DMCreateGlobalVector line 683 /home/jchang23/petsc/src/dm/interface/dm.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.5.2-1351-g37b8796 GIT Date: 2015-01-11 11:57:24 -0600 [0]PETSC ERROR: ./petsc/src/snes/examples/tutorials/ex12 on a arch-linux2-c-opt named compute-0-0.local by jchang23 Sun Jan 11 21:55:42 2015 [0]PETSC ERROR: Configure options --download-fblaslapack --download-hdf5 --download-metis --download-parmetis --with-c=gcc --with-cmake=cmake --with-debugging=1 --with-fc-gfortran --with-mpi-dir=/share/apps/openmpi-1.8.3/ CPPFLAGS=-I/share/apps/openmpi-1.8.3/include LDFLAGS=-L/share/apps/openmpi-1.8.3/lib PETSC_ARCH=arch-linux2-c-opt download-chaco --download-triangle [0]PETSC ERROR: #1 User provided function() line 0 in unknown file [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [1]PETSC ERROR: likely location of problem given in stack below [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [1]PETSC ERROR: INSTEAD the line number of the start of the function [1]PETSC ERROR: is given. [1]PETSC ERROR: [1] DMLabelMakeInvalid_Private line 81 /home/jchang23/petsc/src/dm/impls/plex/plexlabel.c [1]PETSC ERROR: [1] DMLabelSetValue line 466 /home/jchang23/petsc/src/dm/impls/plex/plexlabel.c [1]PETSC ERROR: [1] DMPlexLabelComplete line 72 /home/jchang23/petsc/src/dm/impls/plex/plexsubmesh.c [1]PETSC ERROR: [1] DMCreateDefaultSection_Plex line 5368 /home/jchang23/petsc/src/dm/impls/plex/plex.c [1]PETSC ERROR: [1] DMGetDefaultSection line 2974 /home/jchang23/petsc/src/dm/interface/dm.c [1]PETSC ERROR: [1] DMGetDefaultGlobalSection line 3129 /home/jchang23/petsc/src/dm/interface/dm.c [1]PETSC ERROR: [1] DMCreateGlobalVector_Section_Private line 12 /home/jchang23/petsc/src/dm/interface/dmi.c [1]PETSC ERROR: [1] DMCreateGlobalVector_Plex line 1152 /home/jchang23/petsc/src/dm/impls/plex/plexcreate.c [1]PETSC ERROR: [1] DMCreateGlobalVector line 683 /home/jchang23/petsc/src/dm/interface/dm.c [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Signal received [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Development GIT revision: v3.5.2-1351-g37b8796 GIT Date: 2015-01-11 11:57:24 -0600 [1]PETSC ERROR: ./petsc/src/snes/examples/tutorials/ex12 on a arch-linux2-c-opt named compute-2-7.local by jchang23 Sun Jan 11 21:55:42 2015 [1]PETSC ERROR: Configure options --download-fblaslapack --download-hdf5 --download-metis --download-parmetis --with-c=gcc --with-cmake=cmake --with-debugging=1 --with-fc-gfortran --with-mpi-dir=/share/apps/openmpi-1.8.3/ CPPFLAGS=-I/share/apps/openmpi-1.8.3/include LDFLAGS=-L/share/apps/openmpi-1.8.3/lib PETSC_ARCH=arch-linux2-c-opt download-chaco --download-triangle [1]PETSC ERROR: #1 User provided function() line 0 in unknown file -------------------------------------------------------------------------- I have no idea what this means. Here is the script file used to run the problem with: #!/bin/bash #SBATCH -J test #SBATCH -o test.txt #SBATCH -N 2 mpirun -np 2 ./ex12 -run_type full -refinement_limit 0.00015625 -interpolate 1 -petscspace_order 2 -pc_type gamg -ksp_rtol 1.0e-10 -ksp_monitor_short -ksp_converged_reason -snes_monitor_short -snes_converged_reason -petscpartitioner_type parmetis --- Any help appreciated it, thanks Justin
