Dear PETSC users, I hope that I can have a comment about errors I got during sparse symmetric matrix construction. In this example, I used three processes. The size of a test matrix is 52105-by-52105. The length of array d_nnz and o_nnz is 17461 at rank 0, 17111 at rank 1 and 17535 at rank 2. The two arrays exactly describe nnz of the diagonal and off-diagonal part. At rank 0, I tried to add 148767 element for the diagonal part and 5821 for the off-diagonal part. At rank 1, I tried to add 135826 and 19541 for the diagonal and off-diagonal parts, respectively. At rank 2, I tried to add 138155 and 0 for the diagonal and off-diagonal parts. Rank 0 has matrix rows from 0 to 17460; rank 1 from 17460 to 34570 and rank 2 from 34570 to 52104. During the run, I got an error message: nnz cannot be greater than block row length: local row 17124 value 38762560 rowlength 17368. My d_nnz and o_nnz arrays never have a value such as 38762560. Could anyone help me decode this error message? In advance, I appreciate your help.
Evan // CODE MatCreate(PETSC_COMM_WORLD,&B); MatSetSizes(B,PETSC_DECIDE,PETSC_DECIDE,ngedges_internal,ngedges_internal); MatSetType(B,MATSBAIJ); MatMPISBAIJSetPreallocation(B,1,0,d_nnz,0,o_nnz); delete [] d_nnz; delete [] o_nnz; MatSetValues(B, nentries_mat_b_partitioned, mat_b_i_partitioned, nentries_mat_b_partitioned, mat_b_j_partitioned, mat_b_val_partitioned, ADD_VALUES); delete [] mat_b_i_partitioned; delete [] mat_b_j_partitioned; delete [] mat_b_val_partitioned; MatAssemblyBegin(B,MAT_FINAL_ASSEMBLY); MatAssemblyEnd(B,MAT_FINAL_ASSEMBLY); //Errors: [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Argument out of range [1]PETSC ERROR: nnz cannot be greater than block row length: local row 17124 value 38762560 rowlength 17368 [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 [1]PETSC ERROR: fetdem3dp on a arch-linux2-c-debug named n0024.voltaire0 by esum Thu Nov 6 11:34:23 2014 [1]PETSC ERROR: Configure options --prefix=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 --with-ptscotch=1 --with-ptscotch-dir=/clusterfs/voltaire/home/software/modules/scotch/5.1.12-gcc/ --download-scalapack --download-metis=1 --download-superlu=1 --download-superlu_dist=1 --download-hypre=1 --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ [1]PETSC ERROR: #1 MatSeqSBAIJSetPreallocation_SeqSBAIJ() line 1599 in /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/impls/sbaij/seq/sbaij.c [1]PETSC ERROR: #2 MatSeqSBAIJSetPreallocation() line 2054 in /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/impls/sbaij/seq/sbaij.c [1]PETSC ERROR: #3 MatMPISBAIJSetPreallocation_MPISBAIJ() line 1626 in /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/impls/sbaij/mpi/mpisbaij.c [1]PETSC ERROR: #4 MatMPISBAIJSetPreallocation() line 1930 in /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/impls/sbaij/mpi/mpisbaij.c [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Object is in wrong state [1]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on argument 1 "mat" before MatSetValues() [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 [1]PETSC ERROR: fetdem3dp on a arch-linux2-c-debug named n0024.voltaire0 by esum Thu Nov 6 11:34:23 2014 [1]PETSC ERROR: Configure options --prefix=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 --with-ptscotch=1 --with-ptscotch-dir=/clusterfs/voltaire/home/software/modules/scotch/5.1.12-gcc/ --download-scalapack --download-metis=1 --download-superlu=1 --download-superlu_dist=1 --download-hypre=1 --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ [1]PETSC ERROR: #5 MatSetValues() line 1110 in /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/interface/matrix.c PNT 7 Rank:1 [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Object is in wrong state [1]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on argument 1 "mat" before MatAssemblyBegin() [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 [1]PETSC ERROR: fetdem3dp on a arch-linux2-c-debug named n0024.voltaire0 by esum Thu Nov 6 11:34:23 2014 [1]PETSC ERROR: Configure options --prefix=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 --with-ptscotch=1 --with-ptscotch-dir=/clusterfs/voltaire/home/software/modules/scotch/5.1.12-gcc/ --download-scalapack --download-metis=1 --download-superlu=1 --download-superlu_dist=1 --download-hypre=1 --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ [1]PETSC ERROR: #6 MatAssemblyBegin() line 4799 in /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/interface/matrix.c -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD with errorcode 59. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [1]PETSC ERROR: likely location of problem given in stack below [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [1]PETSC ERROR: INSTEAD the line number of the start of the function [1]PETSC ERROR: is given. [1]PETSC ERROR: [1] MatStashScatterEnd_Private line 112 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/utils/matstash.c [1]PETSC ERROR: [1] MatAssemblyEnd_MPISBAIJ line 517 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/impls/sbaij/mpi/mpisbaij.c [1]PETSC ERROR: [1] MatAssemblyEnd line 4892 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/interface/matrix.c [1]PETSC ERROR: [1] MatAssemblyBegin line 4796 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/interface/matrix.c [1]PETSC ERROR: [1] MatSetValues line 1103 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/interface/matrix.c [1]PETSC ERROR: [1] MatSeqSBAIJSetPreallocation_SeqSBAIJ line 1575 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/impls/sbaij/seq/sbaij.c [1]PETSC ERROR: [1] MatSeqSBAIJSetPreallocation line 2050 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/impls/sbaij/seq/sbaij.c [1]PETSC ERROR: [1] MatMPISBAIJSetPreallocation_MPISBAIJ line 1587 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/impls/sbaij/mpi/mpisbaij.c [1]PETSC ERROR: [1] MatMPISBAIJSetPreallocation line 1926 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/impls/sbaij/mpi/mpisbaij.c [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Signal received [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 [1]PETSC ERROR: fetdem3dp on a arch-linux2-c-debug named n0024.voltaire0 by esum Thu Nov 6 11:34:23 2014 [1]PETSC ERROR: Configure options --prefix=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 --with-ptscotch=1 --with-ptscotch-dir=/clusterfs/voltaire/home/software/modules/scotch/5.1.12-gcc/ --download-scalapack --download-metis=1 --download-superlu=1 --download-superlu_dist=1 --download-hypre=1 --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ [1]PETSC ERROR: #7 User provided function() line 0 in unknown file PNT 5 Rank:2 PNT 5 Rank:0 [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [2]PETSC ERROR: likely location of problem given in stack below [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [2]PETSC ERROR: INSTEAD the line number of the start of the function [2]PETSC ERROR: is given. [2]PETSC ERROR: [2] MatSetValues line 1103 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/interface/matrix.c [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Signal received [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [2]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 [2]PETSC ERROR: fetdem3dp on a arch-linux2-c-debug named n0024.voltaire0 by esum Thu Nov 6 11:34:23 2014 [2]PETSC ERROR: Configure options --prefix=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 --with-ptscotch=1 --with-ptscotch-dir=/clusterfs/voltaire/home/software/modules/scotch/5.1.12-gcc/ --download-scalapack --download-metis=1 --download-superlu=1 --download-superlu_dist=1 --download-hypre=1 --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ [2]PETSC ERROR: #1 User provided function() line 0 in unknown file [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] MatSetValues line 1103 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/interface/matrix.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 [0]PETSC ERROR: fetdem3dp on a arch-linux2-c-debug named n0024.voltaire0 by esum Thu Nov 6 11:34:23 2014 [0]PETSC ERROR: Configure options --prefix=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 --with-ptscotch=1 --with-ptscotch-dir=/clusterfs/voltaire/home/software/modules/scotch/5.1.12-gcc/ --download-scalapack --download-metis=1 --download-superlu=1 --download-superlu_dist=1 --download-hypre=1 --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ [0]PETSC ERROR: #1 User provided function() line 0 in unknown file --------------------------------------------------------------------------
