Try to run with valgrind. --Junchao Zhang
On Mon, Jul 6, 2020 at 3:05 PM Eda Oktay <[email protected]> wrote: > I tried for (count=1;count<2;++count), it is working. When I raised it to > 3, it started giving the same error. > > My loop is inside my compiled program, not bash. > > I started the loop after PetscInitialize and finished it before > PetscFinalize. > > I am not using MPI_Init. > > Thanks! > > On Mon, Jul 6, 2020, 10:51 PM Barry Smith <[email protected]> wrote: > >> >> Where is your loop? In a bash script or inside your compiled program? >> >> If inside the program I assume it is outside of PetscInitialize()? >> >> Is it outside of MPI_Init()? It can't be because MPI_Init() can only be >> called once. >> >> If inside the problem you can use -start_in_debugger to get the true >> location of the crash, if in a bash script you can use >> -on_error_attach_debugger >> >> Barry >> >> >> > On Jul 6, 2020, at 1:14 PM, Eda Oktay <[email protected]> wrote: >> > >> > Hello, >> > >> > I am trying to run my codes 100 times and then take the average >> > elapsed time. Although my program runs correctly for single time, when >> > I put them into loop, I am getting the following error: >> > >> > *** Error in `./avg_coloredge_deneme_new_vecscatter_arastep': >> > munmap_chunk(): invalid pointer: 0x0000000000b0cbc0 *** >> > [1]PETSC ERROR: >> > ------------------------------------------------------------------------ >> > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> > probably memory access out of range >> > [1]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> > [1]PETSC ERROR: or see >> > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >> > OS X to find memory corruption errors >> > [1]PETSC ERROR: likely location of problem given in stack below >> > [1]PETSC ERROR: --------------------- Stack Frames >> > ------------------------------------ >> > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> > [1]PETSC ERROR: INSTEAD the line number of the start of the >> function >> > [1]PETSC ERROR: is given. >> > ======= Backtrace: ========= >> > [1]PETSC ERROR: /lib64/libc.so.6(+0x791eb)[0x7f810f22a1eb] >> > --------------------- Error Message >> > -------------------------------------------------------------- >> > [1]PETSC ERROR: /lib64/libc.so.6(cfree+0x1f8)[0x7f810f237438] >> > Signal received >> > [1]PETSC ERROR: See >> > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> > shooting. >> > [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 >> > [1]PETSC ERROR: ./avg_coloredge_deneme_new_vecscatter_arastep on a >> > arch-linux2-c-debug named localhost.localdomain by edaoktay Mon Jul 6 >> > 21:09:07 2020 >> > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> > --download-mpich --download-openblas --download-slepc --download-metis >> > --download-parmetis --download-chaco --with-X=1 >> > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file >> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402dca] >> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402f1e] >> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402953] >> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x406c1c] >> > application called MPI_Abort(MPI_COMM_WORLD, 50152059) - process 1 >> > /lib64/libc.so.6(__libc_start_main+0xf1)[0x7f810f1d1431] >> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x40260a] >> > ======= Memory map: ======== >> > 00400000-00409000 r-xp 00000000 fd:02 29230434 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep >> > 00608000-00609000 r--p 00008000 fd:02 29230434 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep >> > 00609000-0060a000 rw-p 00009000 fd:02 29230434 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep >> > 00956000-01100000 rw-p 00000000 00:00 0 >> [heap] >> > 7f810c621000-7f810c6ca000 rw-p 00000000 00:00 0 >> > 7f810c6ca000-7f810e6ca000 rw-p 00000000 00:00 0 >> > 7f810e6ca000-7f810e6d4000 r-xp 00000000 fd:00 1974339 >> > /usr/lib64/libnss_files-2.24.so >> > 7f810e6d4000-7f810e8d4000 ---p 0000a000 fd:00 1974339 >> > /usr/lib64/libnss_files-2.24.so >> > 7f810e8d4000-7f810e8d5000 r--p 0000a000 fd:00 1974339 >> > /usr/lib64/libnss_files-2.24.so >> > 7f810e8d5000-7f810e8d6000 rw-p 0000b000 fd:00 1974339 >> > /usr/lib64/libnss_files-2.24.so >> > 7f810e8d6000-7f810eb7d000 rw-p 00000000 00:00 0 >> > 7f810eb7d000-7f810eb7f000 r-xp 00000000 fd:00 1974984 >> > /usr/lib64/libXau.so.6.0.0 >> > 7f810eb7f000-7f810ed7f000 ---p 00002000 fd:00 1974984 >> > /usr/lib64/libXau.so.6.0.0 >> > 7f810ed7f000-7f810ed80000 r--p 00002000 fd:00 1974984 >> > /usr/lib64/libXau.so.6.0.0 >> > 7f810ed80000-7f810ed81000 rw-p 00000000 00:00 0 >> > 7f810ed81000-7f810ed88000 r-xp 00000000 fd:00 1974440 >> > /usr/lib64/librt-2.24.so >> > 7f810ed88000-7f810ef87000 ---p 00007000 fd:00 1974440 >> > /usr/lib64/librt-2.24.so >> > 7f810ef87000-7f810ef88000 r--p 00006000 fd:00 1974440 >> > /usr/lib64/librt-2.24.so >> > 7f810ef88000-7f810ef89000 rw-p 00007000 fd:00 1974440 >> > /usr/lib64/librt-2.24.so >> > 7f810ef89000-7f810efb0000 r-xp 00000000 fd:00 1975953 >> > /usr/lib64/libxcb.so.1.1.0 >> > 7f810efb0000-7f810f1af000 ---p 00027000 fd:00 1975953 >> > /usr/lib64/libxcb.so.1.1.0 >> > 7f810f1af000-7f810f1b0000 r--p 00026000 fd:00 1975953 >> > /usr/lib64/libxcb.so.1.1.0 >> > 7f810f1b0000-7f810f1b1000 rw-p 00027000 fd:00 1975953 >> > /usr/lib64/libxcb.so.1.1.0 >> > 7f810f1b1000-7f810f36e000 r-xp 00000000 fd:00 1973798 >> > /usr/lib64/libc-2.24.so >> > 7f810f36e000-7f810f56d000 ---p 001bd000 fd:00 1973798 >> > /usr/lib64/libc-2.24.so >> > 7f810f56d000-7f810f571000 r--p 001bc000 fd:00 1973798 >> > /usr/lib64/libc-2.24.so >> > 7f810f571000-7f810f573000 rw-p 001c0000 fd:00 1973798 >> > /usr/lib64/libc-2.24.so >> > 7f810f573000-7f810f577000 rw-p 00000000 00:00 0 >> > 7f810f577000-7f810f5b6000 r-xp 00000000 fd:00 1975094 >> > /usr/lib64/libquadmath.so.0.0.0 >> > 7f810f5b6000-7f810f7b5000 ---p 0003f000 fd:00 1975094 >> > /usr/lib64/libquadmath.so.0.0.0 >> > 7f810f7b5000-7f810f7b6000 r--p 0003e000 fd:00 1975094 >> > /usr/lib64/libquadmath.so.0.0.0 >> > 7f810f7b6000-7f810f7b7000 rw-p 0003f000 fd:00 1975094 >> > /usr/lib64/libquadmath.so.0.0.0 >> > 7f810f7b7000-7f810f7cd000 r-xp 00000000 fd:00 1973274 >> > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 >> > 7f810f7cd000-7f810f9cc000 ---p 00016000 fd:00 1973274 >> > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 >> > 7f810f9cc000-7f810f9cd000 r--p 00015000 fd:00 1973274 >> > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 >> > 7f810f9cd000-7f810f9ce000 rw-p 00016000 fd:00 1973274 >> > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 >> > 7f810f9ce000-7f810fafc000 r-xp 00000000 fd:00 1975116 >> > /usr/lib64/libgfortran.so.3.0.0 >> > 7f810fafc000-7f810fcfc000 ---p 0012e000 fd:00 1975116 >> > /usr/lib64/libgfortran.so.3.0.0 >> > 7f810fcfc000-7f810fcfd000 r--p 0012e000 fd:00 1975116 >> > /usr/lib64/libgfortran.so.3.0.0 >> > 7f810fcfd000-7f810fcfe000 rw-p 0012f000 fd:00 1975116 >> > /usr/lib64/libgfortran.so.3.0.0 >> > 7f810fcfe000-7f810fcff000 rw-p 00000000 00:00 0 >> > 7f810fcff000-7f8110168000 r-xp 00000000 fd:02 28457339 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 >> > 7f8110168000-7f8110368000 ---p 00469000 fd:02 28457339 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 >> > 7f8110368000-7f811037a000 r--p 00469000 fd:02 28457339 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 >> > 7f811037a000-7f8110385000 rw-p 0047b000 fd:02 28457339 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 >> > 7f8110385000-7f81103c4000 rw-p 00000000 00:00 0 >> > 7f81103c4000-7f8110435000 r-xp 00000000 fd:02 28457343 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 >> > 7f8110435000-7f8110635000 ---p 00071000 fd:02 28457343 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 >> > 7f8110635000-7f8110639000 r--p 00071000 fd:02 28457343 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 >> > 7f8110639000-7f811063b000 rw-p 00075000 fd:02 28457343 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 >> > 7f811063b000-7f811063e000 r-xp 00000000 fd:00 1973895 >> > /usr/lib64/libdl-2.24.so >> > 7f811063e000-7f811083d000 ---p 00003000 fd:00 1973895 >> > /usr/lib64/libdl-2.24.so >> > 7f811083d000-7f811083e000 r--p 00002000 fd:00 1973895 >> > /usr/lib64/libdl-2.24.so >> > 7f811083e000-7f811083f000 rw-p 00003000 fd:00 1973895 >> > /usr/lib64/libdl-2.24.so >> > 7f811083f000-7f81109b7000 r-xp 00000000 fd:00 1974512 >> > /usr/lib64/libstdc++.so.6.0.22 >> > 7f81109b7000-7f8110bb7000 ---p 00178000 fd:00 1974512 >> > /usr/lib64/libstdc++.so.6.0.22 >> > 7f8110bb7000-7f8110bc1000 r--p 00178000 fd:00 1974512 >> > /usr/lib64/libstdc++.so.6.0.22 >> > 7f8110bc1000-7f8110bc3000 rw-p 00182000 fd:00 1974512 >> > /usr/lib64/libstdc++.so.6.0.22 >> > 7f8110bc3000-7f8110bc7000 rw-p 00000000 00:00 0 >> > 7f8110bc7000-7f8110ccf000 r-xp 00000000 fd:00 1973951 >> > /usr/lib64/libm-2.24.so >> > 7f8110ccf000-7f8110ece000 ---p 00108000 fd:00 1973951 >> > /usr/lib64/libm-2.24.so >> > 7f8110ece000-7f8110ecf000 r--p 00107000 fd:00 1973951 >> > /usr/lib64/libm-2.24.so >> > 7f8110ecf000-7f8110ed0000 rw-p 00108000 fd:00 1973951 >> > /usr/lib64/libm-2.24.so >> > 7f8110ed0000-7f8110f60000 r-xp 00000000 fd:02 28457370 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so >> > 7f8110f60000-7f811115f000 ---p 00090000 fd:02 28457370 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so >> > 7f811115f000-7f8111160000 r--p 0008f000 fd:02 28457370 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so >> > 7f8111160000-7f8111161000 rw-p 00090000 fd:02 28457370 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so >> > 7f8111161000-7f8111169000 rw-p 00000000 00:00 0 >> > 7f8111169000-7f81111bf000 r-xp 00000000 fd:02 28457373 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so >> > 7f81111bf000-7f81113bf000 ---p 00056000 fd:02 28457373 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so >> > 7f81113bf000-7f81113c0000 r--p 00056000 fd:02 28457373 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so >> > 7f81113c0000-7f81113c1000 rw-p 00057000 fd:02 28457373 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so >> > 7f81113c1000-7f81114f9000 r-xp 00000000 fd:00 1973763 >> > /usr/lib64/libX11.so.6.3.0 >> > 7f81114f9000-7f81116f9000 ---p 00138000 fd:00 1973763 >> > /usr/lib64/libX11.so.6.3.0 >> > 7f81116f9000-7f81116fc000 r--p 00138000 fd:00 1973763 >> > /usr/lib64/libX11.so.6.3.0 >> > 7f81116fc000-7f8111700000 rw-p 0013b000 fd:00 1973763 >> > /usr/lib64/libX11.so.6.3.0 >> > 7f8111700000-7f8111718000 r-xp 00000000 fd:00 1974340 >> > /usr/lib64/libpthread-2.24.so >> > 7f8111718000-7f8111918000 ---p 00018000 fd:00 1974340 >> > /usr/lib64/libpthread-2.24.so >> > 7f8111918000-7f8111919000 r--p 00018000 fd:00 1974340 >> > /usr/lib64/libpthread-2.24.so >> > 7f8111919000-7f811191a000 rw-p 00019000 fd:00 1974340 >> > /usr/lib64/libpthread-2.24.so >> > 7f811191a000-7f811191e000 rw-p 00000000 00:00 0 >> > 7f811191e000-7f81124ff000 r-xp 00000000 fd:02 28457384 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ >> libopenblas_haswell-r0.3.8.dev.so >> > 7f81124ff000-7f81126ff000 ---p 00be1000 fd:02 28457384 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ >> libopenblas_haswell-r0.3.8.dev.so >> > 7f81126ff000-7f8112703000 r--p 00be1000 fd:02 28457384 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ >> libopenblas_haswell-r0.3.8.dev.so >> > 7f8112703000-7f811270d000 rw-p 00be5000 fd:02 28457384 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ >> libopenblas_haswell-r0.3.8.dev.so >> > 7f811270d000-7f811270e000 rw-p 00000000 00:00 0 >> > 7f811270e000-7f81147c2000 r-xp 00000000 fd:02 28457400 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 >> > 7f81147c2000-7f81149c1000 ---p 020b4000 fd:02 28457400 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 >> > 7f81149c1000-7f81149c7000 r--p 020b3000 fd:02 28457400 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 >> > 7f81149c7000-7f81149df000 rw-p 020b9000 fd:02 28457400 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 >> > 7f81149df000-7f8114bdf000 rw-p 00000000 00:00 0 >> > 7f8114bdf000-7f8115220000 r-xp 00000000 fd:02 28457416 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 >> > 7f8115220000-7f8115420000 ---p 00641000 fd:02 28457416 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 >> > 7f8115420000-7f8115421000 r--p 00641000 fd:02 28457416 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 >> > 7f8115421000-7f8115425000 rw-p 00642000 fd:02 28457416 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 >> > 7f8115425000-7f811544b000 r-xp 00000000 fd:00 1976838 >> > /usr/lib64/ld-2.24.so >> > 7f81154d3000-7f8115621000 rw-p 00000000 00:00 0 >> > 7f8115647000-7f811564a000 rw-p 00000000 00:00 0 >> > 7f811564a000-7f811564b000 r--p 00025000 fd:00 1976838 >> > /usr/lib64/ld-2.24.so >> > 7f811564b000-7f811564c000 rw-p 00026000 fd:00 1976838 >> > /usr/lib64/ld-2.24.so >> > 7f811564c000-7f811564d000 rw-p 00000000 00:00 0 >> > 7ffc79b33000-7ffc79b55000 rw-p 00000000 00:00 0 >> [stack] >> > 7ffc79b67000-7ffc79b6a000 r--p 00000000 00:00 0 >> [vvar] >> > 7ffc79b6a000-7ffc79b6c000 r-xp 00000000 00:00 0 >> [vdso] >> > ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 >> > [vsyscall] >> > >> > >> > I created and destroyed every vector, matrix and EPS in the loop to >> > make sure that there won't be any segmentation fault but still I am >> > getting some. Why am I getting this type of error? >> > >> > Thanks so much! >> > >> > Eda >> >>
