valgrind first and I think NVIDIA has a "valgrind" for GPUs; then start in debugger.
You could also back off on the aggressive optimization and see if that changes the behavior. > On Apr 16, 2021, at 9:24 AM, Matthew Knepley <[email protected]> wrote: > > Can you get a stack trace? > > Matt > > On Fri, Apr 16, 2021 at 10:19 AM Mark Adams <[email protected] > <mailto:[email protected]>> wrote: > That seems to have changed it. No stack trace. > > srun -G 1 -c 2 -n 1 ./ex2 -petscspace_degree 3 -ex2_test_type spitzer > -dm_landau_Ez 0 -dm_landau_ion_masses .01 -dm_landau_ion_charges 1 > -dm_landau_thermal_temps 2,1 -dm_landau_n 1,1 -ts_type beuler > -ts_exact_final_time stepover -ts_max_steps 2 -ts_dt 1 -ts_monitor > -snes_monitor -snes_max_it 25 -snes_rtol 1.e-14 -snes_stol 1.e-14 -pc_type lu > -ksp_type preonly -dm_landau_type p4est -dm_landau_amr_levels_max 13 > -dm_landau_amr_post_refine 1 -dm_preallocate_only -ex2_plot_dt .0001 > -dm_landau_device_type cuda -dm_mat_type aijcusparse -dm_vec_type cuda > -display :0.0 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 4 Illegal instruction: Likely due to > memory corruption > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > <https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind> > [0]PETSC ERROR: or try http://valgrind.org <http://valgrind.org/> on > GNU/linux and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > <https://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.15.0-205-g2283897782 GIT > Date: 2021-04-15 09:38:17 -0400 > [0]PETSC ERROR: > /global/u2/m/madams/petsc/src/ts/utils/dmplexlandau/tutorials/./ex2 on a > arch-cori-gpu80-opt-kokkos-gcc named cgpu19 by madams Fri Apr 16 07:16:28 2021 > [0]PETSC ERROR: Configure options > --with-mpi-dir=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc > --with-cuda-dir=/usr/common/software/sles15_cgpu/cuda/11.1.1 CFLAGS=" -g > -DPETSC_HAVE_CUDA_ATOMIC" CXXFLAGS=" -g -DPETSC_HAVE_CUDA_ATOMIC" FFLAGS=" > -g " COPTFLAGS=" -O" CXXOPTFLAGS=" -O" FOPTFLAGS=" -O" > --CUDAFLAGS="-arch=sm_80 -Xcompiler -rdynamic -lineinfo > -DPETSC_HAVE_CUDA_ATOMIC -g" --CUDAOPTFLAGS=-O3 --download-fblaslapack=1 > --with-debugging=0 --download-kokkos --download-kokkos-kernels > --with-kokkos-cuda-arch=AMPERE80 --with-kokkos-kernels-tpl=0 --with-make-np=8 > --with-ctable=0 --with-mpiexec="srun -G 1 -c 2" --with-batch=0 > PETSC_ARCH=arch-cori-gpu80-opt-kokkos-gcc --with-cuda=1 --download-p4est=1 > --with-zlib=1 > [0]PETSC ERROR: #1 User provided function() at unknown file:0 > > On Fri, Apr 16, 2021 at 10:02 AM Matthew Knepley <[email protected] > <mailto:[email protected]>> wrote: > On Fri, Apr 16, 2021 at 9:58 AM Mark Adams <[email protected] > <mailto:[email protected]>> wrote: > I am running on a new AMPERE80 node at NERSc and get this error message. I > put this in totalview but did not get anything useful. > > Any ideas? > > Maybe getenv() is failing? > > You can shutoff this behavior using > > -display :0.0 > > Matt > > cgpu19:228520:0:228520] Caught signal 4 (Illegal instruction: illegal operand) > ==== backtrace (tid: 228520) ==== > 0 > /usr/common/software/sles15_cgpu/ucx/1.8.1/lib/libucs.so.0(ucs_handle_error+0x2e4) > [0x2aab2e9a2ac4] > 1 /usr/common/software/sles15_cgpu/ucx/1.8.1/lib/libucs.so.0(+0x21cc4) > [0x2aab2e9a2cc4] > 2 /usr/common/software/sles15_cgpu/ucx/1.8.1/lib/libucs.so.0(+0x21d33) > [0x2aab2e9a2d33] > 3 > /global/homes/m/madams/petsc/arch-cori-gpu80-opt-kokkos-gcc/lib/libpetsc.so.3.015(PetscSetDisplay+0x152) > [0x2aaaaaf19ab1] > 4 > /global/homes/m/madams/petsc/arch-cori-gpu80-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x21c7bc) > [0x2aaaaaeef7bc] > 5 > /global/homes/m/madams/petsc/arch-cori-gpu80-opt-kokkos-gcc/lib/libpetsc.so.3.015(PetscInitialize+0x449) > [0x2aaaaaef5278] > 6 /global/u2/m/madams/petsc/src/ts/utils/dmplexlandau/tutorials/./ex2() > [0x405b62] > 7 /lib64/libc.so.6(__libc_start_main+0xea) [0x2aab1344df8a] > 8 /global/u2/m/madams/petsc/src/ts/utils/dmplexlandau/tutorials/./ex2() > [0x4026aa] > ================================= > srun: error: cgpu19: task 0: Illegal instruction > srun: Terminating job step 1821681.10 > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/> > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
