I get this error on normal runs on SUMMIT when I have an empty process. It does not happen all of the time but it does look to be deterministic. I'm debugging it now.
On Tue, Jul 2, 2019 at 10:41 PM Xiangdong via petsc-users < [email protected]> wrote: > Hello everyone, > > When I run the ex19 with cuda like this: > mpiexec -np 4 ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor > -mat_type aijcusparse -vec_type cuda -log_view > > it worked fine and produced correct results. > > However, when I tried to run this example with nvprof: > nvprof --profile-child-processes mpiexec -np 4 ./extest -da_refine 5 > -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda > -log_view > > I got errors like: > [3]PETSC ERROR: Error in external library > [3]PETSC ERROR: CUDA error 700 > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [3]PETSC ERROR: Petsc Release Version 3.11.2, unknown > [3]PETSC ERROR: ./ex19 on a arch-opt named new by eps Tue Jul 2 22:26:01 > 2019 > [3]PETSC ERROR: Configure options PETSC_ARCH=arch-opt --with-debugging=0 > --with-mpi-dir=/home/eps/MyLocal/mpi/mpich > --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl > --with-cuda-dir=/home/eps/MyLocal/cuda > --with-hypre-dir=/home/eps/MyLocal/hypre-2.15.1/hypre-install > --download-hdf5=1 > [3]PETSC ERROR: #1 VecSet_SeqCUDA() line 785 in > /home/eps/MyLocal/petsc/src/vec/vec/impls/seq/seqcuda/veccuda2.cu > [3]PETSC ERROR: #2 VecSet() line 547 in > /home/eps/MyLocal/petsc/src/vec/vec/interface/rvector.c > [3]PETSC ERROR: #3 VecCreate_MPICUDA() line 178 in > /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu > [3]PETSC ERROR: #4 VecSetType() line 51 in > /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c > [3]PETSC ERROR: #5 VecCreate_CUDA() line 192 in > /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu > [3]PETSC ERROR: #6 VecSetType() line 51 in > /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c > [3]PETSC ERROR: #7 MatCreateVecs() line 8996 in > /home/eps/MyLocal/petsc/src/mat/interface/matrix.c > [3]PETSC ERROR: #8 MatFDColoringCreate() line 482 in > /home/eps/MyLocal/petsc/src/mat/matfd/fdmatrix.c > [3]PETSC ERROR: #9 SNESComputeJacobian_DMDA() line 175 in > /home/eps/MyLocal/petsc/src/snes/utils/dmdasnes.c > [3]PETSC ERROR: #10 SNESComputeJacobian() line 2718 in > /home/eps/MyLocal/petsc/src/snes/interface/snes.c > [3]PETSC ERROR: #11 SNESSolve_NEWTONLS() line 222 in > /home/eps/MyLocal/petsc/src/snes/impls/ls/ls.c > [3]PETSC ERROR: #12 SNESSolve() line 4560 in > /home/eps/MyLocal/petsc/src/snes/interface/snes.c > [3]PETSC ERROR: #13 main() line 161 in /home/eps/MyCodes/petsctest/extest.c > > The full run log is attached. > > I am using NVIDIA-SMI 418.56 Driver Version: 418.56 CUDA > Version: 10.1. > > I do not know why it is okay without nvprof but crashed with nvprof. Any > suggestion to fix this? > > Thank you. > > Best, > Xiangdong > >
