Dear Sir, I am introducing petsc into our fortran + openacc code, https://urldefense.us/v3/__https://gitlab.com/bsc_sod2d/sod2d_gitlab__;!!G_uCfscf7eWS!f0Y_9okAiFSBeUBOlNplMe6jOCPtdRHfhX2s_rz3N9kCK4OJGiGWGjFyRd3JOwojw7MeNEA-VJtgdn8wBbCmRmJTqdFC$ .
My final objective is to run AMG (Boomer from hypre) on the GPU. For the moment I am performing test on the CPU only. I run on Marenostrum-V. https://urldefense.us/v3/__https://www.bsc.es/supportkc/docs/MareNostrum5/intro/__;!!G_uCfscf7eWS!f0Y_9okAiFSBeUBOlNplMe6jOCPtdRHfhX2s_rz3N9kCK4OJGiGWGjFyRd3JOwojw7MeNEA-VJtgdn8wBbCmRraUJcUF$ I am compiling my code with NVHPC and the support team from BSC has compiled petsc + hypre for me. In the configuration they used -with-cuda. I have run petsc and it works correctly both in serial and parallel on the CPU. In my code I use call MatSetType(amat,MATAIJ,ierr). I understand this is teh expected behaviour for Petsc. That is, that even if one compiles with petsc cuda support one can run only on the GPU depending on what one sets in MatSetType. Could you confirm that this is the expected behaviour? Instead it seems that when petsc+hypre is used one needs to compile specific versions for CPU and GPU. When trying to use hypre through petsc if one has compiled petsc using -with-cuda the run fails. >From what I have understood this in not the expected behavior. could you >confirm this? Depending of what options I give, the error is different If I use -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_coarsen_type hmis -pc_hypre_boomeramg_interp_type ext+i -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_relax_type_coarse SOR/Jacobi -pc_hypre_boomeramg_grid_sweeps_coarse 1 -pc_hypre_boomeramg_strong_threshold 0.25 I get [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!f0Y_9okAiFSBeUBOlNplMe6jOCPtdRHfhX2s_rz3N9kCK4OJGiGWGjFyRd3JOwojw7MeNEA-VJtgdn8wBbCmRt2ImXnV$ and https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!f0Y_9okAiFSBeUBOlNplMe6jOCPtdRHfhX2s_rz3N9kCK4OJGiGWGjFyRd3JOwojw7MeNEA-VJtgdn8wBbCmRjGOi2o3$ [0]PETSC ERROR: or try https://urldefense.us/v3/__https://docs.nvidia.com/cuda/cuda-memcheck/index.html__;!!G_uCfscf7eWS!f0Y_9okAiFSBeUBOlNplMe6jOCPtdRHfhX2s_rz3N9kCK4OJGiGWGjFyRd3JOwojw7MeNEA-VJtgdn8wBbCmRqNF46dV$ on NVIDIA CUDA systems to find memory corruption errors [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: The line numbers in the error traceback are not always exact. [0]PETSC ERROR: #1 jac->setup() [0]PETSC ERROR: #2 PCSetUp_HYPRE() at /gpfs/apps/MN5/ACC/PETSC/SRC/petsc-v3.21.0_hypre-debug/src/ksp/pc/impls/hypre/hypre.c:422 [0]PETSC ERROR: #3 PCSetUp() at /gpfs/apps/MN5/ACC/PETSC/SRC/petsc-v3.21.0_hypre-debug/src/ksp/pc/interface/precon.c:1079 [0]PETSC ERROR: #4 KSPSetUp() at /gpfs/apps/MN5/ACC/PETSC/SRC/petsc-v3.21.0_hypre-debug/src/ksp/ksp/interface/itfunc.c:415 -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI COMMUNICATOR 3 SPLIT FROM 0 with errorcode 59. Which does not help much. Instead if I use only -pc_type hypre I get [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: HYPRE_MEMORY_DEVICE expects a device vector. You need to enable PETSc device support, for example, in some cases, -vec_type cuda This helped my realise that hypre was trying to use the GPU despite in my code Both runs are succesfull when compiling without -with-cuda Best, Herbert Owen Senior Researcher, Dpt. Computer Applications in Science and Engineering Barcelona Supercomputing Center (BSC-CNS) Tel: +34 93 413 4038 Skype: herbert.owen https://urldefense.us/v3/__https://scholar.google.es/citations?user=qe5O2IYAAAAJ&hl=en__;!!G_uCfscf7eWS!f0Y_9okAiFSBeUBOlNplMe6jOCPtdRHfhX2s_rz3N9kCK4OJGiGWGjFyRd3JOwojw7MeNEA-VJtgdn8wBbCmRkKVuPTJ$
