On Fri, Jan 5, 2024 at 9:33 AM Edoardo Centofanti < edoardo.centofant...@universitadipavia.it> wrote:
> > > Il giorno ven 5 gen 2024 alle ore 16:21 Junchao Zhang < > junchao.zh...@gmail.com> ha scritto: > >> >> >> >> On Fri, Jan 5, 2024 at 4:40 AM Edoardo Centofanti < >> edoardo.centofant...@universitadipavia.it> wrote: >> >>> Dear all, >>> >>> I have a code running both on GPU and CPU. This code has both cuda >>> kernels and calls to PETSc KSP and related PC. In particular, I am trying >>> to perform tests with Hypre BoomerAMG both on CPU and GPU. In order to do >>> that, on CPU I am running the code with -mat_type mpiaij and -vec_type mpi, >>> while on GPU I am using respectively aijcusparse and cuda. >>> >>> The configuration ran for PETSc (version is 3.20) is >>> ./configure PETSC_ARCH=arch-linux-cuda --with-cuda --download-mumps >>> --download-hypre --with-debugging=0 --download-scalapack >>> --download-parmetis --download-metis --download-fblaslapack=1 >>> --download-mpich --download-make --download-cmake >>> >>> My problem is that when I try to run my code on GPU it works well, while >>> on CPU with mat_type mpiaij and -vec_type mpi it works regularly until the >>> call to Hypre, then freezes (I have to kill it myself), while with GAMG it >>> works on CPU with the same configuration (and the same code, just PC is >>> changed). >>> >> Probably because if hypre is configured with GPU support, even petsc is >> running on CPU (with -mat_type mpiaij and -vec_type mpi), hypre is still >> trying to run on GPU, causing the hanging problem. The petsc/hypre >> interface is not able to support both CPU/GPU with the same build. >> >> >>> On another machine running PETSc version 3.17 everything worked smoothly >>> with the same code and the same configuration, also on Hypre. >>> >> I guess with petsc-3.17, you also used a different hypre version >> (downloaded by petsc). >> > Yes, this is exactly what I concluded after a bit of debugging and > recompiling. In the new version it seems that CPU/GPU with the same build > is not supported anymore. Since the CUDA kernels I have in my code do not > interfere with Hypre, the only workaround I found was removing kernels from > my code and performing an only-CPU run. It slows down a bit overall, but > nothing dramatic. Anyway, in another test with version 3.18 I encountered > the same problem, but the program crashes and throws "Invalid options" > error, while in 3.20 it just hangs. > Improving the petsc/hypre interface to support both CPU/GPU is WIP. > Thanks, > Edoardo > >> >> >> >>> Do you have any insights on what is happening? >>> >>> Best regards, >>> Edoardo >>> >>> >>