Thank you very much for your help! I tried recompiling dealii disabling cuda for HYPRE, but that was not enough. Then I tried disabling cuda for petsc and slepc and, in that case, it works perfectly. Thank you again!
Best, Stefano Il giorno martedì 27 dicembre 2022 alle 23:09:43 UTC+1 [email protected] ha scritto: > Stefano, > > Currently in deal.II we always assume that the PETSc vector is on the CPU > but I don't know what PETSc does internally when it is compiled with GPU > support. I don't think that the deal.II developers who use PETSc, use GPU > so it's not tested. If you can, compile deal.II with GPU support but PETSc > without it. > > Best, > > Bruno > > On Tuesday, December 27, 2022 at 3:40:56 PM UTC-5 [email protected] > wrote: > >> Hello everyone, >> I was interested in doing some experiments starting from step 64 and >> therefore I reinstalled Deal II with cuda support (using Spack). >> Unfortunately, now step 64 works perfectly but many other examples that >> do not use cuda but need PETSc don't. >> For example, if I run step-40 I get the following error: >> >> mpirun -np 1 step-40.debug >> Running with PETSc on 1 MPI rank(s)... >> Cycle 0: >> Number of active cells: 1024 >> Number of degrees of freedom: 4225 >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Invalid argument >> [0]PETSC ERROR: HYPRE_MEMORY_DEVICE expects a device vector. You need to >> enable PETSc device support, for example, in some cases, -vec_type cuda >> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.17.1, Apr 28, 2022 >> [0]PETSC ERROR: step-40.debug on a named Zorn by step Tue Dec 27 21:36:21 >> 2022 >> [0]PETSC ERROR: Configure options >> --prefix=/home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/petsc-3.17.1-wzfeckr7omoetuquumz77nloe7u6mpku >> >> --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 >> --wit >> h-cc=/home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/openmpi-4.1.4-3ea3p3wv5a53ohy3fxui7yygr2m4nlrb/bin/mpicc >> >> --with-cxx=/home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11. >> 3.0/openmpi-4.1.4-3ea3p3wv5a53ohy3fxui7yygr2m4nlrb/bin/mpic++ >> --with-fc=/home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/openmpi-4.1.4-3ea3p3wv5a53ohy3fxui7yygr2m4nlrb/bin/mpif90 >> >> --with-precision=double --with-scala >> r-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 >> --with-64-bit-indices=0 >> --with-blaslapack-lib=/home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/openblas-0.3.21-ozedcu5tn56y23uoxrx6zjf2tnhncvy >> g/lib/libopenblas.so --with-x=0 --with-clanguage=C --with-cuda=1 >> --with-cuda-dir=/home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/cuda-11.8.0-xzhbnrb3zgd6zsrdjaa4an3saa7dkkf3 >> >> --with-hip=0 --with-metis=1 --with-metis >> -include=/home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/metis-5.1.0-imnzuza4aintah4v6c3e3nciaux2opc7/include >> >> --with-metis-lib=/home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/g >> cc-11.3.0/metis-5.1.0-imnzuza4aintah4v6c3e3nciaux2opc7/lib/libmetis.so >> --with-hypre=1 >> --with-hypre-include=/home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/hypre-2.27.0-luxpg32fzxo2vgcnb5iq53lbrvxberiw/include >> >> --wit >> h-hypre-lib=/home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/hypre-2.27.0-luxpg32fzxo2vgcnb5iq53lbrvxberiw/lib/libHYPRE.so >> >> --with-parmetis=1 >> --with-parmetis-include=/home/step/environments/dealii-9.4_gcc-11.3/spack/ >> opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/parmetis-4.0.3-uckybgzpowx5rdwtskefzomdapasgpoc/include >> >> --with-parmetis-lib=/home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/parmetis-4.0.3-uckybgzpowx5rdwtskefzomdapas >> gpoc/lib/libparmetis.so --with-kokkos=0 --with-kokkos-kernels=0 >> --with-superlu_dist=1 >> --with-superlu_dist-include=/home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/superlu-dist-8.1.2-qlmlkwfer4h53d7vis7lpjtcu77bp3yn/ >> include >> --with-superlu_dist-lib=/home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/superlu-dist-8.1.2-qlmlkwfer4h53d7vis7lpjtcu77bp3yn/lib/libsuperlu_dist.so >> >> --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with >> -hdf5-include=/home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/hdf5-1.10.7-xjt3eeewg3eihjdvjnzdavfpuy5mmpjr/include >> >> --with-hdf5-lib="/home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skyl >> ake/gcc-11.3.0/hdf5-1.10.7-xjt3eeewg3eihjdvjnzdavfpuy5mmpjr/lib/libhdf5_hl_fortran.so >> >> /home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/hdf5-1.10.7-xjt3eeewg3eihjdvjnzdavfpuy5mmpjr/lib/libhdf5_hl_f90cstub.so >> >> /home/st >> ep/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/hdf5-1.10.7-xjt3eeewg3eihjdvjnzdavfpuy5mmpjr/lib/libhdf5_hl.so >> >> /home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/hdf5-1.10.7-xj >> t3eeewg3eihjdvjnzdavfpuy5mmpjr/lib/libhdf5_fortran.so >> /home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/hdf5-1.10.7-xjt3eeewg3eihjdvjnzdavfpuy5mmpjr/lib/libhdf5_f90cstub.so >> >> /home/step/environments/dealii-9.4_gcc-11.3 >> /spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/hdf5-1.10.7-xjt3eeewg3eihjdvjnzdavfpuy5mmpjr/lib/libhdf5.so" >> >> --with-zlib=1 >> --with-zlib-include=/home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/zlib-1.2.13-rcig5 >> et6juu4y42r6v5z6fvkakzygdox/include >> --with-zlib-lib=/home/step/environments/dealii-9.4_gcc-11.3/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/zlib-1.2.13-rcig5et6juu4y42r6v5z6fvkakzygdox/lib/libz.so >> >> --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-val >> grind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 >> --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 >> --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 >> --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with >> -scalapack=0 --with-strumpack=0 --with-mmg=0 --with-parmmg=0 >> --with-tetgen=0 --with-cuda-arch=61 >> [0]PETSC ERROR: #1 VecGetArrayForHYPRE() at >> /tmp/step/spack-stage/spack-stage-petsc-3.17.1-wzfeckr7omoetuquumz77nloe7u6mpku/spack-src/src/vec/vec/impls/hypre/vhyp.c:94 >> >> >> [0]PETSC ERROR: #2 VecHYPRE_IJVectorPushVecRead() at >> /tmp/step/spack-stage/spack-stage-petsc-3.17.1-wzfeckr7omoetuquumz77nloe7u6mpku/spack-src/src/vec/vec/impls/hypre/vhyp.c:137 >> >> >> [0]PETSC ERROR: #3 PCApply_HYPRE() at >> /tmp/step/spack-stage/spack-stage-petsc-3.17.1-wzfeckr7omoetuquumz77nloe7u6mpku/spack-src/src/ksp/pc/impls/hypre/hypre.c:434 >> >> >> [0]PETSC ERROR: #4 PCApply() at >> /tmp/step/spack-stage/spack-stage-petsc-3.17.1-wzfeckr7omoetuquumz77nloe7u6mpku/spack-src/src/ksp/pc/interface/precon.c:432 >> >> >> [0]PETSC ERROR: #5 KSP_PCApply() at >> /tmp/step/spack-stage/spack-stage-petsc-3.17.1-wzfeckr7omoetuquumz77nloe7u6mpku/spack-src/include/petsc/private/kspimpl.h:376 >> >> >> [0]PETSC ERROR: #6 KSPSolve_CG() at >> /tmp/step/spack-stage/spack-stage-petsc-3.17.1-wzfeckr7omoetuquumz77nloe7u6mpku/spack-src/src/ksp/ksp/impls/cg/cg.c:135 >> >> >> [0]PETSC ERROR: #7 KSPSolve_Private() at >> /tmp/step/spack-stage/spack-stage-petsc-3.17.1-wzfeckr7omoetuquumz77nloe7u6mpku/spack-src/src/ksp/ksp/interface/itfunc.c:902 >> >> >> [0]PETSC ERROR: #8 KSPSolve() at >> /tmp/step/spack-stage/spack-stage-petsc-3.17.1-wzfeckr7omoetuquumz77nloe7u6mpku/spack-src/src/ksp/ksp/interface/itfunc.c:1078 >> >> >> --------------------------------------------------------- >> TimerOutput objects finalize timed values printed to the >> screen by communicating over MPI in their destructors. >> Since an exception is currently uncaught, this >> synchronization (and subsequent output) will be skipped >> to avoid a possible deadlock. >> --------------------------------------------------------- >> >> >> ---------------------------------------------------- >> Exception on processing: >> >> -------------------------------------------------------- >> An error occurred in line <138> of file >> </home/step/environments/dealii-9.4_gcc-11.3/dealii-9.4.1-source/source/lac/petsc_solver.cc> >> >> in function >> void dealii::PETScWrappers::SolverBase::solve(const >> dealii::PETScWrappers::MatrixBase&, dealii::PETScWrappers::VectorBase&, >> const dealii::PETScWrappers::VectorBase&, const >> dealii::PETScWrappers::PreconditionBase&) >> The violated condition was: >> ierr == 0 >> Additional information: >> deal.II encountered an error while calling a PETSc function. >> The description of the error provided by PETSc is "Invalid argument". >> The numerical value of the original error code is 62. >> >> Stacktrace: >> ----------- >> #0 >> /home/step/environments/dealii-9.4_gcc-11.3/dealii-9.4.1-build/lib/libdeal_II.g.so.9.4.1: >> >> dealii::PETScWrappers::SolverBase::solve(dealii::PETScWrappers::MatrixBase >> const&, dealii::PETScWrappers::VectorBase&, >> dealii::PETScWrappers::VectorBase const&, dealii:: >> PETScWrappers::PreconditionBase const&) >> #1 step-40.debug: Step40::LaplaceProblem<2>::solve() >> #2 step-40.debug: Step40::LaplaceProblem<2>::run() >> #3 step-40.debug: main >> -------------------------------------------------------- >> >> Aborting! >> ---------------------------------------------------- >> -------------------------------------------------------------------------- >> >> Primary job terminated normally, but 1 process returned >> a non-zero exit code. Per user-direction, the job has been aborted. >> -------------------------------------------------------------------------- >> >> -------------------------------------------------------------------------- >> >> mpirun detected that one or more processes exited with non-zero status, >> thus causing >> the job to be terminated. The first process to do so was: >> >> Process name: [[46368,1],0] >> Exit code: 1 >> -------------------------------------------------------------------------- >> >> I understand that probably the error is related to the fact that I never >> specified to PETSc that my vector should stay on my CPU, but I don't know >> how to do that. In PETSc there is a function that I could use, but I can >> not find an equivalent for the Deal II wrappers. >> I am using Deal II 9.4.1, PETSc 3.17.1 and Hypre 2.27.0. Thank you! >> >> Stefano >> > -- The deal.II project is located at http://www.dealii.org/ For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en --- You received this message because you are subscribed to the Google Groups "deal.II User Group" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/79c93925-af5d-4cba-b33c-1f73a7aac108n%40googlegroups.com.
