Re: [petsc-users] Using PETSc GPU backend

Barry, I tried again today on Perlmutter and running on multiple GPU nodes worked. Likely, I had messed up something the other day. Also, I was able to have multiple MPI tasks on a GPU using Nvidia MPS. The petsc output shows the number of MPI tasks: KSP Object: 32 MPI processes Can petsc

Re: [petsc-users] CUDA error trying to run a job with two mpi processes and 1 GPU

Before digging into the details, could you try to run src/ksp/ksp/tests/ex60.c to make sure the environment is ok. The comment at the end shows how to run it test: requires: cuda suffix: 1_cuda nsize: 4 args: -ksp_view -mat_type aijcusparse

Re: [petsc-users] error related to 'valgrind' when using MatView

New error checking to prevent this confusion in the future: https://gitlab.com/petsc/petsc/-/merge_requests/6804 > On Aug 10, 2023, at 6:54 AM, Matthew Knepley wrote: > > On Thu, Aug 10, 2023 at 2:30 AM maitri ksh > wrote: >> I am unable to understand what

Re: [petsc-users] 32-bit vs 64-bit GPU support

On Fri, 11 Aug 2023, Jed Brown wrote: > Jacob Faibussowitsch writes: > > > More generally, it would be interesting to know the breakdown of installed > > CUDA versions for users. Unlike compilers etc, I suspect that cluster > > admins (and those running on local machines) are much more likely

Re: [petsc-users] 32-bit vs 64-bit GPU support

Jacob Faibussowitsch writes: > More generally, it would be interesting to know the breakdown of installed > CUDA versions for users. Unlike compilers etc, I suspect that cluster admins > (and those running on local machines) are much more likely to be updating > their CUDA toolkits to the

Re: [petsc-users] 32-bit vs 64-bit GPU support

> We should support it, but it still seems hypothetical and not urgent. FWIW, cuBLAS only just added 64-bit int support with CUDA 12 (naturally, with a completely separate API). More generally, it would be interesting to know the breakdown of installed CUDA versions for users. Unlike

Re: [petsc-users] 32-bit vs 64-bit GPU support

Rohan Yadav writes: > With modern GPU sizes, for example A100's with 80GB of memory, a vector of > length 2^31 is not that much memory -- one could conceivably run a CG solve > with local vectors > 2^31. Yeah, each vector would be 8 GB (single precision) or 16 GB (double). You can't store a

Re: [petsc-users] CUDA error trying to run a job with two mpi processes and 1 GPU

Marcos, We do not have good petsc/gpu documentation, but see https://petsc.org/main/faq/#doc-faq-gpuhowto, and also search "requires: cuda" in petsc tests and you will find examples using GPU. For the Fortran compile errors, attach your configure.log and Satish (Cc'ed) or others should know

Re: [petsc-users] 32-bit vs 64-bit GPU support

>We do not currently have any code for using 64 bit integer sizes on the GPUs. Thank you, just wanted confirmation. >Given the current memory available on GPUs is 64 bit integer support needed? I think even a single vector of length 2^31 will use up most of the GPU's memory? Are the

Re: [petsc-users] 32-bit vs 64-bit GPU support

Rohan, You could try the petsc/kokkos backend. I have not tested it, but I guess it should handle 64 bit CUDA index types. I guess the petsc/cuda 32-bit limit came from old CUDA versions where only 32-bit indices were supported such that the original developers hardwired the type to

Re: [petsc-users] CUDA error trying to run a job with two mpi processes and 1 GPU

Hi, Macros, I saw MatSetPreallocationCOO_MPIAIJCUSPARSE_Basic() in the error stack. We recently refactored the COO code and got rid of that function. So could you try petsc/main? We map MPI processes to GPUs in a round-robin fashion. We query the number of visible CUDA devices (g), and assign

Re: [petsc-users] 32-bit vs 64-bit GPU support

We do not currently have any code for using 64 bit integer sizes on the GPUs. Given the current memory available on GPUs is 64 bit integer support needed? I think even a single vector of length 2^31 will use up most of the GPU's memory? Are the practical, not synthetic, situations that

Re: [petsc-users] CUDA error trying to run a job with two mpi processes and 1 GPU

Hi Junchao, thank you for replying. I compiled petsc in debug mode and this is what I get for the case: terminate called after throwing an instance of 'thrust::system::system_error' what(): merge_sort: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered

[petsc-users] 32-bit vs 64-bit GPU support

Hi, I was wondering what the official status of 64-bit integer support in the PETSc GPU backend is (specifically CUDA). This question comes from the result of benchmarking some PETSc code and looking at some sources. In particular, I found that PETSc's call to cuSPARSE SpMV seems to always be

Re: [petsc-users] CUDA error trying to run a job with two mpi processes and 1 GPU

Hi, Marcos, Could you build petsc in debug mode and then copy and paste the whole error stack message? Thanks --Junchao Zhang On Thu, Aug 10, 2023 at 5:51 PM Vanella, Marcos (Fed) via petsc-users < petsc-users@mcs.anl.gov> wrote: > Hi, I'm trying to run a parallel matrix vector build and