We absolutely should have a MatGetDiagonal_SeqAIJCUSPARSE(). It's somewhat embarrassing that we don't provide this.
I have found some potential code at https://urldefense.us/v3/__https://stackoverflow.com/questions/60311408/how-to-get-the-diagonal-of-a-sparse-matrix-in-cusparse__;!!G_uCfscf7eWS!cOVqDE53HIAwLx-ILxDMFmg5MAzIZUHjcusb9V2nHf3y_wMh6ZB0ApPcybv2hEBDJlag4Q1oiMX3d-f6nEAtnSI$ Barry > On Jul 28, 2025, at 11:43 AM, Junchao Zhang <junchao.zh...@gmail.com> wrote: > > Yes, MatGetDiagonal_SeqAIJCUSPARSE hasn't been implemented. petsc/cuda and > petsc/kokkos backends are separate code. > If petsc/kokkos meet your needs, then just use them. For petsc users, we > hope it will be just a difference of extra --download-kokkos > --download-kokkos-kernels in configuration. > > --Junchao Zhang > > > On Mon, Jul 28, 2025 at 2:51 AM LEDAC Pierre <pierre.le...@cea.fr > <mailto:pierre.le...@cea.fr>> wrote: >> Hello all, >> >> >> >> We are solving with PETSc a linear system updated every time step (constant >> stencil but coefficients changing). >> >> >> >> The matrix is preallocated once with MatSetPreallocationCOO() then filled >> each time step with MatSetValuesCOO() and we use device pointers for coo_i, >> coo_j, and coefficients values. >> >> >> >> It is working fine with a GMRES Ksp solver and PC Jacobi but we are >> surprised to see that every time step, during PCSetUp, MatGetDiagonal_SeqAIJ >> is called whereas the matrix is on the device. Looking at the API, it seems >> there is no MatGetDiagonal_SeqAIJCUSPARSE() but a >> MatGetDiagonal_SeqAIJKOKKOS(). >> >> >> >> Does it mean we should use Kokkos backend in PETSc to have Jacobi >> preconditioner built directly on device ? Or I am doing something wrong ? >> >> NB: Gmres is running well on device. >> >> >> >> I could use -ksp_reuse_preconditioner to avoid Jacobi being recreated each >> solve on host but it increases significantly the number of iterations. >> >> >> >> Thanks, >> >> >> >> <pastedImage.png> >> >> >> >> >> >> Pierre LEDAC >> Commissariat à l’énergie atomique et aux énergies alternatives >> Centre de SACLAY >> DES/ISAS/DM2S/SGLS/LCAN >> Bâtiment 451 – point courrier n°41 >> F-91191 Gif-sur-Yvette >> +33 1 69 08 04 03 >> +33 6 83 42 05 79
