Moving this to the users list (We can not talk about Crusher on public forums, but this is on Summit. I had to check this thread carefully!)
Treb is using hypre on Summit and getting this error: CUSPARSE ERROR (code = 11, insufficient resources) at csr_spgemm_device_cusparse.c:128 This is probably from Hypre's RAP. He has contacted OLCF, which seems like the right place to go, but does anyone have any ideas? Treb: You might ask Hypre also. We do actually have a fair amount of experience with hypre but hypre has more! Thanks, Mark On Fri, Jan 28, 2022 at 1:31 PM David Trebotich <[email protected]> wrote: > Thought I sent you this...will change the order of MatSetOption to see if > that helps > > I ran it and get that error which I have already sent a ticket to OLCF: > CUSPARSE ERROR (code = 11, insufficient resources) at > csr_spgemm_device_cusparse.c:128 > > Here's my petscrc > #do not use -mat_view with hypre-cuda if running on gpu > #-mat_view :A.m:ascii_matlab > -help > -proj_mac_pc_type hypre > -proj_mac_pc_hypre_type boomeramg > -proj_mac_pc_hypre_boomeramg_no_CF > -proj_mac_pc_hypre_boomeramg_agg_nl 0 > -proj_mac_pc_hypre_boomeramg_coarsen_type PMIS > -proj_mac_pc_hypre_boomeramg_interp_type ext+i > -proj_mac_pc_hypre_boomeramg_print_statistics > -proj_mac_pc_hypre_boomeramg_relax_type_all l1scaled-Jacobi > -proj_mac_ksp_type gmres > -proj_mac_ksp_max_it 50 > -proj_mac_ksp_rtol 1.e-12 > -proj_mac_ksp_atol 1.e-30 > -mat_type hypre > -use_gpu_aware_mpi 0 > -log_view > -history PETSc.history > -visc_ksp_rtol 1.e-12 > -visc_pc_type jacobi > -visc_ksp_type gmres > -visc_ksp_max_it 50 > -diff_ksp_rtol 1.e-6 > -diff_pc_type jacobi > -diff_ksp_max_it 50 > -proj_mac_ksp_converged_reason > -visc_ksp_converged_reason > -diff_ksp_converged_reason > -proj_mac_ksp_norm_type unpreconditioned > -diff_ksp_norm_type unpreconditioned > -visc_ksp_norm_type unpreconditioned > > And here's my code: > ierr = > MatSetSizes(m_mat,NN,NN,PETSC_DECIDE,PETSC_DECIDE);CHKERRQ(ierr); > ierr = MatSetBlockSize(m_mat,nc);CHKERRQ(ierr); > ierr = MatSetType(m_mat,MATAIJ);CHKERRQ(ierr); > // ierr = > MatSetOption(m_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) > ;CHKERRQ(ierr); > ierr = MatSetFromOptions( m_mat ); CHKERRQ(ierr); > ierr = MatSeqAIJSetPreallocation(m_mat,nnzrow, d_nnz);CHKERRQ(ierr); > ierr = MatMPIAIJSetPreallocation(m_mat,nnzrow, d_nnz, nnzrow/2, > o_nnz);CHKERRQ(ierr); > ierr = > MatSetOption(m_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) > ;CHKERRQ(ierr); > > #if defined(PETSC_HAVE_HYPRE) > ierr = MatHYPRESetPreallocation(m_mat,nnzrow, d_nnz, nnzrow/2, > o_nnz);CHKERRQ(ierr); > #endif > > On Wed, Jan 26, 2022 at 6:18 PM Mark Adams <[email protected]> wrote: > >> >> >> >> On Wed, Jan 26, 2022 at 7:43 PM David Trebotich <[email protected]> >> wrote: >> >>> Can you confirm with me on the settings in .petscrc for Summit with >>> -pc_type hypre? We were using >>> -mat_type aijcusparse >>> which worked a few months ago and now is not working. >>> >>> I don't know the difference between cusparse, aijcusparse and hypre as >>> -mat_type >>> >>> >> cusparse is not a matrix type. The other two are and they both should >> work. >> >> I tested the builds that I just sent in another email (eg, >> PETSC_DIR=/gpfs/alpine/world-shared/geo127/petsc/arch-summit-dbg-gcc-cuda >> PETSC_ARCH="") >> >> $ make PETSC_DIR=$PWD PETSC_ARCH=arch-summit-opt-gnu-hypre-cuda -f >> gmakefile test search='ksp_ksp_tutorials-ex55_hypre_device' >> Using MAKEFLAGS: -- search=ksp_ksp_tutorials-ex55_hypre_device >> PETSC_ARCH=arch-summit-opt-gnu-hypre-cuda >> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc2 >> CC arch-summit-opt-gnu-hypre-cuda/tests/ksp/ksp/tutorials/ex55.o >> CLINKER arch-summit-opt-gnu-hypre-cuda/tests/ksp/ksp/tutorials/ex55 >> TEST >> arch-summit-opt-gnu-hypre-cuda/tests/counts/ksp_ksp_tutorials-ex55_hypre_device.counts >> ok ksp_ksp_tutorials-ex55_hypre_device >> ok diff-ksp_ksp_tutorials-ex55_hypre_device >> >> So this work. In this file (attached) you will see an example of a >> construction a matrix that we have gone over before: >> >> /* create stiffness matrix */ >> ierr = MatCreate(comm,&Amat);CHKERRQ(ierr); >> ierr = MatSetSizes(Amat,m,m,M,M);CHKERRQ(ierr); >> ierr = MatSetType(Amat,MATAIJ);CHKERRQ(ierr); >> ierr = MatSetOption(Amat,MAT_SPD,PETSC_TRUE);CHKERRQ(ierr); >> ierr = MatSetFromOptions(Amat);CHKERRQ(ierr); >> ierr = MatSetBlockSize(Amat,2);CHKERRQ(ierr); >> ierr = MatSeqAIJSetPreallocation(Amat,18,NULL);CHKERRQ(ierr); >> ierr = MatMPIAIJSetPreallocation(Amat,18,NULL,18,NULL);CHKERRQ(ierr); >> #if defined(PETSC_HAVE_HYPRE) >> ierr = MatHYPRESetPreallocation(Amat,18,NULL,18,NULL);CHKERRQ(ierr); >> #endif >> >> At the end of the file you will what is executed with this test for >> "hypre_device": >> >> # command line options match GPU defaults >> test: >> suffix: hypre_device >> nsize: 4 >> requires: hypre !complex >> args: *-mat_type hypre* -ksp_view -ne 29 -alpha 1.e-3 -ksp_type cg >> *-pc_type >> hypre **-pc_hypre_type boomeramg *-ksp_monitor_short >> *-pc_hypre_boomeramg_relax_type_all >> l1scaled-Jacobi -pc_hypre_boomeramg_interp_type ext+i >> -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_no_CF* >> >> All you need is *-mat_type hypre and -pc_type hypre*. *You could also >> add these hypre arguments.* >> >> If this is not working please send me a description of the problem, like >> any error output on your screen and your petsc.history file. >> >> >> > > > -- > ---------------------- > David Trebotich > Lawrence Berkeley National Laboratory > Computational Research Division > Applied Numerical Algorithms Group > [email protected] > (510) 486-5984 office > (510) 384-6868 mobile >
