On Fri, Jan 28, 2022 at 3:27 PM Mark Adams <[email protected]> wrote: > (Junchao), Ruipeng said this was an OOM error and suggested trying the > in-house with > > SpGEMM: HYPRE_SetSpGemmUseCusparse(FALSE); > > Should I clone a hypre argument to make a -pc_hypre_use_tpl_spgemm or > -pc_hypre_use_cusparse_spgemm ? > Better use the same words as hypre, e.g., -pc_hypre_set_spgemm_use_cusparse
> > > On Fri, Jan 28, 2022 at 1:42 PM Mark Adams <[email protected]> wrote: > >> Moving this to the users list (We can not talk about Crusher on public >> forums, but this is on Summit. I had to check this thread carefully!) >> >> Treb is using hypre on Summit and getting this error: >> >> CUSPARSE ERROR (code = 11, insufficient resources) at >> csr_spgemm_device_cusparse.c:128 >> >> This is probably from Hypre's RAP. >> >> He has contacted OLCF, which seems like the right place to go, but does >> anyone have any ideas? >> >> Treb: You might ask Hypre also. We do actually have a fair amount of >> experience with hypre but hypre has more! >> >> Thanks, >> Mark >> >> >> >> >> >> On Fri, Jan 28, 2022 at 1:31 PM David Trebotich <[email protected]> >> wrote: >> >>> Thought I sent you this...will change the order of MatSetOption to see >>> if that helps >>> >>> I ran it and get that error which I have already sent a ticket to OLCF: >>> CUSPARSE ERROR (code = 11, insufficient resources) at >>> csr_spgemm_device_cusparse.c:128 >>> >>> Here's my petscrc >>> #do not use -mat_view with hypre-cuda if running on gpu >>> #-mat_view :A.m:ascii_matlab >>> -help >>> -proj_mac_pc_type hypre >>> -proj_mac_pc_hypre_type boomeramg >>> -proj_mac_pc_hypre_boomeramg_no_CF >>> -proj_mac_pc_hypre_boomeramg_agg_nl 0 >>> -proj_mac_pc_hypre_boomeramg_coarsen_type PMIS >>> -proj_mac_pc_hypre_boomeramg_interp_type ext+i >>> -proj_mac_pc_hypre_boomeramg_print_statistics >>> -proj_mac_pc_hypre_boomeramg_relax_type_all l1scaled-Jacobi >>> -proj_mac_ksp_type gmres >>> -proj_mac_ksp_max_it 50 >>> -proj_mac_ksp_rtol 1.e-12 >>> -proj_mac_ksp_atol 1.e-30 >>> -mat_type hypre >>> -use_gpu_aware_mpi 0 >>> -log_view >>> -history PETSc.history >>> -visc_ksp_rtol 1.e-12 >>> -visc_pc_type jacobi >>> -visc_ksp_type gmres >>> -visc_ksp_max_it 50 >>> -diff_ksp_rtol 1.e-6 >>> -diff_pc_type jacobi >>> -diff_ksp_max_it 50 >>> -proj_mac_ksp_converged_reason >>> -visc_ksp_converged_reason >>> -diff_ksp_converged_reason >>> -proj_mac_ksp_norm_type unpreconditioned >>> -diff_ksp_norm_type unpreconditioned >>> -visc_ksp_norm_type unpreconditioned >>> >>> And here's my code: >>> ierr = >>> MatSetSizes(m_mat,NN,NN,PETSC_DECIDE,PETSC_DECIDE);CHKERRQ(ierr); >>> ierr = MatSetBlockSize(m_mat,nc);CHKERRQ(ierr); >>> ierr = MatSetType(m_mat,MATAIJ);CHKERRQ(ierr); >>> // ierr = >>> MatSetOption(m_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) >>> ;CHKERRQ(ierr); >>> ierr = MatSetFromOptions( m_mat ); CHKERRQ(ierr); >>> ierr = MatSeqAIJSetPreallocation(m_mat,nnzrow, >>> d_nnz);CHKERRQ(ierr); >>> ierr = MatMPIAIJSetPreallocation(m_mat,nnzrow, d_nnz, nnzrow/2, >>> o_nnz);CHKERRQ(ierr); >>> ierr = >>> MatSetOption(m_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) >>> ;CHKERRQ(ierr); >>> >>> #if defined(PETSC_HAVE_HYPRE) >>> ierr = MatHYPRESetPreallocation(m_mat,nnzrow, d_nnz, nnzrow/2, >>> o_nnz);CHKERRQ(ierr); >>> #endif >>> >>> On Wed, Jan 26, 2022 at 6:18 PM Mark Adams <[email protected]> wrote: >>> >>>> >>>> >>>> >>>> On Wed, Jan 26, 2022 at 7:43 PM David Trebotich <[email protected]> >>>> wrote: >>>> >>>>> Can you confirm with me on the settings in .petscrc for Summit with >>>>> -pc_type hypre? We were using >>>>> -mat_type aijcusparse >>>>> which worked a few months ago and now is not working. >>>>> >>>>> I don't know the difference between cusparse, aijcusparse and hypre as >>>>> -mat_type >>>>> >>>>> >>>> cusparse is not a matrix type. The other two are and they both should >>>> work. >>>> >>>> I tested the builds that I just sent in another email (eg, >>>> PETSC_DIR=/gpfs/alpine/world-shared/geo127/petsc/arch-summit-dbg-gcc-cuda >>>> PETSC_ARCH="") >>>> >>>> $ make PETSC_DIR=$PWD PETSC_ARCH=arch-summit-opt-gnu-hypre-cuda -f >>>> gmakefile test search='ksp_ksp_tutorials-ex55_hypre_device' >>>> Using MAKEFLAGS: -- search=ksp_ksp_tutorials-ex55_hypre_device >>>> PETSC_ARCH=arch-summit-opt-gnu-hypre-cuda >>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc2 >>>> CC >>>> arch-summit-opt-gnu-hypre-cuda/tests/ksp/ksp/tutorials/ex55.o >>>> CLINKER arch-summit-opt-gnu-hypre-cuda/tests/ksp/ksp/tutorials/ex55 >>>> TEST >>>> arch-summit-opt-gnu-hypre-cuda/tests/counts/ksp_ksp_tutorials-ex55_hypre_device.counts >>>> ok ksp_ksp_tutorials-ex55_hypre_device >>>> ok diff-ksp_ksp_tutorials-ex55_hypre_device >>>> >>>> So this work. In this file (attached) you will see an example of a >>>> construction a matrix that we have gone over before: >>>> >>>> /* create stiffness matrix */ >>>> ierr = MatCreate(comm,&Amat);CHKERRQ(ierr); >>>> ierr = MatSetSizes(Amat,m,m,M,M);CHKERRQ(ierr); >>>> ierr = MatSetType(Amat,MATAIJ);CHKERRQ(ierr); >>>> ierr = MatSetOption(Amat,MAT_SPD,PETSC_TRUE);CHKERRQ(ierr); >>>> ierr = MatSetFromOptions(Amat);CHKERRQ(ierr); >>>> ierr = MatSetBlockSize(Amat,2);CHKERRQ(ierr); >>>> ierr = MatSeqAIJSetPreallocation(Amat,18,NULL);CHKERRQ(ierr); >>>> ierr = MatMPIAIJSetPreallocation(Amat,18,NULL,18,NULL);CHKERRQ(ierr); >>>> #if defined(PETSC_HAVE_HYPRE) >>>> ierr = MatHYPRESetPreallocation(Amat,18,NULL,18,NULL);CHKERRQ(ierr); >>>> #endif >>>> >>>> At the end of the file you will what is executed with this test for >>>> "hypre_device": >>>> >>>> # command line options match GPU defaults >>>> test: >>>> suffix: hypre_device >>>> nsize: 4 >>>> requires: hypre !complex >>>> args: *-mat_type hypre* -ksp_view -ne 29 -alpha 1.e-3 -ksp_type >>>> cg *-pc_type hypre **-pc_hypre_type boomeramg *-ksp_monitor_short >>>> *-pc_hypre_boomeramg_relax_type_all >>>> l1scaled-Jacobi -pc_hypre_boomeramg_interp_type ext+i >>>> -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_no_CF* >>>> >>>> All you need is *-mat_type hypre and -pc_type hypre*. *You could also >>>> add these hypre arguments.* >>>> >>>> If this is not working please send me a description of the problem, >>>> like any error output on your screen and your petsc.history file. >>>> >>>> >>>> >>> >>> >>> -- >>> ---------------------- >>> David Trebotich >>> Lawrence Berkeley National Laboratory >>> Computational Research Division >>> Applied Numerical Algorithms Group >>> [email protected] >>> (510) 486-5984 office >>> (510) 384-6868 mobile >>> >>
