I am running snes/ex13 on Perlmutter and doing a scaling study in a script.
The first case runs fine:
+ srun -G 32 -n 256 --cpu-bind=cores --ntasks-per-core=1
/global/homes/m/madams/mps-wrapper.sh ../ex13 -dm_plex_box_faces 4,8,8
-petscpartitioner_simple_node_grid 2,2,2 *-dm_refine 3 *-dm_mat_type
aijcusparse -dm_vec_type cuda -dm_view -ksp_max_it 15 -log_view
but the next case *with another levels of refinement*:
+ srun -G 32 -n 256 --cpu-bind=cores --ntasks-per-core=1
/global/homes/m/madams/mps-wrapper.sh ../ex13 -dm_plex_box_faces 4,8,8
-petscpartitioner_simple_node_grid 2,2,2 *-dm_refine 4 *-dm_mat_type
aijcusparse -dm_vec_type cuda -dm_view -ksp_max_it 15 -log_
hangs in BuildTwoSided. With log_trace I see this (grepping on 177). Args
appended.
Any ideas?
Thanks
[177] 21.4441 Event begin: MatGetBrAoCol
[177] 21.4441 Event begin: SFSetUp
[177] 21.4441 Event begin: BuildTwoSided
[177] 21.5443 Event end: BuildTwoSided
[177] 21.5443 Event end: SFSetUp
[177] 21.5443 Event begin: MatAssemblyBegin
[177] 21.5444 Event end: MatAssemblyBegin
[177] 21.5444 Event begin: MatAssemblyEnd
[177] 21.5444 Event end: MatAssemblyEnd
[177] 21.5444 Event end: MatGetBrAoCol
[177] 21.5444 Event begin: MatGetLocalMat
[177] 21.5569 Event begin: MatCUSPARSCopyTo
[177] 21.557 Event end: MatCUSPARSCopyTo
[177] 21.557 Event begin: MatCUSPARSCopyTo
[177] 21.5571 Event end: MatCUSPARSCopyTo
[177] 21.5571 Event end: MatGetLocalMat
[177] 21.5571 Event begin: MatCUSPARSCopyTo
[177] 21.5571 Event end: MatCUSPARSCopyTo
[177] 21.5698 Event begin: MatCUSPARSCopyTo
[177] 21.5698 Event end: MatCUSPARSCopyTo
[177] 21.5827 Event begin: MatConvert
[177] 21.5954 Event end: MatConvert
[177] 21.5954 Event begin: MatCUSPARSCopyTo
[177] 21.5954 Event end: MatCUSPARSCopyTo
[177] 21.5954 Event begin: MatCUSPARSCopyTo
[177] 21.5955 Event end: MatCUSPARSCopyTo
[177] 21.6208 Event begin: SFSetGraph
[177] 21.6208 Event end: SFSetGraph
[177] 21.6208 Event begin: SFSetUp
[177] 21.6208 Event begin: BuildTwoSided
#PETSc Option Table entries:
-benchmark_it 10
-dm_distribute
-dm_mat_type aijcusparse
-dm_plex_box_faces 4,8,8
-dm_plex_box_lower 0,0,0
-dm_plex_box_upper 2,4,4
-dm_plex_dim 3
-dm_plex_simplex 0
-dm_refine 3
-dm_vec_type cuda
-dm_view
-ksp_max_it 15
-ksp_monitor_short
-ksp_norm_type unpreconditioned
-ksp_rtol 1.e-12
-ksp_type cg
-log_view
-matptap_via scalable
-mg_levels_esteig_ksp_max_it 5
-mg_levels_esteig_ksp_type cg
-mg_levels_ksp_max_it 2
-mg_levels_ksp_type richardson
-mg_levels_pc_type jacobi
-options_left
-pc_gamg_coarse_eq_limit 100
-pc_gamg_coarse_grid_layout_type spread
-pc_gamg_esteig_ksp_max_it 5
-pc_gamg_esteig_ksp_type cg
-pc_gamg_process_eq_limit 100
-pc_gamg_repartition false
-pc_gamg_reuse_interpolation true
-pc_gamg_square_graph 1
-pc_gamg_threshold 0.01
-pc_gamg_threshold_scale .5
-pc_type gamg
-petscpartitioner_simple_node_grid 2,2,2
-petscpartitioner_simple_process_grid 2,4,4
-petscpartitioner_type simple
-potential_petscspace_degree 2
-snes_max_it 1
-snes_rtol 1.e-8
-snes_type ksponly
-use_gpu_aware_mpi 0
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8
sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: --CFLAGS=" -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10
-DLANDAU_MAX_Q=4" --CXXFLAGS=" -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10
-DLANDAU_MAX_Q=4" --CUDAFLAGS="-g -Xcompiler -rdynamic -DLANDAU_DIM=2
-DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_
Q=4" --with-cc=cc --with-cxx=CC --with-fc=ftn
--with-cudac=/global/common/software/nersc/cos1.3/cuda/11.3.0/bin/nvcc
--FFLAGS=" -g " --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS="
-O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with
-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1
PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda
-----------------------------------------
Libraries compiled on 2021-10-16 18:33:45 on login02
