I tested -pc_gamg_repartition with 216 processors again. First I tested with these options
-log_view \ -ksp_rtol 1E-6 \ -ksp_type cg \ -ksp_norm_type unpreconditioned \ -mg_levels_ksp_type richardson \ -mg_levels_ksp_norm_type none \ -mg_levels_pc_type sor \ -mg_levels_ksp_max_it 1 \ -mg_levels_pc_sor_its 1 \ -mg_levels_esteig_ksp_type cg \ -mg_levels_esteig_ksp_max_it 10 \ -pc_type gamg \ -pc_gamg_type agg \ -pc_gamg_threshold 0.05 \ -pc_gamg_type classical \ -gamg_est_ksp_type cg \ -pc_gamg_square_graph 10 \ -pc_gamg_threshold 0.0 then I tested with an extra -pc_gamg_repartition. With repartition, the time increased from 120s to 140s. The code measures first KSPSolve and the remaining in separate stages, so the repartition time was not counted in the stage of interest. Actually, log_view says GMAG :repartition time (in the first event stage) is about 1.5 sec., so it is not a big deal. I also tested -pc_gamg_square_graph 4. It did not change the time. I tested hypre with options "-log_view -ksp_rtol 1E-6 -ksp_type cg -ksp_norm_type unpreconditioned -pc_type hypre" and nothing else. The code ran out of time. In old tests, a job (1000 KSPSolve with 7 KSP iterations each) took 4 minutes. With hypre, 1 KSPSolve + 6 KSP iterations each, takes 6 minutes. I will test and profile the code on a single node, and apply some vecscatter optimizations I recently did to see what happens. --Junchao Zhang On Thu, Jun 14, 2018 at 11:03 AM, Mark Adams <[email protected]> wrote: > And with 7-point stensils and no large material discontinuities you > probably want -pc_gamg_square_graph 10 -pc_gamg_threshold 0.0 and you > could test the square graph parameter (eg, 1,2,3,4). > > And I would definitely test hypre. > > On Thu, Jun 14, 2018 at 8:54 AM Mark Adams <[email protected]> wrote: > >> >>> Just -pc_type hypre instead of -pc_type gamg. >>> >>> >> And you need to have configured PETSc with hypre. >> >> >
