Excellent! Now the only real hopes for improvement are 1) run on an Intel system with the highest possible achievable memory bandwidth (this could possibly speed up the code by 50%)
2) to decrease the iteration counts a bit try -pressure_mg_levels_ksp_type chebyshev 3) some optimization on MatPtAPNumeric() sequential; we would have to do this, could shave off a small amount of time Barry Here is the percentage of time spent in the pressure correction MatMult 11 MatMultAdd 4 MatSOR 57 MatPtAPNumeric 17 > On Feb 20, 2015, at 3:35 AM, Fabian Gabel <[email protected]> wrote: > > Barry, > >> >> First, is your pressure problem changing dramatically at each new solver? >> That is, for example, is the mesh moving or are there very different >> numerical values in the matrix? Is the nonzero structure of the pressure >> matrix changing? > > No moving grids, the non-zero structure is maintained throughout the > entire solution process. I am not sure about the "very different > numerical values". I determined the minimal matrix coefficient to be > approx -5e-7 and the maximal matrix coefficient to be 3e-6 (but once I > use block structured grids with locally refined blocks the range will > become wider), but there are some lines containing only a 1 on the > diagonal. This comes from the variable indexing I use, which includes > boundary values. If this should present a problem, I think I could scale > the corresponding rows with a factor depending on the maximal/minimal > element of the matrix. > >> Currently the entire GAMG process is done for each new solve, if you use the >> flag >> >> -pressure_pc_gamg_reuse_interpolation true >> >> it will create the interpolation needed for GAMG once and reuse it for all >> the solves. Please try that and see what happens. > > I attached the output for the additional solver option > (-reuse_interpolation). Since there appear to be some inconsistencies > with the previous output file for the GAMG solve I provided, I'll attach > the results for the solution process without the flag for reusing the > interpolation once again. So far wall clock time has been reduced by > almost 50%. > > Fabian > >> >> Then I will have many more suggestions. >> >> >> Barry >> >> >> >>> On Feb 17, 2015, at 9:14 AM, Fabian Gabel <[email protected]> wrote: >>> >>> Dear PETSc team, >>> >>> I am trying to optimize the solver parameters for the linear system I >>> get, when I discretize the pressure correction equation Poisson equation >>> with Neumann boundary conditions) in a SIMPLE-type algorithm using a >>> finite volume method. >>> >>> The resulting system is symmetric and positive semi-definite. A basis to >>> the associated nullspace has been provided to the KSP object. >>> >>> Using a CG solver with ICC preconditioning the solver needs a lot of >>> inner iterations to converge (-ksp_monitor -ksp_view output attached for >>> a case with approx. 2e6 unknowns; the lines beginning with 000XXXX show >>> the relative residual regarding the initial residual in the outer >>> iteration no. 1 for the variables u,v,w,p). Furthermore I don't quite >>> understand, why the solver reports >>> >>> Linear solve did not converge due to DIVERGED_INDEFINITE_PC >>> >>> at the later stages of my Picard iteration process (iteration 0001519). >>> >>> I then tried out CG+GAMG preconditioning with success regarding the >>> number of inner iterations, but without advantages regarding wall time >>> (output attached). Also the DIVERGED_INDEFINITE_PC reason shows up >>> repeatedly after iteration 0001487. I used the following options >>> >>> -pressure_mg_coarse_sub_pc_type svd >>> -pressure_mg_levels_ksp_rtol 1e-4 >>> -pressure_mg_levels_ksp_type richardson >>> -pressure_mg_levels_pc_type sor >>> -pressure_pc_gamg_agg_nsmooths 1 >>> -pressure_pc_type gamg >>> >>> I would like to get an opinion on how the solver performance could be >>> increased further. -log_summary shows that my code spends 80% of the >>> time solving the linear systems for the pressure correction (STAGE 2: >>> PRESSCORR). Furthermore, do you know what could be causing the >>> DIVERGED_INDEFINITE_PC converged reason? >>> >>> Regards, >>> Fabian Gabel >>> <gamg.128.out.531314><icc.128.out.429762> >> > > > <reuse_interpolation.gamg.128.out.544044><gamg.128.out.541300>
