> On Jul 16, 2015, at 7:18 PM, Michele Rosso <[email protected]> wrote: > > Barry, > > thank you very much for the detailed answer. I tried what you suggested and > it works. > So far I tried on a small system but the final goal is to use it for very > large runs. How does PCGAMG compares to PCMG as far as performances and > scalability are concerned?
Algebraic multigrid has much larger setup times than geometric multigrid, but since it is is running on a much smaller problem with geometric multigrid on top of it that is much less of an issue. > Also, could you help me to tune the GAMG part ( my current setup is in the > attached ksp_view.txt file )? The defaults are probably fine. > > I also tried to use superlu_dist for the LU decomposition on mg_coarse_mg_sub_ > -mg_coarse_mg_coarse_sub_pc_type lu > -mg_coarse_mg_coarse_sub_pc_factor_mat_solver_package superlu_dist Yeah, gamg tries to put all the rows of the matrix on one process and leave the other processes empty, I suspect superlu_dist doesn't like empty matrices. Just use -mg_coarse_mg_coarse_sub_pc_type lu it is PETSc's native sequential sparse solver and will run faster than superlu_dist anyways except for really large coarse problems which you don't want anyway. How many iterations is the multigrid using? Instead of using -mg_coarse_pc_type some of the PETSc guys are fans of -mg_coarse_pc_type chebyshev which may decrease the iterations slightly. You can run with -log_summary and send the output, this shows where the time is being spent. Barry > > but I got an error: > > ****** Error in MC64A/AD. INFO(1) = -2 > ****** Error in MC64A/AD. INFO(1) = -2 > ****** Error in MC64A/AD. INFO(1) = -2 > ****** Error in MC64A/AD. INFO(1) = -2 > ****** Error in MC64A/AD. INFO(1) = -2 > ****** Error in MC64A/AD. INFO(1) = -2 > ****** Error in MC64A/AD. INFO(1) = -2 > symbfact() error returns 0 > symbfact() error returns 0 > symbfact() error returns 0 > symbfact() error returns 0 > symbfact() error returns 0 > symbfact() error returns 0 > symbfact() error returns 0 > > > Thank you, > Michele > > > On Thu, 2015-07-16 at 18:07 -0500, Barry Smith wrote: >> > On Jul 16, 2015, at 5:42 PM, Michele Rosso <[email protected]> wrote: >> > >> > Barry, >> > >> > thanks for your reply. So if I want it fixed, I will have to use the >> > master branch, correct? >> >> >> Yes, or edit mg.c and remove the offending lines of code (easy enough). >> >> > >> > On a side note, what I am trying to achieve is to be able to use how many >> > levels of MG I want, despite the limitation imposed by the local number of >> > grid nodes. >> >> >> I assume you are talking about with DMDA? There is no generic limitation >> for PETSc's multigrid, it is only with the way the DMDA code figures out the >> interpolation that causes a restriction. >> >> >> > So far I am using a borrowed code that implements a PC that creates a sub >> > communicator and perform MG on it. >> > While reading the documentation I found out that PCMGSetLevels takes in an >> > optional array of communicators. How does this work? >> >> >> It doesn't work. It was an idea that never got pursued. >> >> >> > Can I can simply define my matrix and rhs on the fine grid as I would do >> > normally ( I do not use kspsetoperators and kspsetrhs ) and KSP would take >> > care of it by using the correct communicator for each level? >> >> >> No. >> >> You can use the PCMG geometric multigrid with DMDA for as many levels as >> it works and then use PCGAMG as the coarse grid solver. PCGAMG automatically >> uses fewer processes for the coarse level matrices and vectors. You could do >> this all from the command line without writing code. >> >> For example if your code uses a DMDA and calls KSPSetDM() use for example >> -da_refine 3 -pc_type mg -pc_mg_galerkin -mg_coarse_pc_type gamg -ksp_view >> >> >> >> Barry >> >> >> >> > >> > Thanks, >> > Michele >> > >> > >> > >> > >> > On Thu, 2015-07-16 at 17:30 -0500, Barry Smith wrote: >> >> Michel, >> >> >> >> This is a very annoying feature that has been fixed in master >> >> http://www.mcs.anl.gov/petsc/developers/index.html >> >> I would like to have changed it in maint but Jed would have a shit-fit >> >> :-) since it changes behavior. >> >> >> >> Barry >> >> >> >> >> >> > On Jul 16, 2015, at 4:53 PM, Michele Rosso <[email protected]> wrote: >> >> > >> >> > Hi, >> >> > >> >> > I am performing a series of solves inside a loop. The matrix for each >> >> > solve changes but not enough to justify a rebuilt of the PC at each >> >> > solve. >> >> > Therefore I am using KSPSetReusePreconditioner to avoid rebuilding >> >> > unless necessary. The solver is CG + MG with a custom PC at the coarse >> >> > level. >> >> > If KSP is not updated each time, everything works as it is supposed to. >> >> > When instead I allow the default PETSc behavior, i.e. updating PC >> >> > every time the matrix changes, the coarse level KSP , initially set to >> >> > PREONLY, is changed into GMRES >> >> > after the first solve. I am not sure where the problem lies (my PC or >> >> > PETSc), so I would like to have your opinion on this. >> >> > I attached the ksp_view for the 2 successive solve and the options >> >> > stack. >> >> > >> >> > Thanks for your help, >> >> > Michel >> >> > >> >> > >> >> > >> >> > <ksp_view.txt><petsc_options.txt> >> >> >> >> >> >> >> > >> >> >> > > <ksp_view.txt>
