The coarse problem for the PCMG (geometric multigrid) is
Mat Object: 8192 MPI processes
type: mpiaij
rows=8192, cols=8192
then it tries to solve it with algebraic multigrid on 8192 processes (which is
completely insane). A lot of the time is spent in setting up the algebraic
multigrid (not surprisingly).
8192 is kind of small to parallelize. Please run the same code but with the
default coarse grid problem instead of PCGAMG and send us the -log_summary again
Barry
> On Jul 24, 2015, at 1:35 PM, Michele Rosso <[email protected]> wrote:
>
> Hi Mark and Barry,
>
> I am sorry for my late reply: it was a busy week!
> I run a test case for a larger problem with as many levels (i.e. 5) of MG I
> could and GAMG as PC at the coarse level. I attached the output of info (
> after grep for "gmag"), ksp_view and log_summary.
> The solve takes about 2 seconds on 8192 cores, which is way too much. The
> number of iterations to convergence is 24.
> I hope there is a way to speed it up.
>
> Thanks,
> Michele
>
>
> On Fri, 2015-07-17 at 09:38 -0400, Mark Adams wrote:
>>
>>
>> On Thu, Jul 16, 2015 at 8:18 PM, Michele Rosso <[email protected]> wrote:
>> Barry,
>>
>> thank you very much for the detailed answer. I tried what you suggested and
>> it works.
>> So far I tried on a small system but the final goal is to use it for very
>> large runs. How does PCGAMG compares to PCMG as far as performances and
>> scalability are concerned?
>> Also, could you help me to tune the GAMG part ( my current setup is in the
>> attached ksp_view.txt file )?
>>
>>
>>
>> I am going to add this to the document today but you can run with -info.
>> This is very noisy so you might want to do the next step at run time. Then
>> grep on GAMG. This will be about 20 lines. Send that to us and we can go
>> from there.
>>
>>
>> Mark
>>
>>
>>
>>
>> I also tried to use superlu_dist for the LU decomposition on
>> mg_coarse_mg_sub_
>> -mg_coarse_mg_coarse_sub_pc_type lu
>> -mg_coarse_mg_coarse_sub_pc_factor_mat_solver_package superlu_dist
>>
>> but I got an error:
>>
>> ****** Error in MC64A/AD. INFO(1) = -2
>> ****** Error in MC64A/AD. INFO(1) = -2
>> ****** Error in MC64A/AD. INFO(1) = -2
>> ****** Error in MC64A/AD. INFO(1) = -2
>> ****** Error in MC64A/AD. INFO(1) = -2
>> ****** Error in MC64A/AD. INFO(1) = -2
>> ****** Error in MC64A/AD. INFO(1) = -2
>> symbfact() error returns 0
>> symbfact() error returns 0
>> symbfact() error returns 0
>> symbfact() error returns 0
>> symbfact() error returns 0
>> symbfact() error returns 0
>> symbfact() error returns 0
>>
>>
>> Thank you,
>> Michele
>>
>>
>> On Thu, 2015-07-16 at 18:07 -0500, Barry Smith wrote:
>>>
>>> > On Jul 16, 2015, at 5:42 PM, Michele Rosso <[email protected]> wrote:
>>> >
>>> > Barry,
>>> >
>>> > thanks for your reply. So if I want it fixed, I will have to use the
>>> > master branch, correct?
>>>
>>>
>>> Yes, or edit mg.c and remove the offending lines of code (easy enough).
>>>
>>> >
>>> > On a side note, what I am trying to achieve is to be able to use how many
>>> > levels of MG I want, despite the limitation imposed by the local number
>>> > of grid nodes.
>>>
>>>
>>> I assume you are talking about with DMDA? There is no generic limitation
>>> for PETSc's multigrid, it is only with the way the DMDA code figures out
>>> the interpolation that causes a restriction.
>>>
>>>
>>> > So far I am using a borrowed code that implements a PC that creates a sub
>>> > communicator and perform MG on it.
>>> > While reading the documentation I found out that PCMGSetLevels takes in
>>> > an optional array of communicators. How does this work?
>>>
>>>
>>> It doesn't work. It was an idea that never got pursued.
>>>
>>>
>>> > Can I can simply define my matrix and rhs on the fine grid as I would do
>>> > normally ( I do not use kspsetoperators and kspsetrhs ) and KSP would
>>> > take care of it by using the correct communicator for each level?
>>>
>>>
>>> No.
>>>
>>> You can use the PCMG geometric multigrid with DMDA for as many levels as
>>> it works and then use PCGAMG as the coarse grid solver. PCGAMG
>>> automatically uses fewer processes for the coarse level matrices and
>>> vectors. You could do this all from the command line without writing code.
>>>
>>> For example if your code uses a DMDA and calls KSPSetDM() use for
>>> example -da_refine 3 -pc_type mg -pc_mg_galerkin -mg_coarse_pc_type gamg
>>> -ksp_view
>>>
>>>
>>>
>>> Barry
>>>
>>>
>>>
>>> >
>>> > Thanks,
>>> > Michele
>>> >
>>> >
>>> >
>>> >
>>> > On Thu, 2015-07-16 at 17:30 -0500, Barry Smith wrote:
>>> >> Michel,
>>> >>
>>> >> This is a very annoying feature that has been fixed in master
>>> >> http://www.mcs.anl.gov/petsc/developers/index.html
>>> >> I would like to have changed it in maint but Jed would have a shit-fit
>>> >> :-) since it changes behavior.
>>> >>
>>> >> Barry
>>> >>
>>> >>
>>> >> > On Jul 16, 2015, at 4:53 PM, Michele Rosso <[email protected]> wrote:
>>> >> >
>>> >> > Hi,
>>> >> >
>>> >> > I am performing a series of solves inside a loop. The matrix for each
>>> >> > solve changes but not enough to justify a rebuilt of the PC at each
>>> >> > solve.
>>> >> > Therefore I am using KSPSetReusePreconditioner to avoid rebuilding
>>> >> > unless necessary. The solver is CG + MG with a custom PC at the
>>> >> > coarse level.
>>> >> > If KSP is not updated each time, everything works as it is supposed
>>> >> > to.
>>> >> > When instead I allow the default PETSc behavior, i.e. updating PC
>>> >> > every time the matrix changes, the coarse level KSP , initially set to
>>> >> > PREONLY, is changed into GMRES
>>> >> > after the first solve. I am not sure where the problem lies (my PC or
>>> >> > PETSc), so I would like to have your opinion on this.
>>> >> > I attached the ksp_view for the 2 successive solve and the options
>>> >> > stack.
>>> >> >
>>> >> > Thanks for your help,
>>> >> > Michel
>>> >> >
>>> >> >
>>> >> >
>>> >> > <ksp_view.txt><petsc_options.txt>
>>> >>
>>> >>
>>> >>
>>> >
>>>
>>>
>>>
>>
>>
>>
>>
>
> <info.txt><ksp_view.txt><log_gamg.txt>
