Thank you, Barry.
I have send my code to petsc-maint at mcs.anl.gov. Thank you for your help.
Yuqi
---- Original message ----
>Date: Thu, 5 Apr 2012 08:10:03 -0500
>From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at
>mcs.anl.gov>)
>Subject: Re: [petsc-users] Questions about PCMG
>To: PETSc users list <petsc-users at mcs.anl.gov>
>
>
> It looks like the logic for that line has gotten so convoluted that it lost
> its purpose and is not triggering the rebuilding on the finest level. Thanks
> for finding this, we will investigate and get back to you.
>
>
> Barry
>
> If your code is not complicated and you are willing to send it to us at
> petsc-maint at mcs.anl.gov that will make it quicker for us to determine the
> problem.
>
>
>On Apr 4, 2012, at 11:55 PM, Yuqi Wu wrote:
>
>> Thank you Barry.
>>
>> I am using exact LU as subdomain solver, so I find out PCSetUp_LU() in my
>> debugging output. But there are only three PCSetUp_LU() was called in my
>> program. One in the down smoother of the first SNES, one in the coarse solve
>> of the first SNES, and one in the coarse solve of the second SNES.
>>
>> In those PCSetUp_LU(), MatLUFactorSymbolic and MatLUFactorNumeric is called.
>> But I still have no ideas about the actions taken on the down smoother of
>> the second SNES.
>>
>> I try to setup a breakpoint for PCSetUp_MG(). I find out that the IF
>> statement shown in below is called in the first SNES but not the second
>> SNES. Is it has something related to this issue?
>>
>> /* If user did not provide fine grid operators OR operator was not updated
>> since last global KSPSetOperators() */
>> /* so use those from global PC */
>> /* Is this what we always want? What if user wants to keep old one? */
>> ierr =
>> KSPGetOperatorsSet(mglevels[n-1]->smoothd,PETSC_NULL,&opsset);CHKERRQ(ierr);
>> ierr = KSPGetPC(mglevels[0]->smoothd,&cpc);CHKERRQ(ierr);
>> ierr = KSPGetPC(mglevels[n-1]->smoothd,&mpc);CHKERRQ(ierr);
>> if (!opsset || ((cpc->setupcalled == 1) && (mpc->setupcalled == 2)) ||
>> ((mpc == cpc) && (mpc->setupcalled == 2))) {
>> ierr = PetscInfo(pc,"Using outer operators to define finest grid operator
>> \n because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was
>> not called.\n");CHKERRQ(ierr);
>> ierr =
>> KSPSetOperators(mglevels[n-1]->smoothd,pc->mat,pc->pmat,pc->flag);CHKERRQ(ierr);
>> }
>>
>> Thank you.
>>
>> Yuqi
>>
>>
>>
>> ---- Original message ----
>>> Date: Wed, 4 Apr 2012 19:51:23 -0500
>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith
>>> at mcs.anl.gov>)
>>> Subject: Re: [petsc-users] Questions about PCMG
>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>
>>>
>>> Step through the PCSetUp() function using "next" and you will it first
>>> making the decision of whether to continue with the setup or simply return
>>> early.
>>>
>>> When it gets to the lines
>>>
>>> if (pc->ops->setup) {
>>> ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr);
>>> }
>>>
>>> don't use "next" instead use "step" and it will go into the function
>>> PCSetUp_ILU() and you can move through that one with "next"
>>>
>>>> I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the
>>>> debugger.
>>>
>>> It has to be calling them. Did you use "break MatLUFactorSymbolic" in the
>>> debugger?
>>>
>>> Barry
>>>
>>>
>>>
>>> On Apr 4, 2012, at 7:45 PM, Yuqi Wu wrote:
>>>
>>>> ---- Original message ----
>>>>> Date: Wed, 4 Apr 2012 13:55:55 -0500
>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith
>>>>> <bsmith at mcs.anl.gov>)
>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>
>>>>>
>>>>> Note: In most applications the flag SAME_NONZERO_PATTERN is provided in
>>>>> the compute Jacobian routine, this means that the SYMBOLIC factorization
>>>>> needs to be only done ONCE per matrix; only the numeric factorization
>>>>> needs to be done when the nonzero values have changed (the symbolic need
>>>>> not be repeated). Are you using this flag? How many times in the NUMERIC
>>>>> factorization being done?
>>>>>
>>>>> When you run the program with -info it will print information of the
>>>>> form: (run on one process to make life simple)
>>>>>
>>>>> Setting PC with identical preconditioner\
>>>>> Setting up new PC
>>>>> Setting up PC with same nonzero pattern\
>>>>> Setting up PC with different nonzero pattern\n
>>>>>
>>>>> How many, and exactly what messages of this form are you getting?
>>>>>
>>>>> When all else fails you can run the program in the debugger to track
>>>>> what is happening and why.
>>>>>
>>>>> Put a breakpoint in PCSetUp() then each time it gets called use next to
>>>>> step through it to see what is happening.
>>>>>
>>>>> First thing to check, is PCSetUp() getting called on each level for each
>>>>> new SNES iteration?
>>>>>
>>>>> Second thing, if it is then why is it not triggering the new numerical
>>>>> factorization.
>>>>
>>>> I run the program in debugger. I got total 8 PCSetUp() in the program. And
>>>> I believe that PCSetup() is called on each level for each new SNES
>>>> iteration.
>>>>
>>>> But I don't know how to check why is it not triggering the new numerical
>>>> factorization in the debugger. I cannot find any MatLUFactorSymbolic or
>>>> MatLUFactorNumeric calls in the debugger. Could you give me some
>>>> suggestions to find out the problem?
>>>>
>>>> Best
>>>>
>>>> Yuqi
>>>>
>>>>
>>>>>
>>>>>
>>>>> Barry
>>>>>
>>>>> On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>>>>>
>>>>>> Thanks, Adam.
>>>>>>
>>>>>> Yes. I am using the Galerkin coarse grids. But I am not sure whether the
>>>>>> coarse grid is not getting refactored in the second SNES solve or the
>>>>>> fine grid smoother is not getting refactored in the second SNES solve.
>>>>>>
>>>>>> In the -info output attached in the previous email, the fine grid matrix
>>>>>> is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by
>>>>>> 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ
>>>>>> routines, one for fine martrix, and two for coarse matrix.
>>>>>>
>>>>>>
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed
>>>>>> 11.401
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5.
>>>>>> Using Inode routines
>>>>>>
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed
>>>>>> 7.07175
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or
>>>>>> use
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5.
>>>>>> Using Inode routines
>>>>>>
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed
>>>>>> 7.07175
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or
>>>>>> use
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5.
>>>>>> Using Inode routines
>>>>>>
>>>>>> So I believe that the fine grid smoother is not getting refactored in
>>>>>> the second SNES solve.
>>>>>>
>>>>>> Best
>>>>>>
>>>>>> Yuqi
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---- Original message ----
>>>>>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams"
>>>>>>> <mark.adams at columbia.edu>)
>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>
>>>>>>> I would expect 4 calls to MatLUFactorSym here. It looks like the
>>>>>>> coarse grid is not getting refactored in the second SNES solve.
>>>>>>>
>>>>>>> Are you using Galerkin coarse grids? Perhaps you are not setting a new
>>>>>>> coarse grid with KSPSetOperator and so MG does not bother refactoring
>>>>>>> it.
>>>>>>>
>>>>>>> Mark
>>>>>>>
>>>>>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>>>>>>
>>>>>>>> Thank you.
>>>>>>>>
>>>>>>>> Can I ask another question?
>>>>>>>>
>>>>>>>> In my log summary output, it shows that although there are two SNES
>>>>>>>> iteration and total 9 linear iterations. The functions MatLUFactorSym
>>>>>>>> and MatLUFactorNum are only called for three times.
>>>>>>>>
>>>>>>>> MatLUFactorSym 3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00
>>>>>>>> 0.0e+00 1.5e+01 1 0 0 0 2 1 0 0 0 2 0
>>>>>>>> MatLUFactorNum 3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00
>>>>>>>> 0.0e+00 0.0e+00 31 97 0 0 0 32 97 0 0 0 280
>>>>>>>>
>>>>>>>> I checked the -info output. It shows that One
>>>>>>>> MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first
>>>>>>>> SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve
>>>>>>>> of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in
>>>>>>>> the down smoother of the second SNES.
>>>>>>>>
>>>>>>>> Do you have any ideas why there are 9 multigrid iterations, but only 3
>>>>>>>> MatLUFactorSymbolic calls in the program?
>>>>>>>>
>>>>>>>> Best
>>>>>>>>
>>>>>>>> Yuqi
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---- Original message ----
>>>>>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith
>>>>>>>>> <bsmith at mcs.anl.gov>)
>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two
>>>>>>>>> MGSetUp on each level. Then a total of 9 multigrid iterations (in
>>>>>>>>> both linear solves together) hence 9 smoother on level 0 (level 0
>>>>>>>>> means coarse grid solve). One smooth down and one smooth up on level
>>>>>>>>> 1 hence 18 total smooths on level 1. 9 computation of residual on
>>>>>>>>> level 1 and 18 MgInterp because that logs both the restriction to
>>>>>>>>> level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>>>>>>>
>>>>>>>>> Barry
>>>>>>>>>
>>>>>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>>>>>>
>>>>>>>>>> Hi, Barry,
>>>>>>>>>>
>>>>>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last
>>>>>>>>>> norm 0.000000e+00
>>>>>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last
>>>>>>>>>> norm 2.268574e-06.
>>>>>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last
>>>>>>>>>> norm 1.312605e-12.
>>>>>>>>>>
>>>>>>>>>> And -pc_mg_log shows the following output
>>>>>>>>>>
>>>>>>>>>> MGSetup Level 0 2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02
>>>>>>>>>> 6.0e+04 3.0e+01 1 0 3 11 2 1 0 3 11 2 0
>>>>>>>>>> MGSmooth Level 0 9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03
>>>>>>>>>> 1.7e+03 6.4e+02 33 71 28 3 34 35 71 28 3 35 415
>>>>>>>>>> MGSetup Level 1 2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02
>>>>>>>>>> 3.1e+04 2.3e+01 0 0 1 3 1 0 0 1 3 1 0
>>>>>>>>>> MGSmooth Level 1 18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03
>>>>>>>>>> 2.9e+03 3.6e+02 16 28 15 3 19 17 28 15 3 19 339
>>>>>>>>>> MGResid Level 1 9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02
>>>>>>>>>> 2.7e+03 9.0e+00 0 0 2 0 0 0 0 2 0 0 70
>>>>>>>>>> MGInterp Level 1 18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02
>>>>>>>>>> 1.1e+03 1.8e+01 0 0 4 0 1 0 0 4 0 1 37
>>>>>>>>>>
>>>>>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>>>>>>
>>>>>>>>>> Best
>>>>>>>>>>
>>>>>>>>>> Yuqi
>>>>>>>>>>
>>>>>>>>>> ---- Original message ----
>>>>>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith
>>>>>>>>>>> <bsmith at mcs.anl.gov>)
>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part
>>>>>>>>>>> of the basic PCMG. Are you sure you are calling
>>>>>>>>>>> SNESSetFromOptions()?
>>>>>>>>>>>
>>>>>>>>>>> Barry
>>>>>>>>>>>
>>>>>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi, Mark,
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you so much for your suggestion.
>>>>>>>>>>>>
>>>>>>>>>>>> The problem 1 is resolved by avoiding calling
>>>>>>>>>>>> PCMGSetNumberSmoothUp.
>>>>>>>>>>>>
>>>>>>>>>>>> But since I am using the unstructured grid in my application, I
>>>>>>>>>>>> didn't use DA or dmmg, so -pc_mg_log didn't give any level
>>>>>>>>>>>> information. I try to run my code using -info with 1 processor,
>>>>>>>>>>>> and I find out some interesting issues.
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>
>
