Yes, at least that is the intention.
Barry
On Apr 5, 2012, at 2:06 PM, Yuqi Wu wrote:
> Hi, Barry,
>
> If my matrix is of type mpibaij, then I cannot use the Galerkin flag in the
> PCMG, e.g. PCMGSetGalerkin(finepc,PETSC_TRUE);
>
> Otherwise, I got the following error messages in PETSc
> [0]PETSC ERROR: No support for this operation for this object type!
> [0]PETSC ERROR: Matrix of type <mpibaij> does not support PtAP!
>
> Can I use the following commands to setup the coarse operator
>
> ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
> ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J,
> DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
>
> Thank you.
>
> Best
>
> Yuqi
>
>
> ---- Original message ----
>> Date: Thu, 5 Apr 2012 08:10:03 -0500
>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith
>> at mcs.anl.gov>)
>> Subject: Re: [petsc-users] Questions about PCMG
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>
>>
>> It looks like the logic for that line has gotten so convoluted that it lost
>> its purpose and is not triggering the rebuilding on the finest level. Thanks
>> for finding this, we will investigate and get back to you.
>>
>>
>> Barry
>>
>> If your code is not complicated and you are willing to send it to us at
>> petsc-maint at mcs.anl.gov that will make it quicker for us to determine the
>> problem.
>>
>>
>> On Apr 4, 2012, at 11:55 PM, Yuqi Wu wrote:
>>
>>> Thank you Barry.
>>>
>>> I am using exact LU as subdomain solver, so I find out PCSetUp_LU() in my
>>> debugging output. But there are only three PCSetUp_LU() was called in my
>>> program. One in the down smoother of the first SNES, one in the coarse
>>> solve of the first SNES, and one in the coarse solve of the second SNES.
>>>
>>> In those PCSetUp_LU(), MatLUFactorSymbolic and MatLUFactorNumeric is
>>> called. But I still have no ideas about the actions taken on the down
>>> smoother of the second SNES.
>>>
>>> I try to setup a breakpoint for PCSetUp_MG(). I find out that the IF
>>> statement shown in below is called in the first SNES but not the second
>>> SNES. Is it has something related to this issue?
>>>
>>> /* If user did not provide fine grid operators OR operator was not updated
>>> since last global KSPSetOperators() */
>>> /* so use those from global PC */
>>> /* Is this what we always want? What if user wants to keep old one? */
>>> ierr =
>>> KSPGetOperatorsSet(mglevels[n-1]->smoothd,PETSC_NULL,&opsset);CHKERRQ(ierr);
>>> ierr = KSPGetPC(mglevels[0]->smoothd,&cpc);CHKERRQ(ierr);
>>> ierr = KSPGetPC(mglevels[n-1]->smoothd,&mpc);CHKERRQ(ierr);
>>> if (!opsset || ((cpc->setupcalled == 1) && (mpc->setupcalled == 2)) ||
>>> ((mpc == cpc) && (mpc->setupcalled == 2))) {
>>> ierr = PetscInfo(pc,"Using outer operators to define finest grid operator
>>> \n because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...);
>>> was not called.\n");CHKERRQ(ierr);
>>> ierr =
>>> KSPSetOperators(mglevels[n-1]->smoothd,pc->mat,pc->pmat,pc->flag);CHKERRQ(ierr);
>>> }
>>>
>>> Thank you.
>>>
>>> Yuqi
>>>
>>>
>>>
>>> ---- Original message ----
>>>> Date: Wed, 4 Apr 2012 19:51:23 -0500
>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith
>>>> at mcs.anl.gov>)
>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>
>>>>
>>>> Step through the PCSetUp() function using "next" and you will it first
>>>> making the decision of whether to continue with the setup or simply return
>>>> early.
>>>>
>>>> When it gets to the lines
>>>>
>>>> if (pc->ops->setup) {
>>>> ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr);
>>>> }
>>>>
>>>> don't use "next" instead use "step" and it will go into the function
>>>> PCSetUp_ILU() and you can move through that one with "next"
>>>>
>>>>> I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the
>>>>> debugger.
>>>>
>>>> It has to be calling them. Did you use "break MatLUFactorSymbolic" in the
>>>> debugger?
>>>>
>>>> Barry
>>>>
>>>>
>>>>
>>>> On Apr 4, 2012, at 7:45 PM, Yuqi Wu wrote:
>>>>
>>>>> ---- Original message ----
>>>>>> Date: Wed, 4 Apr 2012 13:55:55 -0500
>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith
>>>>>> <bsmith at mcs.anl.gov>)
>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>
>>>>>>
>>>>>> Note: In most applications the flag SAME_NONZERO_PATTERN is provided in
>>>>>> the compute Jacobian routine, this means that the SYMBOLIC factorization
>>>>>> needs to be only done ONCE per matrix; only the numeric factorization
>>>>>> needs to be done when the nonzero values have changed (the symbolic need
>>>>>> not be repeated). Are you using this flag? How many times in the NUMERIC
>>>>>> factorization being done?
>>>>>>
>>>>>> When you run the program with -info it will print information of the
>>>>>> form: (run on one process to make life simple)
>>>>>>
>>>>>> Setting PC with identical preconditioner\
>>>>>> Setting up new PC
>>>>>> Setting up PC with same nonzero pattern\
>>>>>> Setting up PC with different nonzero pattern\n
>>>>>>
>>>>>> How many, and exactly what messages of this form are you getting?
>>>>>>
>>>>>> When all else fails you can run the program in the debugger to track
>>>>>> what is happening and why.
>>>>>>
>>>>>> Put a breakpoint in PCSetUp() then each time it gets called use next to
>>>>>> step through it to see what is happening.
>>>>>>
>>>>>> First thing to check, is PCSetUp() getting called on each level for each
>>>>>> new SNES iteration?
>>>>>>
>>>>>> Second thing, if it is then why is it not triggering the new numerical
>>>>>> factorization.
>>>>>
>>>>> I run the program in debugger. I got total 8 PCSetUp() in the program.
>>>>> And I believe that PCSetup() is called on each level for each new SNES
>>>>> iteration.
>>>>>
>>>>> But I don't know how to check why is it not triggering the new numerical
>>>>> factorization in the debugger. I cannot find any MatLUFactorSymbolic or
>>>>> MatLUFactorNumeric calls in the debugger. Could you give me some
>>>>> suggestions to find out the problem?
>>>>>
>>>>> Best
>>>>>
>>>>> Yuqi
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> Barry
>>>>>>
>>>>>> On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>>>>>>
>>>>>>> Thanks, Adam.
>>>>>>>
>>>>>>> Yes. I am using the Galerkin coarse grids. But I am not sure whether
>>>>>>> the coarse grid is not getting refactored in the second SNES solve or
>>>>>>> the fine grid smoother is not getting refactored in the second SNES
>>>>>>> solve.
>>>>>>>
>>>>>>> In the -info output attached in the previous email, the fine grid
>>>>>>> matrix is of size 11585 by 11585, and the coarse grid matrix is of size
>>>>>>> 4186 by 4186. In the -info output, I found out three
>>>>>>> MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for
>>>>>>> coarse matrix.
>>>>>>>
>>>>>>>
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed
>>>>>>> 11.401
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or
>>>>>>> use
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used:
>>>>>>> 5. Using Inode routines
>>>>>>>
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed
>>>>>>> 7.07175
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or
>>>>>>> use
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5.
>>>>>>> Using Inode routines
>>>>>>>
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed
>>>>>>> 7.07175
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or
>>>>>>> use
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5.
>>>>>>> Using Inode routines
>>>>>>>
>>>>>>> So I believe that the fine grid smoother is not getting refactored in
>>>>>>> the second SNES solve.
>>>>>>>
>>>>>>> Best
>>>>>>>
>>>>>>> Yuqi
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---- Original message ----
>>>>>>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams"
>>>>>>>> <mark.adams at columbia.edu>)
>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>
>>>>>>>> I would expect 4 calls to MatLUFactorSym here. It looks like the
>>>>>>>> coarse grid is not getting refactored in the second SNES solve.
>>>>>>>>
>>>>>>>> Are you using Galerkin coarse grids? Perhaps you are not setting a
>>>>>>>> new coarse grid with KSPSetOperator and so MG does not bother
>>>>>>>> refactoring it.
>>>>>>>>
>>>>>>>> Mark
>>>>>>>>
>>>>>>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>>>>>>>
>>>>>>>>> Thank you.
>>>>>>>>>
>>>>>>>>> Can I ask another question?
>>>>>>>>>
>>>>>>>>> In my log summary output, it shows that although there are two SNES
>>>>>>>>> iteration and total 9 linear iterations. The functions MatLUFactorSym
>>>>>>>>> and MatLUFactorNum are only called for three times.
>>>>>>>>>
>>>>>>>>> MatLUFactorSym 3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00
>>>>>>>>> 0.0e+00 1.5e+01 1 0 0 0 2 1 0 0 0 2 0
>>>>>>>>> MatLUFactorNum 3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00
>>>>>>>>> 0.0e+00 0.0e+00 31 97 0 0 0 32 97 0 0 0 280
>>>>>>>>>
>>>>>>>>> I checked the -info output. It shows that One
>>>>>>>>> MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first
>>>>>>>>> SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve
>>>>>>>>> of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in
>>>>>>>>> the down smoother of the second SNES.
>>>>>>>>>
>>>>>>>>> Do you have any ideas why there are 9 multigrid iterations, but only
>>>>>>>>> 3 MatLUFactorSymbolic calls in the program?
>>>>>>>>>
>>>>>>>>> Best
>>>>>>>>>
>>>>>>>>> Yuqi
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---- Original message ----
>>>>>>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith
>>>>>>>>>> <bsmith at mcs.anl.gov>)
>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two
>>>>>>>>>> MGSetUp on each level. Then a total of 9 multigrid iterations (in
>>>>>>>>>> both linear solves together) hence 9 smoother on level 0 (level 0
>>>>>>>>>> means coarse grid solve). One smooth down and one smooth up on level
>>>>>>>>>> 1 hence 18 total smooths on level 1. 9 computation of residual on
>>>>>>>>>> level 1 and 18 MgInterp because that logs both the restriction to
>>>>>>>>>> level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>>>>>>>>
>>>>>>>>>> Barry
>>>>>>>>>>
>>>>>>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi, Barry,
>>>>>>>>>>>
>>>>>>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last
>>>>>>>>>>> norm 0.000000e+00
>>>>>>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last
>>>>>>>>>>> norm 2.268574e-06.
>>>>>>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last
>>>>>>>>>>> norm 1.312605e-12.
>>>>>>>>>>>
>>>>>>>>>>> And -pc_mg_log shows the following output
>>>>>>>>>>>
>>>>>>>>>>> MGSetup Level 0 2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02
>>>>>>>>>>> 6.0e+04 3.0e+01 1 0 3 11 2 1 0 3 11 2 0
>>>>>>>>>>> MGSmooth Level 0 9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03
>>>>>>>>>>> 1.7e+03 6.4e+02 33 71 28 3 34 35 71 28 3 35 415
>>>>>>>>>>> MGSetup Level 1 2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02
>>>>>>>>>>> 3.1e+04 2.3e+01 0 0 1 3 1 0 0 1 3 1 0
>>>>>>>>>>> MGSmooth Level 1 18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03
>>>>>>>>>>> 2.9e+03 3.6e+02 16 28 15 3 19 17 28 15 3 19 339
>>>>>>>>>>> MGResid Level 1 9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02
>>>>>>>>>>> 2.7e+03 9.0e+00 0 0 2 0 0 0 0 2 0 0 70
>>>>>>>>>>> MGInterp Level 1 18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02
>>>>>>>>>>> 1.1e+03 1.8e+01 0 0 4 0 1 0 0 4 0 1 37
>>>>>>>>>>>
>>>>>>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>>>>>>>
>>>>>>>>>>> Best
>>>>>>>>>>>
>>>>>>>>>>> Yuqi
>>>>>>>>>>>
>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith
>>>>>>>>>>>> <bsmith at mcs.anl.gov>)
>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part
>>>>>>>>>>>> of the basic PCMG. Are you sure you are calling
>>>>>>>>>>>> SNESSetFromOptions()?
>>>>>>>>>>>>
>>>>>>>>>>>> Barry
>>>>>>>>>>>>
>>>>>>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi, Mark,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you so much for your suggestion.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The problem 1 is resolved by avoiding calling
>>>>>>>>>>>>> PCMGSetNumberSmoothUp.
>>>>>>>>>>>>>
>>>>>>>>>>>>> But since I am using the unstructured grid in my application, I
>>>>>>>>>>>>> didn't use DA or dmmg, so -pc_mg_log didn't give any level
>>>>>>>>>>>>> information. I try to run my code using -info with 1 processor,
>>>>>>>>>>>>> and I find out some interesting issues.
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>
