Thank you, Barry.
Now I put in the KSPSetOperators() in the ComputeJacobian() routine. It works.
And I found out some interesting things, it also triggers the LU factorization
for the fine grid smoothers. Now I get 2 MatLUFactorSym (one for coarse and one
the fine grid), and 4 MatLUFactorNum (two for coarse and two for fine) for 2
SNES iteration.
Can I also ask another question? Because MatPtAP is not supported for baij
matrices. I want to setup a routine to calculate coarse operator if the fine
grid matrices is of type baij. Is there any approach so that I can keep the
symbolic pattern and only redo the numeric part in each SNES, similar to
MatPtAPSymbolic and MatPtAPNumeric?
Best
Yuqi
---- Original message ----
>Date: Thu, 5 Apr 2012 21:35:58 -0500
>From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at
>mcs.anl.gov>)
>Subject: Re: [petsc-users] Questions about PCMG
>To: PETSc users list <petsc-users at mcs.anl.gov>
>
>
>On Apr 5, 2012, at 9:31 PM, Yuqi Wu wrote:
>
>> Hi, Barry,
>>
>> If I use the following routines to setup the coarse operator instead of
>> calling PCMGSetGalerkin()
>>
>> ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
>> ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J,
>> DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
>
> Are you calling this at every SNES iteration? You should call this inside
> your ComputeJacobian() function, is that where you call it? Passing the
> DIFFERENT_NONZERO_PATTERN will also trigger a new symbolic factorization also.
>
> The refactorization fo the find grid smoother not being redone only effects
> the fine grid smoother, not the coarse grid.
> Barry
>
>
>>
>> Then in the case I have 2 SNES iteration, the MatLUFactorSym and
>> MatLUFactorNum are only called twice in the program. Is this issue also
>> related to the refactorization issue for the fine grid smoother?
>>
>> Best
>>
>> Yuqi
>>
>>
>>
>> ---- Original message ----
>>> Date: Thu, 5 Apr 2012 14:10:02 -0500
>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith
>>> at mcs.anl.gov>)
>>> Subject: Re: [petsc-users] Questions about PCMG
>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>
>>>
>>> Yes, at least that is the intention.
>>>
>>> Barry
>>>
>>> On Apr 5, 2012, at 2:06 PM, Yuqi Wu wrote:
>>>
>>>> Hi, Barry,
>>>>
>>>> If my matrix is of type mpibaij, then I cannot use the Galerkin flag in
>>>> the PCMG, e.g. PCMGSetGalerkin(finepc,PETSC_TRUE);
>>>>
>>>> Otherwise, I got the following error messages in PETSc
>>>> [0]PETSC ERROR: No support for this operation for this object type!
>>>> [0]PETSC ERROR: Matrix of type <mpibaij> does not support PtAP!
>>>>
>>>> Can I use the following commands to setup the coarse operator
>>>>
>>>> ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
>>>> ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J,
>>>> DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
>>>>
>>>> Thank you.
>>>>
>>>> Best
>>>>
>>>> Yuqi
>>>>
>>>>
>>>> ---- Original message ----
>>>>> Date: Thu, 5 Apr 2012 08:10:03 -0500
>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith
>>>>> <bsmith at mcs.anl.gov>)
>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>
>>>>>
>>>>> It looks like the logic for that line has gotten so convoluted that it
>>>>> lost its purpose and is not triggering the rebuilding on the finest
>>>>> level. Thanks for finding this, we will investigate and get back to you.
>>>>>
>>>>>
>>>>> Barry
>>>>>
>>>>> If your code is not complicated and you are willing to send it to us at
>>>>> petsc-maint at mcs.anl.gov that will make it quicker for us to determine
>>>>> the problem.
>>>>>
>>>>>
>>>>> On Apr 4, 2012, at 11:55 PM, Yuqi Wu wrote:
>>>>>
>>>>>> Thank you Barry.
>>>>>>
>>>>>> I am using exact LU as subdomain solver, so I find out PCSetUp_LU() in
>>>>>> my debugging output. But there are only three PCSetUp_LU() was called in
>>>>>> my program. One in the down smoother of the first SNES, one in the
>>>>>> coarse solve of the first SNES, and one in the coarse solve of the
>>>>>> second SNES.
>>>>>>
>>>>>> In those PCSetUp_LU(), MatLUFactorSymbolic and MatLUFactorNumeric is
>>>>>> called. But I still have no ideas about the actions taken on the down
>>>>>> smoother of the second SNES.
>>>>>>
>>>>>> I try to setup a breakpoint for PCSetUp_MG(). I find out that the IF
>>>>>> statement shown in below is called in the first SNES but not the second
>>>>>> SNES. Is it has something related to this issue?
>>>>>>
>>>>>> /* If user did not provide fine grid operators OR operator was not
>>>>>> updated since last global KSPSetOperators() */
>>>>>> /* so use those from global PC */
>>>>>> /* Is this what we always want? What if user wants to keep old one? */
>>>>>> ierr =
>>>>>> KSPGetOperatorsSet(mglevels[n-1]->smoothd,PETSC_NULL,&opsset);CHKERRQ(ierr);
>>>>>> ierr = KSPGetPC(mglevels[0]->smoothd,&cpc);CHKERRQ(ierr);
>>>>>> ierr = KSPGetPC(mglevels[n-1]->smoothd,&mpc);CHKERRQ(ierr);
>>>>>> if (!opsset || ((cpc->setupcalled == 1) && (mpc->setupcalled == 2)) ||
>>>>>> ((mpc == cpc) && (mpc->setupcalled == 2))) {
>>>>>> ierr = PetscInfo(pc,"Using outer operators to define finest grid
>>>>>> operator \n because
>>>>>> PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not
>>>>>> called.\n");CHKERRQ(ierr);
>>>>>> ierr =
>>>>>> KSPSetOperators(mglevels[n-1]->smoothd,pc->mat,pc->pmat,pc->flag);CHKERRQ(ierr);
>>>>>> }
>>>>>>
>>>>>> Thank you.
>>>>>>
>>>>>> Yuqi
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---- Original message ----
>>>>>>> Date: Wed, 4 Apr 2012 19:51:23 -0500
>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith
>>>>>>> <bsmith at mcs.anl.gov>)
>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>
>>>>>>>
>>>>>>> Step through the PCSetUp() function using "next" and you will it first
>>>>>>> making the decision of whether to continue with the setup or simply
>>>>>>> return early.
>>>>>>>
>>>>>>> When it gets to the lines
>>>>>>>
>>>>>>> if (pc->ops->setup) {
>>>>>>> ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr);
>>>>>>> }
>>>>>>>
>>>>>>> don't use "next" instead use "step" and it will go into the function
>>>>>>> PCSetUp_ILU() and you can move through that one with "next"
>>>>>>>
>>>>>>>> I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in
>>>>>>>> the debugger.
>>>>>>>
>>>>>>> It has to be calling them. Did you use "break MatLUFactorSymbolic" in
>>>>>>> the debugger?
>>>>>>>
>>>>>>> Barry
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Apr 4, 2012, at 7:45 PM, Yuqi Wu wrote:
>>>>>>>
>>>>>>>> ---- Original message ----
>>>>>>>>> Date: Wed, 4 Apr 2012 13:55:55 -0500
>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith
>>>>>>>>> <bsmith at mcs.anl.gov>)
>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Note: In most applications the flag SAME_NONZERO_PATTERN is provided
>>>>>>>>> in the compute Jacobian routine, this means that the SYMBOLIC
>>>>>>>>> factorization needs to be only done ONCE per matrix; only the numeric
>>>>>>>>> factorization needs to be done when the nonzero values have changed
>>>>>>>>> (the symbolic need not be repeated). Are you using this flag? How
>>>>>>>>> many times in the NUMERIC factorization being done?
>>>>>>>>>
>>>>>>>>> When you run the program with -info it will print information of the
>>>>>>>>> form: (run on one process to make life simple)
>>>>>>>>>
>>>>>>>>> Setting PC with identical preconditioner\
>>>>>>>>> Setting up new PC
>>>>>>>>> Setting up PC with same nonzero pattern\
>>>>>>>>> Setting up PC with different nonzero pattern\n
>>>>>>>>>
>>>>>>>>> How many, and exactly what messages of this form are you getting?
>>>>>>>>>
>>>>>>>>> When all else fails you can run the program in the debugger to track
>>>>>>>>> what is happening and why.
>>>>>>>>>
>>>>>>>>> Put a breakpoint in PCSetUp() then each time it gets called use next
>>>>>>>>> to step through it to see what is happening.
>>>>>>>>>
>>>>>>>>> First thing to check, is PCSetUp() getting called on each level for
>>>>>>>>> each new SNES iteration?
>>>>>>>>>
>>>>>>>>> Second thing, if it is then why is it not triggering the new
>>>>>>>>> numerical factorization.
>>>>>>>>
>>>>>>>> I run the program in debugger. I got total 8 PCSetUp() in the program.
>>>>>>>> And I believe that PCSetup() is called on each level for each new SNES
>>>>>>>> iteration.
>>>>>>>>
>>>>>>>> But I don't know how to check why is it not triggering the new
>>>>>>>> numerical factorization in the debugger. I cannot find any
>>>>>>>> MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. Could
>>>>>>>> you give me some suggestions to find out the problem?
>>>>>>>>
>>>>>>>> Best
>>>>>>>>
>>>>>>>> Yuqi
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Barry
>>>>>>>>>
>>>>>>>>> On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>>>>>>>>>
>>>>>>>>>> Thanks, Adam.
>>>>>>>>>>
>>>>>>>>>> Yes. I am using the Galerkin coarse grids. But I am not sure whether
>>>>>>>>>> the coarse grid is not getting refactored in the second SNES solve
>>>>>>>>>> or the fine grid smoother is not getting refactored in the second
>>>>>>>>>> SNES solve.
>>>>>>>>>>
>>>>>>>>>> In the -info output attached in the previous email, the fine grid
>>>>>>>>>> matrix is of size 11585 by 11585, and the coarse grid matrix is of
>>>>>>>>>> size 4186 by 4186. In the -info output, I found out three
>>>>>>>>>> MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two
>>>>>>>>>> for coarse matrix.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5
>>>>>>>>>> needed 11.401
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or
>>>>>>>>>> use
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit
>>>>>>>>>> used: 5. Using Inode routines
>>>>>>>>>>
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5
>>>>>>>>>> needed 7.07175
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175
>>>>>>>>>> or use
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used:
>>>>>>>>>> 5. Using Inode routines
>>>>>>>>>>
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5
>>>>>>>>>> needed 7.07175
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175
>>>>>>>>>> or use
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used:
>>>>>>>>>> 5. Using Inode routines
>>>>>>>>>>
>>>>>>>>>> So I believe that the fine grid smoother is not getting refactored
>>>>>>>>>> in the second SNES solve.
>>>>>>>>>>
>>>>>>>>>> Best
>>>>>>>>>>
>>>>>>>>>> Yuqi
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---- Original message ----
>>>>>>>>>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F.
>>>>>>>>>>> Adams" <mark.adams at columbia.edu>)
>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>
>>>>>>>>>>> I would expect 4 calls to MatLUFactorSym here. It looks like the
>>>>>>>>>>> coarse grid is not getting refactored in the second SNES solve.
>>>>>>>>>>>
>>>>>>>>>>> Are you using Galerkin coarse grids? Perhaps you are not setting a
>>>>>>>>>>> new coarse grid with KSPSetOperator and so MG does not bother
>>>>>>>>>>> refactoring it.
>>>>>>>>>>>
>>>>>>>>>>> Mark
>>>>>>>>>>>
>>>>>>>>>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>
>>>>>>>>>>>> Can I ask another question?
>>>>>>>>>>>>
>>>>>>>>>>>> In my log summary output, it shows that although there are two
>>>>>>>>>>>> SNES iteration and total 9 linear iterations. The functions
>>>>>>>>>>>> MatLUFactorSym and MatLUFactorNum are only called for three times.
>>>>>>>>>>>>
>>>>>>>>>>>> MatLUFactorSym 3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00
>>>>>>>>>>>> 0.0e+00 1.5e+01 1 0 0 0 2 1 0 0 0 2 0
>>>>>>>>>>>> MatLUFactorNum 3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00
>>>>>>>>>>>> 0.0e+00 0.0e+00 31 97 0 0 0 32 97 0 0 0 280
>>>>>>>>>>>>
>>>>>>>>>>>> I checked the -info output. It shows that One
>>>>>>>>>>>> MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the
>>>>>>>>>>>> first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the
>>>>>>>>>>>> coarse solve of the first SNES, and one
>>>>>>>>>>>> MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the
>>>>>>>>>>>> second SNES.
>>>>>>>>>>>>
>>>>>>>>>>>> Do you have any ideas why there are 9 multigrid iterations, but
>>>>>>>>>>>> only 3 MatLUFactorSymbolic calls in the program?
>>>>>>>>>>>>
>>>>>>>>>>>> Best
>>>>>>>>>>>>
>>>>>>>>>>>> Yuqi
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry
>>>>>>>>>>>>> Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are
>>>>>>>>>>>>> two MGSetUp on each level. Then a total of 9 multigrid iterations
>>>>>>>>>>>>> (in both linear solves together) hence 9 smoother on level 0
>>>>>>>>>>>>> (level 0 means coarse grid solve). One smooth down and one smooth
>>>>>>>>>>>>> up on level 1 hence 18 total smooths on level 1. 9 computation
>>>>>>>>>>>>> of residual on level 1 and 18 MgInterp because that logs both the
>>>>>>>>>>>>> restriction to level 0 and the interpolation back to level 1 and
>>>>>>>>>>>>> 18 = 9 + 9.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi, Barry,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>>>>>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average),
>>>>>>>>>>>>>> last norm 0.000000e+00
>>>>>>>>>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average),
>>>>>>>>>>>>>> last norm 2.268574e-06.
>>>>>>>>>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average),
>>>>>>>>>>>>>> last norm 1.312605e-12.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And -pc_mg_log shows the following output
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> MGSetup Level 0 2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02
>>>>>>>>>>>>>> 6.0e+04 3.0e+01 1 0 3 11 2 1 0 3 11 2 0
>>>>>>>>>>>>>> MGSmooth Level 0 9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03
>>>>>>>>>>>>>> 1.7e+03 6.4e+02 33 71 28 3 34 35 71 28 3 35 415
>>>>>>>>>>>>>> MGSetup Level 1 2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02
>>>>>>>>>>>>>> 3.1e+04 2.3e+01 0 0 1 3 1 0 0 1 3 1 0
>>>>>>>>>>>>>> MGSmooth Level 1 18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03
>>>>>>>>>>>>>> 2.9e+03 3.6e+02 16 28 15 3 19 17 28 15 3 19 339
>>>>>>>>>>>>>> MGResid Level 1 9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02
>>>>>>>>>>>>>> 2.7e+03 9.0e+00 0 0 2 0 0 0 0 2 0 0 70
>>>>>>>>>>>>>> MGInterp Level 1 18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02
>>>>>>>>>>>>>> 1.1e+03 1.8e+01 0 0 4 0 1 0 0 4 0 1 37
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yuqi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry
>>>>>>>>>>>>>>> Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is
>>>>>>>>>>>>>>> part of the basic PCMG. Are you sure you are calling
>>>>>>>>>>>>>>> SNESSetFromOptions()?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi, Mark,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you so much for your suggestion.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The problem 1 is resolved by avoiding calling
>>>>>>>>>>>>>>>> PCMGSetNumberSmoothUp.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> But since I am using the unstructured grid in my application,
>>>>>>>>>>>>>>>> I didn't use DA or dmmg, so -pc_mg_log didn't give any level
>>>>>>>>>>>>>>>> information. I try to run my code using -info with 1
>>>>>>>>>>>>>>>> processor, and I find out some interesting issues.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>
