On Apr 5, 2012, at 9:31 PM, Yuqi Wu wrote:
> Hi, Barry,
>
> If I use the following routines to setup the coarse operator instead of
> calling PCMGSetGalerkin()
>
> ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
> ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J,
> DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
Are you calling this at every SNES iteration? You should call this inside
your ComputeJacobian() function, is that where you call it? Passing the
DIFFERENT_NONZERO_PATTERN will also trigger a new symbolic factorization also.
The refactorization fo the find grid smoother not being redone only effects
the fine grid smoother, not the coarse grid.
Barry
>
> Then in the case I have 2 SNES iteration, the MatLUFactorSym and
> MatLUFactorNum are only called twice in the program. Is this issue also
> related to the refactorization issue for the fine grid smoother?
>
> Best
>
> Yuqi
>
>
>
> ---- Original message ----
>> Date: Thu, 5 Apr 2012 14:10:02 -0500
>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith
>> at mcs.anl.gov>)
>> Subject: Re: [petsc-users] Questions about PCMG
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>
>>
>> Yes, at least that is the intention.
>>
>> Barry
>>
>> On Apr 5, 2012, at 2:06 PM, Yuqi Wu wrote:
>>
>>> Hi, Barry,
>>>
>>> If my matrix is of type mpibaij, then I cannot use the Galerkin flag in the
>>> PCMG, e.g. PCMGSetGalerkin(finepc,PETSC_TRUE);
>>>
>>> Otherwise, I got the following error messages in PETSc
>>> [0]PETSC ERROR: No support for this operation for this object type!
>>> [0]PETSC ERROR: Matrix of type <mpibaij> does not support PtAP!
>>>
>>> Can I use the following commands to setup the coarse operator
>>>
>>> ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
>>> ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J,
>>> DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
>>>
>>> Thank you.
>>>
>>> Best
>>>
>>> Yuqi
>>>
>>>
>>> ---- Original message ----
>>>> Date: Thu, 5 Apr 2012 08:10:03 -0500
>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith
>>>> at mcs.anl.gov>)
>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>
>>>>
>>>> It looks like the logic for that line has gotten so convoluted that it
>>>> lost its purpose and is not triggering the rebuilding on the finest level.
>>>> Thanks for finding this, we will investigate and get back to you.
>>>>
>>>>
>>>> Barry
>>>>
>>>> If your code is not complicated and you are willing to send it to us at
>>>> petsc-maint at mcs.anl.gov that will make it quicker for us to determine
>>>> the problem.
>>>>
>>>>
>>>> On Apr 4, 2012, at 11:55 PM, Yuqi Wu wrote:
>>>>
>>>>> Thank you Barry.
>>>>>
>>>>> I am using exact LU as subdomain solver, so I find out PCSetUp_LU() in my
>>>>> debugging output. But there are only three PCSetUp_LU() was called in my
>>>>> program. One in the down smoother of the first SNES, one in the coarse
>>>>> solve of the first SNES, and one in the coarse solve of the second SNES.
>>>>>
>>>>> In those PCSetUp_LU(), MatLUFactorSymbolic and MatLUFactorNumeric is
>>>>> called. But I still have no ideas about the actions taken on the down
>>>>> smoother of the second SNES.
>>>>>
>>>>> I try to setup a breakpoint for PCSetUp_MG(). I find out that the IF
>>>>> statement shown in below is called in the first SNES but not the second
>>>>> SNES. Is it has something related to this issue?
>>>>>
>>>>> /* If user did not provide fine grid operators OR operator was not
>>>>> updated since last global KSPSetOperators() */
>>>>> /* so use those from global PC */
>>>>> /* Is this what we always want? What if user wants to keep old one? */
>>>>> ierr =
>>>>> KSPGetOperatorsSet(mglevels[n-1]->smoothd,PETSC_NULL,&opsset);CHKERRQ(ierr);
>>>>> ierr = KSPGetPC(mglevels[0]->smoothd,&cpc);CHKERRQ(ierr);
>>>>> ierr = KSPGetPC(mglevels[n-1]->smoothd,&mpc);CHKERRQ(ierr);
>>>>> if (!opsset || ((cpc->setupcalled == 1) && (mpc->setupcalled == 2)) ||
>>>>> ((mpc == cpc) && (mpc->setupcalled == 2))) {
>>>>> ierr = PetscInfo(pc,"Using outer operators to define finest grid
>>>>> operator \n because
>>>>> PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not
>>>>> called.\n");CHKERRQ(ierr);
>>>>> ierr =
>>>>> KSPSetOperators(mglevels[n-1]->smoothd,pc->mat,pc->pmat,pc->flag);CHKERRQ(ierr);
>>>>> }
>>>>>
>>>>> Thank you.
>>>>>
>>>>> Yuqi
>>>>>
>>>>>
>>>>>
>>>>> ---- Original message ----
>>>>>> Date: Wed, 4 Apr 2012 19:51:23 -0500
>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith
>>>>>> <bsmith at mcs.anl.gov>)
>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>
>>>>>>
>>>>>> Step through the PCSetUp() function using "next" and you will it first
>>>>>> making the decision of whether to continue with the setup or simply
>>>>>> return early.
>>>>>>
>>>>>> When it gets to the lines
>>>>>>
>>>>>> if (pc->ops->setup) {
>>>>>> ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr);
>>>>>> }
>>>>>>
>>>>>> don't use "next" instead use "step" and it will go into the function
>>>>>> PCSetUp_ILU() and you can move through that one with "next"
>>>>>>
>>>>>>> I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in
>>>>>>> the debugger.
>>>>>>
>>>>>> It has to be calling them. Did you use "break MatLUFactorSymbolic" in
>>>>>> the debugger?
>>>>>>
>>>>>> Barry
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Apr 4, 2012, at 7:45 PM, Yuqi Wu wrote:
>>>>>>
>>>>>>> ---- Original message ----
>>>>>>>> Date: Wed, 4 Apr 2012 13:55:55 -0500
>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith
>>>>>>>> <bsmith at mcs.anl.gov>)
>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>
>>>>>>>>
>>>>>>>> Note: In most applications the flag SAME_NONZERO_PATTERN is provided
>>>>>>>> in the compute Jacobian routine, this means that the SYMBOLIC
>>>>>>>> factorization needs to be only done ONCE per matrix; only the numeric
>>>>>>>> factorization needs to be done when the nonzero values have changed
>>>>>>>> (the symbolic need not be repeated). Are you using this flag? How many
>>>>>>>> times in the NUMERIC factorization being done?
>>>>>>>>
>>>>>>>> When you run the program with -info it will print information of the
>>>>>>>> form: (run on one process to make life simple)
>>>>>>>>
>>>>>>>> Setting PC with identical preconditioner\
>>>>>>>> Setting up new PC
>>>>>>>> Setting up PC with same nonzero pattern\
>>>>>>>> Setting up PC with different nonzero pattern\n
>>>>>>>>
>>>>>>>> How many, and exactly what messages of this form are you getting?
>>>>>>>>
>>>>>>>> When all else fails you can run the program in the debugger to track
>>>>>>>> what is happening and why.
>>>>>>>>
>>>>>>>> Put a breakpoint in PCSetUp() then each time it gets called use next
>>>>>>>> to step through it to see what is happening.
>>>>>>>>
>>>>>>>> First thing to check, is PCSetUp() getting called on each level for
>>>>>>>> each new SNES iteration?
>>>>>>>>
>>>>>>>> Second thing, if it is then why is it not triggering the new numerical
>>>>>>>> factorization.
>>>>>>>
>>>>>>> I run the program in debugger. I got total 8 PCSetUp() in the program.
>>>>>>> And I believe that PCSetup() is called on each level for each new SNES
>>>>>>> iteration.
>>>>>>>
>>>>>>> But I don't know how to check why is it not triggering the new
>>>>>>> numerical factorization in the debugger. I cannot find any
>>>>>>> MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. Could
>>>>>>> you give me some suggestions to find out the problem?
>>>>>>>
>>>>>>> Best
>>>>>>>
>>>>>>> Yuqi
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Barry
>>>>>>>>
>>>>>>>> On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>>>>>>>>
>>>>>>>>> Thanks, Adam.
>>>>>>>>>
>>>>>>>>> Yes. I am using the Galerkin coarse grids. But I am not sure whether
>>>>>>>>> the coarse grid is not getting refactored in the second SNES solve or
>>>>>>>>> the fine grid smoother is not getting refactored in the second SNES
>>>>>>>>> solve.
>>>>>>>>>
>>>>>>>>> In the -info output attached in the previous email, the fine grid
>>>>>>>>> matrix is of size 11585 by 11585, and the coarse grid matrix is of
>>>>>>>>> size 4186 by 4186. In the -info output, I found out three
>>>>>>>>> MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two
>>>>>>>>> for coarse matrix.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5
>>>>>>>>> needed 11.401
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or
>>>>>>>>> use
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used:
>>>>>>>>> 5. Using Inode routines
>>>>>>>>>
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5
>>>>>>>>> needed 7.07175
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or
>>>>>>>>> use
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used:
>>>>>>>>> 5. Using Inode routines
>>>>>>>>>
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5
>>>>>>>>> needed 7.07175
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or
>>>>>>>>> use
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used:
>>>>>>>>> 5. Using Inode routines
>>>>>>>>>
>>>>>>>>> So I believe that the fine grid smoother is not getting refactored in
>>>>>>>>> the second SNES solve.
>>>>>>>>>
>>>>>>>>> Best
>>>>>>>>>
>>>>>>>>> Yuqi
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---- Original message ----
>>>>>>>>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F.
>>>>>>>>>> Adams" <mark.adams at columbia.edu>)
>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>
>>>>>>>>>> I would expect 4 calls to MatLUFactorSym here. It looks like the
>>>>>>>>>> coarse grid is not getting refactored in the second SNES solve.
>>>>>>>>>>
>>>>>>>>>> Are you using Galerkin coarse grids? Perhaps you are not setting a
>>>>>>>>>> new coarse grid with KSPSetOperator and so MG does not bother
>>>>>>>>>> refactoring it.
>>>>>>>>>>
>>>>>>>>>> Mark
>>>>>>>>>>
>>>>>>>>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>>>>>>>>>
>>>>>>>>>>> Thank you.
>>>>>>>>>>>
>>>>>>>>>>> Can I ask another question?
>>>>>>>>>>>
>>>>>>>>>>> In my log summary output, it shows that although there are two SNES
>>>>>>>>>>> iteration and total 9 linear iterations. The functions
>>>>>>>>>>> MatLUFactorSym and MatLUFactorNum are only called for three times.
>>>>>>>>>>>
>>>>>>>>>>> MatLUFactorSym 3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00
>>>>>>>>>>> 0.0e+00 1.5e+01 1 0 0 0 2 1 0 0 0 2 0
>>>>>>>>>>> MatLUFactorNum 3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00
>>>>>>>>>>> 0.0e+00 0.0e+00 31 97 0 0 0 32 97 0 0 0 280
>>>>>>>>>>>
>>>>>>>>>>> I checked the -info output. It shows that One
>>>>>>>>>>> MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the
>>>>>>>>>>> first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the
>>>>>>>>>>> coarse solve of the first SNES, and one
>>>>>>>>>>> MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the
>>>>>>>>>>> second SNES.
>>>>>>>>>>>
>>>>>>>>>>> Do you have any ideas why there are 9 multigrid iterations, but
>>>>>>>>>>> only 3 MatLUFactorSymbolic calls in the program?
>>>>>>>>>>>
>>>>>>>>>>> Best
>>>>>>>>>>>
>>>>>>>>>>> Yuqi
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith
>>>>>>>>>>>> <bsmith at mcs.anl.gov>)
>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are
>>>>>>>>>>>> two MGSetUp on each level. Then a total of 9 multigrid iterations
>>>>>>>>>>>> (in both linear solves together) hence 9 smoother on level 0
>>>>>>>>>>>> (level 0 means coarse grid solve). One smooth down and one smooth
>>>>>>>>>>>> up on level 1 hence 18 total smooths on level 1. 9 computation of
>>>>>>>>>>>> residual on level 1 and 18 MgInterp because that logs both the
>>>>>>>>>>>> restriction to level 0 and the interpolation back to level 1 and
>>>>>>>>>>>> 18 = 9 + 9.
>>>>>>>>>>>>
>>>>>>>>>>>> Barry
>>>>>>>>>>>>
>>>>>>>>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi, Barry,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>>>>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average),
>>>>>>>>>>>>> last norm 0.000000e+00
>>>>>>>>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average),
>>>>>>>>>>>>> last norm 2.268574e-06.
>>>>>>>>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average),
>>>>>>>>>>>>> last norm 1.312605e-12.
>>>>>>>>>>>>>
>>>>>>>>>>>>> And -pc_mg_log shows the following output
>>>>>>>>>>>>>
>>>>>>>>>>>>> MGSetup Level 0 2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02
>>>>>>>>>>>>> 6.0e+04 3.0e+01 1 0 3 11 2 1 0 3 11 2 0
>>>>>>>>>>>>> MGSmooth Level 0 9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03
>>>>>>>>>>>>> 1.7e+03 6.4e+02 33 71 28 3 34 35 71 28 3 35 415
>>>>>>>>>>>>> MGSetup Level 1 2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02
>>>>>>>>>>>>> 3.1e+04 2.3e+01 0 0 1 3 1 0 0 1 3 1 0
>>>>>>>>>>>>> MGSmooth Level 1 18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03
>>>>>>>>>>>>> 2.9e+03 3.6e+02 16 28 15 3 19 17 28 15 3 19 339
>>>>>>>>>>>>> MGResid Level 1 9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02
>>>>>>>>>>>>> 2.7e+03 9.0e+00 0 0 2 0 0 0 0 2 0 0 70
>>>>>>>>>>>>> MGInterp Level 1 18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02
>>>>>>>>>>>>> 1.1e+03 1.8e+01 0 0 4 0 1 0 0 4 0 1 37
>>>>>>>>>>>>>
>>>>>>>>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yuqi
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry
>>>>>>>>>>>>>> Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is
>>>>>>>>>>>>>> part of the basic PCMG. Are you sure you are calling
>>>>>>>>>>>>>> SNESSetFromOptions()?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi, Mark,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you so much for your suggestion.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The problem 1 is resolved by avoiding calling
>>>>>>>>>>>>>>> PCMGSetNumberSmoothUp.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> But since I am using the unstructured grid in my application, I
>>>>>>>>>>>>>>> didn't use DA or dmmg, so -pc_mg_log didn't give any level
>>>>>>>>>>>>>>> information. I try to run my code using -info with 1 processor,
>>>>>>>>>>>>>>> and I find out some interesting issues.
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>
