Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

[Jed Brown]

"Klaij, Christiaan" <c.kl...@marin.nl> writes:

> Hi Jed,
>
> Thanks for clarifying, I understand the two-sided bound and the
> condition number. So for left preconditioning we would have P_L A
> instead of A in this bound and P_L r instead of r. For right
> preconditioning it is less obvious to me. How much would that
> loosen the bound?

Right preconditioning works with the *unpreconditioned* residual norm.

> Chris
>
>
> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 | mailto:c.kl...@marin.nl | http://www.marin.nl
>
> MARIN news: 
> http://www.marin.nl/web/News/News-items/Improved-modelling-of-sheet-cavitation-dynamics-on-Delft-Twistll-Hydrofoil-1.htm
>
> 
> From: Jed Brown <j...@jedbrown.org>
> Sent: Thursday, August 24, 2017 6:01 PM
> To: Klaij, Christiaan; Matthew Knepley; Barry Smith
> Cc: petsc-users@mcs.anl.gov
> Subject: Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?
>
> "Klaij, Christiaan" <c.kl...@marin.nl> writes:
>
>> Matt,
>>
>> Thanks, I can understand the lower condition number of P A, but
>> what about r? Doesn't that change to P r and if so why can we
>> assume that ||r|| and ||P r|| have the same order?
>
> Matt's equation was of course wrong in a literal sense, but is based on
> the right moral convictions.  We have
>
>   r = A (x - x_exact)
>
> which implies that
>
>   ||r|| <= ||A|| || x - x_exact ||.
>
> We also have
>
>   x - x_exact = A^{-1} r
>
> so
>
>   || x - x_exact || <= || A^{-1} || ||r||.
>
> Combining these gives the two-sided bound
>
>   || A ||^{-1} ||r|| <= || x - x_exact || <= || A^{-1} || ||r||.
>
> The ratio of the high and low bounds is the condition number.  Our
> convergence tolerance controls the norm of the residual (in a relative
> or absolute sense).  If the condition number is smaller, we get tighter
> control on the error.  If you are confident that your preconditioner
> reduces the condition number, it makes sense to measure convergence in
> the preconditioned norm (this is natural with left preconditioning for
> GMRES), in which the bounds above are in terms of the preconditioned
> operator.
>
> The main reason for PETSc to keep the preconditioned norm as a default
> is that many users, especially beginners, produce poorly scaled
> equations, e.g., with penalty boundary conditions or with disparate
> scales between fields with different units (displacement, pressure,
> velocity, energy, etc.).  You will often see the unpreconditioned
> residual drop by 10 orders of magnitude on the first iteration because
> the penalty boundary conditions are satisfied despite the solution being
> completely wrong inside the domain.
>
> On the other hand, the preconditioned residual causes misdiagnosis of
> convergence if the preconditioner is (nearly) singular, as is the case
> with some preconditioners applied to saddle point problems, for example.
> But this is easy to identify using -ksp_monitor_true_residual.
>
>
>>
>> Chris
>>
>>
>> dr. ir. Christiaan Klaij | Senior Researcher | Research & Development
>> MARIN | T +31 317 49 33 44 | c.kl...@marin.nl<mailto:c.kl...@marin.nl> | 
>> www.marin.nl<http://www.marin.nl>
>>
>> [LinkedIn]<https://www.linkedin.com/company/marin> [YouTube] 
>> <http://www.youtube.com/marinmultimedia>  [Twitter] 
>> <https://twitter.com/MARIN_nieuws>  [Facebook] 
>> <https://www.facebook.com/marin.wageningen>
>> MARIN news: New C-DRONE - for undisturbed wave spectrum 
>> measurements<http://www.marin.nl/web/News/News-items/New-CDRONE-for-undisturbed-wave-spectrum-measurements-1.htm>
>>
>> 
>> From: Matthew Knepley <knep...@gmail.com>
>> Sent: Wednesday, August 23, 2017 8:37 AM
>> To: Barry Smith
>> Cc: Klaij, Christiaan; petsc-users@mcs.anl.gov
>> Subject: Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?
>>
>> On Wed, Aug 23, 2017 at 2:30 AM, Barry Smith 
>> <bsm...@mcs.anl.gov<mailto:bsm...@mcs.anl.gov>> wrote:
>>
>>Some argue that the preconditioned residual is "closer to" the norm of 
>> the error than the unpreconditioned norm. I don't have a solid mathematical 
>> reason to prefer left preconditioning with the preconditioned norm.
>>
>> Because you have || x - x_exact || < k(A) || r ||
>>
>> where r is the residual and k is the condition number of A. If instead of A 
>> you use P A, which we assume has a lower condition 

Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

[Klaij, Christiaan]

Hi Jed,

Thanks for clarifying, I understand the two-sided bound and the
condition number. So for left preconditioning we would have P_L A
instead of A in this bound and P_L r instead of r. For right
preconditioning it is less obvious to me. How much would that
loosen the bound?

Chris


dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
MARIN | T +31 317 49 33 44 | mailto:c.kl...@marin.nl | http://www.marin.nl

MARIN news: 
http://www.marin.nl/web/News/News-items/Improved-modelling-of-sheet-cavitation-dynamics-on-Delft-Twistll-Hydrofoil-1.htm


From: Jed Brown <j...@jedbrown.org>
Sent: Thursday, August 24, 2017 6:01 PM
To: Klaij, Christiaan; Matthew Knepley; Barry Smith
Cc: petsc-users@mcs.anl.gov
Subject: Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

"Klaij, Christiaan" <c.kl...@marin.nl> writes:

> Matt,
>
> Thanks, I can understand the lower condition number of P A, but
> what about r? Doesn't that change to P r and if so why can we
> assume that ||r|| and ||P r|| have the same order?

Matt's equation was of course wrong in a literal sense, but is based on
the right moral convictions.  We have

  r = A (x - x_exact)

which implies that

  ||r|| <= ||A|| || x - x_exact ||.

We also have

  x - x_exact = A^{-1} r

so

  || x - x_exact || <= || A^{-1} || ||r||.

Combining these gives the two-sided bound

  || A ||^{-1} ||r|| <= || x - x_exact || <= || A^{-1} || ||r||.

The ratio of the high and low bounds is the condition number.  Our
convergence tolerance controls the norm of the residual (in a relative
or absolute sense).  If the condition number is smaller, we get tighter
control on the error.  If you are confident that your preconditioner
reduces the condition number, it makes sense to measure convergence in
the preconditioned norm (this is natural with left preconditioning for
GMRES), in which the bounds above are in terms of the preconditioned
operator.

The main reason for PETSc to keep the preconditioned norm as a default
is that many users, especially beginners, produce poorly scaled
equations, e.g., with penalty boundary conditions or with disparate
scales between fields with different units (displacement, pressure,
velocity, energy, etc.).  You will often see the unpreconditioned
residual drop by 10 orders of magnitude on the first iteration because
the penalty boundary conditions are satisfied despite the solution being
completely wrong inside the domain.

On the other hand, the preconditioned residual causes misdiagnosis of
convergence if the preconditioner is (nearly) singular, as is the case
with some preconditioners applied to saddle point problems, for example.
But this is easy to identify using -ksp_monitor_true_residual.


>
> Chris
>
>
> dr. ir. Christiaan Klaij | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 | c.kl...@marin.nl<mailto:c.kl...@marin.nl> | 
> www.marin.nl<http://www.marin.nl>
>
> [LinkedIn]<https://www.linkedin.com/company/marin> [YouTube] 
> <http://www.youtube.com/marinmultimedia>  [Twitter] 
> <https://twitter.com/MARIN_nieuws>  [Facebook] 
> <https://www.facebook.com/marin.wageningen>
> MARIN news: New C-DRONE - for undisturbed wave spectrum 
> measurements<http://www.marin.nl/web/News/News-items/New-CDRONE-for-undisturbed-wave-spectrum-measurements-1.htm>
>
> 
> From: Matthew Knepley <knep...@gmail.com>
> Sent: Wednesday, August 23, 2017 8:37 AM
> To: Barry Smith
> Cc: Klaij, Christiaan; petsc-users@mcs.anl.gov
> Subject: Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?
>
> On Wed, Aug 23, 2017 at 2:30 AM, Barry Smith 
> <bsm...@mcs.anl.gov<mailto:bsm...@mcs.anl.gov>> wrote:
>
>Some argue that the preconditioned residual is "closer to" the norm of the 
> error than the unpreconditioned norm. I don't have a solid mathematical 
> reason to prefer left preconditioning with the preconditioned norm.
>
> Because you have || x - x_exact || < k(A) || r ||
>
> where r is the residual and k is the condition number of A. If instead of A 
> you use P A, which we assume has a lower condition number, then
> this bound is improved.
>
>   Thanks,
>
>  Matt
>
>
>Barry
>
>
>
>> On Aug 22, 2017, at 11:27 PM, Klaij, Christiaan 
>> <c.kl...@marin.nl<mailto:c.kl...@marin.nl>> wrote:
>>
>> Barry,
>>
>> Thanks for the explanation.
>>
>> We do have some rare cases that give false convergence, but
>> decided to use
>>
>> CALL KSPSetNormType(ksp,KSP_NORM_UNPRECONDITIONED,ierr)
>>
>> so that convergence is always based on the true residual. Our
>> results are much more 

Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

[Jed Brown]

"Klaij, Christiaan" <c.kl...@marin.nl> writes:

> Matt,
>
> Thanks, I can understand the lower condition number of P A, but
> what about r? Doesn't that change to P r and if so why can we
> assume that ||r|| and ||P r|| have the same order?

Matt's equation was of course wrong in a literal sense, but is based on
the right moral convictions.  We have

  r = A (x - x_exact)

which implies that

  ||r|| <= ||A|| || x - x_exact ||.

We also have

  x - x_exact = A^{-1} r

so

  || x - x_exact || <= || A^{-1} || ||r||.

Combining these gives the two-sided bound

  || A ||^{-1} ||r|| <= || x - x_exact || <= || A^{-1} || ||r||.

The ratio of the high and low bounds is the condition number.  Our
convergence tolerance controls the norm of the residual (in a relative
or absolute sense).  If the condition number is smaller, we get tighter
control on the error.  If you are confident that your preconditioner
reduces the condition number, it makes sense to measure convergence in
the preconditioned norm (this is natural with left preconditioning for
GMRES), in which the bounds above are in terms of the preconditioned
operator.

The main reason for PETSc to keep the preconditioned norm as a default
is that many users, especially beginners, produce poorly scaled
equations, e.g., with penalty boundary conditions or with disparate
scales between fields with different units (displacement, pressure,
velocity, energy, etc.).  You will often see the unpreconditioned
residual drop by 10 orders of magnitude on the first iteration because
the penalty boundary conditions are satisfied despite the solution being
completely wrong inside the domain.

On the other hand, the preconditioned residual causes misdiagnosis of
convergence if the preconditioner is (nearly) singular, as is the case
with some preconditioners applied to saddle point problems, for example.
But this is easy to identify using -ksp_monitor_true_residual.


>
> Chris
>
>
> dr. ir. Christiaan Klaij | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 | c.kl...@marin.nl<mailto:c.kl...@marin.nl> | 
> www.marin.nl<http://www.marin.nl>
>
> [LinkedIn]<https://www.linkedin.com/company/marin> [YouTube] 
> <http://www.youtube.com/marinmultimedia>  [Twitter] 
> <https://twitter.com/MARIN_nieuws>  [Facebook] 
> <https://www.facebook.com/marin.wageningen>
> MARIN news: New C-DRONE - for undisturbed wave spectrum 
> measurements<http://www.marin.nl/web/News/News-items/New-CDRONE-for-undisturbed-wave-spectrum-measurements-1.htm>
>
> 
> From: Matthew Knepley <knep...@gmail.com>
> Sent: Wednesday, August 23, 2017 8:37 AM
> To: Barry Smith
> Cc: Klaij, Christiaan; petsc-users@mcs.anl.gov
> Subject: Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?
>
> On Wed, Aug 23, 2017 at 2:30 AM, Barry Smith 
> <bsm...@mcs.anl.gov<mailto:bsm...@mcs.anl.gov>> wrote:
>
>Some argue that the preconditioned residual is "closer to" the norm of the 
> error than the unpreconditioned norm. I don't have a solid mathematical 
> reason to prefer left preconditioning with the preconditioned norm.
>
> Because you have || x - x_exact || < k(A) || r ||
>
> where r is the residual and k is the condition number of A. If instead of A 
> you use P A, which we assume has a lower condition number, then
> this bound is improved.
>
>   Thanks,
>
>  Matt
>
>
>Barry
>
>
>
>> On Aug 22, 2017, at 11:27 PM, Klaij, Christiaan 
>> <c.kl...@marin.nl<mailto:c.kl...@marin.nl>> wrote:
>>
>> Barry,
>>
>> Thanks for the explanation.
>>
>> We do have some rare cases that give false convergence, but
>> decided to use
>>
>> CALL KSPSetNormType(ksp,KSP_NORM_UNPRECONDITIONED,ierr)
>>
>> so that convergence is always based on the true residual. Our
>> results are much more consistent now. So that could have been
>> your protection against the rare case as well, right? Why do you
>> prefer left preconditioning?
>>
>> Chris
>>
>>
>>
>> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
>> MARIN | T +31 317 49 33 44<tel:%2B31%20317%2049%2033%2044> | 
>> mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl> | http://www.marin.nl
>>
>> MARIN news: 
>> http://www.marin.nl/web/News/News-items/BlueWeek-October-911-Rostock.htm
>>
>> 
>> From: Barry Smith <bsm...@mcs.anl.gov<mailto:bsm...@mcs.anl.gov>>
>> Sent: Tuesday, August 22, 2017 6:25 PM
>> To: Klaij, Christiaan
>> Cc: petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov&g

Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

[Matthew Knepley]

On Thu, Aug 24, 2017 at 2:29 AM, Klaij, Christiaan <c.kl...@marin.nl> wrote:

> Matt,
>
> Thanks, I can understand the lower condition number of P A, but
> what about r? Doesn't that change to P r and if so why can we
> assume that ||r|| and ||P r|| have the same order?
>
r and Pr are the things we can control. We make them whatever we want.

   Matt

> Chris
>
>
> dr. ir. Christiaan Klaij | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 <+31%20317%20493%20344> | c.kl...@marin.nl |
> www.marin.nl
>
> [image: LinkedIn] <https://www.linkedin.com/company/marin> [image:
> YouTube] <http://www.youtube.com/marinmultimedia> [image: Twitter]
> <https://twitter.com/MARIN_nieuws> [image: Facebook]
> <https://www.facebook.com/marin.wageningen>
> MARIN news: New C-DRONE - for undisturbed wave spectrum measurements
> <http://www.marin.nl/web/News/News-items/New-CDRONE-for-undisturbed-wave-spectrum-measurements-1.htm>
>
> --
> *From:* Matthew Knepley <knep...@gmail.com>
> *Sent:* Wednesday, August 23, 2017 8:37 AM
> *To:* Barry Smith
> *Cc:* Klaij, Christiaan; petsc-users@mcs.anl.gov
> *Subject:* Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?
>
> On Wed, Aug 23, 2017 at 2:30 AM, Barry Smith <bsm...@mcs.anl.gov> wrote:
>
>>
>>Some argue that the preconditioned residual is "closer to" the norm of
>> the error than the unpreconditioned norm. I don't have a solid mathematical
>> reason to prefer left preconditioning with the preconditioned norm.
>
>
> Because you have || x - x_exact || < k(A) || r ||
>
> where r is the residual and k is the condition number of A. If instead of
> A you use P A, which we assume has a lower condition number, then
> this bound is improved.
>
>   Thanks,
>
>  Matt
>
>
>>
>>Barry
>>
>>
>>
>> > On Aug 22, 2017, at 11:27 PM, Klaij, Christiaan <c.kl...@marin.nl>
>> wrote:
>> >
>> > Barry,
>> >
>> > Thanks for the explanation.
>> >
>> > We do have some rare cases that give false convergence, but
>> > decided to use
>> >
>> > CALL KSPSetNormType(ksp,KSP_NORM_UNPRECONDITIONED,ierr)
>> >
>> > so that convergence is always based on the true residual. Our
>> > results are much more consistent now. So that could have been
>> > your protection against the rare case as well, right? Why do you
>> > prefer left preconditioning?
>> >
>> > Chris
>> >
>> >
>> >
>> > dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
>> > MARIN | T +31 317 49 33 44 | mailto:c.kl...@marin.nl |
>> http://www.marin.nl
>> >
>> > MARIN news: http://www.marin.nl/web/News/N
>> ews-items/BlueWeek-October-911-Rostock.htm
>> >
>> > 
>> > From: Barry Smith <bsm...@mcs.anl.gov>
>> > Sent: Tuesday, August 22, 2017 6:25 PM
>> > To: Klaij, Christiaan
>> > Cc: petsc-users@mcs.anl.gov
>> > Subject: Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?
>> >
>> >> On Aug 22, 2017, at 6:49 AM, Klaij, Christiaan <c.kl...@marin.nl>
>> wrote:
>> >>
>> >> We also faced this problem in our code. So I've added:
>> >>
>> >> CALL PetscOptionsSetValue(PETSC_NULL_OBJECT,"-sub_pc_factor_shift
>> _type","nonzero",ierr)
>> >>
>> >> since there seems to be no setter function for this (correct me
>> >> if I'm wrong). Then everythings fine again.
>> >>
>> >> Out of curiosity, what was the reason to change the default
>> >> behaviour?
>> >
>> >   The reason we changed this is that we would rather have a failure
>> that makes the user aware of a serious problem then to produce "garbage"
>> results. In some rare cases the shift can cause a huge jump in the
>> preconditioned residual which then decreases rapidly while the true
>> residual does not improve. This results in the KSP thinking it has
>> converged while in fact it has essentially garbage for an answer. Under the
>> previous model, where we shifted by default, users would in this rare case
>> think they had reasonable solutions when they did not.
>> >
>> >   For many users, such as yourself, the previous default behavior was
>> fine because you didn't have the "rare case" but we decided it was best to
>> protect against the rare case even though it would require other users such
>> as yourself to add the option.
>> >
>> >   Barry
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> http://www.caam.rice.edu/~mk51/
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

http://www.caam.rice.edu/~mk51/

Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

[Klaij, Christiaan]

Matt,

Thanks, I can understand the lower condition number of P A, but
what about r? Doesn't that change to P r and if so why can we
assume that ||r|| and ||P r|| have the same order?

Chris


dr. ir. Christiaan Klaij | Senior Researcher | Research & Development
MARIN | T +31 317 49 33 44 | c.kl...@marin.nl<mailto:c.kl...@marin.nl> | 
www.marin.nl<http://www.marin.nl>

[LinkedIn]<https://www.linkedin.com/company/marin> [YouTube] 
<http://www.youtube.com/marinmultimedia>  [Twitter] 
<https://twitter.com/MARIN_nieuws>  [Facebook] 
<https://www.facebook.com/marin.wageningen>
MARIN news: New C-DRONE - for undisturbed wave spectrum 
measurements<http://www.marin.nl/web/News/News-items/New-CDRONE-for-undisturbed-wave-spectrum-measurements-1.htm>


From: Matthew Knepley <knep...@gmail.com>
Sent: Wednesday, August 23, 2017 8:37 AM
To: Barry Smith
Cc: Klaij, Christiaan; petsc-users@mcs.anl.gov
Subject: Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

On Wed, Aug 23, 2017 at 2:30 AM, Barry Smith 
<bsm...@mcs.anl.gov<mailto:bsm...@mcs.anl.gov>> wrote:

   Some argue that the preconditioned residual is "closer to" the norm of the 
error than the unpreconditioned norm. I don't have a solid mathematical reason 
to prefer left preconditioning with the preconditioned norm.

Because you have || x - x_exact || < k(A) || r ||

where r is the residual and k is the condition number of A. If instead of A you 
use P A, which we assume has a lower condition number, then
this bound is improved.

  Thanks,

 Matt


   Barry



> On Aug 22, 2017, at 11:27 PM, Klaij, Christiaan 
> <c.kl...@marin.nl<mailto:c.kl...@marin.nl>> wrote:
>
> Barry,
>
> Thanks for the explanation.
>
> We do have some rare cases that give false convergence, but
> decided to use
>
> CALL KSPSetNormType(ksp,KSP_NORM_UNPRECONDITIONED,ierr)
>
> so that convergence is always based on the true residual. Our
> results are much more consistent now. So that could have been
> your protection against the rare case as well, right? Why do you
> prefer left preconditioning?
>
> Chris
>
>
>
> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44<tel:%2B31%20317%2049%2033%2044> | 
> mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl> | http://www.marin.nl
>
> MARIN news: 
> http://www.marin.nl/web/News/News-items/BlueWeek-October-911-Rostock.htm
>
> 
> From: Barry Smith <bsm...@mcs.anl.gov<mailto:bsm...@mcs.anl.gov>>
> Sent: Tuesday, August 22, 2017 6:25 PM
> To: Klaij, Christiaan
> Cc: petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>
> Subject: Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?
>
>> On Aug 22, 2017, at 6:49 AM, Klaij, Christiaan 
>> <c.kl...@marin.nl<mailto:c.kl...@marin.nl>> wrote:
>>
>> We also faced this problem in our code. So I've added:
>>
>> CALL 
>> PetscOptionsSetValue(PETSC_NULL_OBJECT,"-sub_pc_factor_shift_type","nonzero",ierr)
>>
>> since there seems to be no setter function for this (correct me
>> if I'm wrong). Then everythings fine again.
>>
>> Out of curiosity, what was the reason to change the default
>> behaviour?
>
>   The reason we changed this is that we would rather have a failure that 
> makes the user aware of a serious problem then to produce "garbage" results. 
> In some rare cases the shift can cause a huge jump in the preconditioned 
> residual which then decreases rapidly while the true residual does not 
> improve. This results in the KSP thinking it has converged while in fact it 
> has essentially garbage for an answer. Under the previous model, where we 
> shifted by default, users would in this rare case think they had reasonable 
> solutions when they did not.
>
>   For many users, such as yourself, the previous default behavior was fine 
> because you didn't have the "rare case" but we decided it was best to protect 
> against the rare case even though it would require other users such as 
> yourself to add the option.
>
>   Barry




--
What most experimenters take for granted before they begin their experiments is 
infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

http://www.caam.rice.edu/~mk51/

Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

[Barry Smith]

  In complicated scenarios with deep nesting of solvers it is sometimes not 
possible to set the options for the deeply nested solvers directly from the 
code with function calls. Thus we recommend using the options database for such 
cases. Note that one can provide options on the command line, in a file, with 
calls to PetscOptionsSetValue(), PetscOptionsInsertString(), 
PetscOptionsInsertFile()



> On Aug 22, 2017, at 11:51 PM, Klaij, Christiaan <c.kl...@marin.nl> wrote:
> 
> Lawrence,
> 
> Well, I was looking for something like SubPCFactorSetShiftType,
> or is it supposed to somehow "trickle down" to the sub pc's?
> 
> Or should I setup the pc, get the sub-pc's and then apply
> PCFactorSetShiftType on the sub pc's, something like
> 
> CALL KSPSetup(ksp)
> CALL PCBJacobiGetSubKSP(ksp, subksp)
> CALL KSPGetPC(subksp, subpc, ierr)
> CALL PCFactorSetShiftType(subpc,MAT_SHIFT_NONZERO,ierr)
> 
> Chris
> 
>> Message: 2
>> Date: Tue, 22 Aug 2017 15:17:58 +0100
>> From: Lawrence Mitchell <lawrence.mitch...@imperial.ac.uk>
>> To: petsc-users@mcs.anl.gov
>> Subject: Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and
>>3.7.x?
>> Message-ID: <14d4c158-95b1-96ab-90b7-b81d3239b...@imperial.ac.uk>
>> Content-Type: text/plain; charset=utf-8
>> 
>> 
>> 
>> On 22/08/17 14:49, Klaij, Christiaan wrote:
>>> CALL 
>>> PetscOptionsSetValue(PETSC_NULL_OBJECT,"-sub_pc_factor_shift_type","nonzero",ierr)
>>> 
>>> since there seems to be no setter function for this (correct me
>>> if I'm wrong).
>> 
>> I think you want PCFactorSetShiftType.
>> 
>> Lawrence
> 
> 
> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 | mailto:c.kl...@marin.nl | http://www.marin.nl
> 
> MARIN news: 
> http://www.marin.nl/web/News/News-items/AMT17-October-1113-Glasgow.htm
> 

Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

[Lawrence Mitchell]

On 23/08/17 07:51, Klaij, Christiaan wrote:
> Lawrence,
> 
> Well, I was looking for something like SubPCFactorSetShiftType,
> or is it supposed to somehow "trickle down" to the sub pc's?

Ah, I see.  No, the approach below is what you need.  Imagine that you
used a sub_pc inside a multigrid smoother, you'd then need an
MGLevelsSubPCFactorSetShiftType, which swiftly leads down the path of
madness.

> Or should I setup the pc, get the sub-pc's and then apply
> PCFactorSetShiftType on the sub pc's, something like
> 
> CALL KSPSetup(ksp)
> CALL PCBJacobiGetSubKSP(ksp, subksp)
> CALL KSPGetPC(subksp, subpc, ierr)
> CALL PCFactorSetShiftType(subpc,MAT_SHIFT_NONZERO,ierr)

Cheers,

Lawrence

Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

[Klaij, Christiaan]

Lawrence,

Well, I was looking for something like SubPCFactorSetShiftType,
or is it supposed to somehow "trickle down" to the sub pc's?

Or should I setup the pc, get the sub-pc's and then apply
PCFactorSetShiftType on the sub pc's, something like

CALL KSPSetup(ksp)
CALL PCBJacobiGetSubKSP(ksp, subksp)
CALL KSPGetPC(subksp, subpc, ierr)
CALL PCFactorSetShiftType(subpc,MAT_SHIFT_NONZERO,ierr)

Chris

> Message: 2
> Date: Tue, 22 Aug 2017 15:17:58 +0100
> From: Lawrence Mitchell <lawrence.mitch...@imperial.ac.uk>
> To: petsc-users@mcs.anl.gov
> Subject: Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and
> 3.7.x?
> Message-ID: <14d4c158-95b1-96ab-90b7-b81d3239b...@imperial.ac.uk>
> Content-Type: text/plain; charset=utf-8
>
>
>
> On 22/08/17 14:49, Klaij, Christiaan wrote:
> > CALL 
> > PetscOptionsSetValue(PETSC_NULL_OBJECT,"-sub_pc_factor_shift_type","nonzero",ierr)
> >
> > since there seems to be no setter function for this (correct me
> > if I'm wrong).
>
> I think you want PCFactorSetShiftType.
>
> Lawrence


dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
MARIN | T +31 317 49 33 44 | mailto:c.kl...@marin.nl | http://www.marin.nl

MARIN news: 
http://www.marin.nl/web/News/News-items/AMT17-October-1113-Glasgow.htm

Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

[Matthew Knepley]

On Wed, Aug 23, 2017 at 2:30 AM, Barry Smith <bsm...@mcs.anl.gov> wrote:

>
>Some argue that the preconditioned residual is "closer to" the norm of
> the error than the unpreconditioned norm. I don't have a solid mathematical
> reason to prefer left preconditioning with the preconditioned norm.


Because you have || x - x_exact || < k(A) || r ||

where r is the residual and k is the condition number of A. If instead of A
you use P A, which we assume has a lower condition number, then
this bound is improved.

  Thanks,

 Matt


>
>Barry
>
>
>
> > On Aug 22, 2017, at 11:27 PM, Klaij, Christiaan <c.kl...@marin.nl>
> wrote:
> >
> > Barry,
> >
> > Thanks for the explanation.
> >
> > We do have some rare cases that give false convergence, but
> > decided to use
> >
> > CALL KSPSetNormType(ksp,KSP_NORM_UNPRECONDITIONED,ierr)
> >
> > so that convergence is always based on the true residual. Our
> > results are much more consistent now. So that could have been
> > your protection against the rare case as well, right? Why do you
> > prefer left preconditioning?
> >
> > Chris
> >
> >
> >
> > dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
> > MARIN | T +31 317 49 33 44 | mailto:c.kl...@marin.nl |
> http://www.marin.nl
> >
> > MARIN news: http://www.marin.nl/web/News/News-items/BlueWeek-October-
> 911-Rostock.htm
> >
> > ________________________
> > From: Barry Smith <bsm...@mcs.anl.gov>
> > Sent: Tuesday, August 22, 2017 6:25 PM
> > To: Klaij, Christiaan
> > Cc: petsc-users@mcs.anl.gov
> > Subject: Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?
> >
> >> On Aug 22, 2017, at 6:49 AM, Klaij, Christiaan <c.kl...@marin.nl>
> wrote:
> >>
> >> We also faced this problem in our code. So I've added:
> >>
> >> CALL PetscOptionsSetValue(PETSC_NULL_OBJECT,"-sub_pc_factor_
> shift_type","nonzero",ierr)
> >>
> >> since there seems to be no setter function for this (correct me
> >> if I'm wrong). Then everythings fine again.
> >>
> >> Out of curiosity, what was the reason to change the default
> >> behaviour?
> >
> >   The reason we changed this is that we would rather have a failure that
> makes the user aware of a serious problem then to produce "garbage"
> results. In some rare cases the shift can cause a huge jump in the
> preconditioned residual which then decreases rapidly while the true
> residual does not improve. This results in the KSP thinking it has
> converged while in fact it has essentially garbage for an answer. Under the
> previous model, where we shifted by default, users would in this rare case
> think they had reasonable solutions when they did not.
> >
> >   For many users, such as yourself, the previous default behavior was
> fine because you didn't have the "rare case" but we decided it was best to
> protect against the rare case even though it would require other users such
> as yourself to add the option.
> >
> >   Barry
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

http://www.caam.rice.edu/~mk51/

Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

[Barry Smith]

   Some argue that the preconditioned residual is "closer to" the norm of the 
error than the unpreconditioned norm. I don't have a solid mathematical reason 
to prefer left preconditioning with the preconditioned norm.

   Barry



> On Aug 22, 2017, at 11:27 PM, Klaij, Christiaan <c.kl...@marin.nl> wrote:
> 
> Barry,
> 
> Thanks for the explanation.
> 
> We do have some rare cases that give false convergence, but
> decided to use
> 
> CALL KSPSetNormType(ksp,KSP_NORM_UNPRECONDITIONED,ierr)
> 
> so that convergence is always based on the true residual. Our
> results are much more consistent now. So that could have been
> your protection against the rare case as well, right? Why do you
> prefer left preconditioning?
> 
> Chris
> 
> 
> 
> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 | mailto:c.kl...@marin.nl | http://www.marin.nl
> 
> MARIN news: 
> http://www.marin.nl/web/News/News-items/BlueWeek-October-911-Rostock.htm
> 
> 
> From: Barry Smith <bsm...@mcs.anl.gov>
> Sent: Tuesday, August 22, 2017 6:25 PM
> To: Klaij, Christiaan
> Cc: petsc-users@mcs.anl.gov
> Subject: Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?
> 
>> On Aug 22, 2017, at 6:49 AM, Klaij, Christiaan <c.kl...@marin.nl> wrote:
>> 
>> We also faced this problem in our code. So I've added:
>> 
>> CALL 
>> PetscOptionsSetValue(PETSC_NULL_OBJECT,"-sub_pc_factor_shift_type","nonzero",ierr)
>> 
>> since there seems to be no setter function for this (correct me
>> if I'm wrong). Then everythings fine again.
>> 
>> Out of curiosity, what was the reason to change the default
>> behaviour?
> 
>   The reason we changed this is that we would rather have a failure that 
> makes the user aware of a serious problem then to produce "garbage" results. 
> In some rare cases the shift can cause a huge jump in the preconditioned 
> residual which then decreases rapidly while the true residual does not 
> improve. This results in the KSP thinking it has converged while in fact it 
> has essentially garbage for an answer. Under the previous model, where we 
> shifted by default, users would in this rare case think they had reasonable 
> solutions when they did not.
> 
>   For many users, such as yourself, the previous default behavior was fine 
> because you didn't have the "rare case" but we decided it was best to protect 
> against the rare case even though it would require other users such as 
> yourself to add the option.
> 
>   Barry

Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

[Klaij, Christiaan]

Barry,

Thanks for the explanation.

We do have some rare cases that give false convergence, but
decided to use

 CALL KSPSetNormType(ksp,KSP_NORM_UNPRECONDITIONED,ierr)

so that convergence is always based on the true residual. Our
results are much more consistent now. So that could have been
your protection against the rare case as well, right? Why do you
prefer left preconditioning?

Chris



dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
MARIN | T +31 317 49 33 44 | mailto:c.kl...@marin.nl | http://www.marin.nl

MARIN news: 
http://www.marin.nl/web/News/News-items/BlueWeek-October-911-Rostock.htm


From: Barry Smith <bsm...@mcs.anl.gov>
Sent: Tuesday, August 22, 2017 6:25 PM
To: Klaij, Christiaan
Cc: petsc-users@mcs.anl.gov
Subject: Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

> On Aug 22, 2017, at 6:49 AM, Klaij, Christiaan <c.kl...@marin.nl> wrote:
>
> We also faced this problem in our code. So I've added:
>
> CALL 
> PetscOptionsSetValue(PETSC_NULL_OBJECT,"-sub_pc_factor_shift_type","nonzero",ierr)
>
> since there seems to be no setter function for this (correct me
> if I'm wrong). Then everythings fine again.
>
> Out of curiosity, what was the reason to change the default
> behaviour?

   The reason we changed this is that we would rather have a failure that makes 
the user aware of a serious problem then to produce "garbage" results. In some 
rare cases the shift can cause a huge jump in the preconditioned residual which 
then decreases rapidly while the true residual does not improve. This results 
in the KSP thinking it has converged while in fact it has essentially garbage 
for an answer. Under the previous model, where we shifted by default, users 
would in this rare case think they had reasonable solutions when they did not.

   For many users, such as yourself, the previous default behavior was fine 
because you didn't have the "rare case" but we decided it was best to protect 
against the rare case even though it would require other users such as yourself 
to add the option.

   Barry

Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

[Lawrence Mitchell]

On 22/08/17 14:49, Klaij, Christiaan wrote:
> CALL 
> PetscOptionsSetValue(PETSC_NULL_OBJECT,"-sub_pc_factor_shift_type","nonzero",ierr)
> 
> since there seems to be no setter function for this (correct me
> if I'm wrong). 

I think you want PCFactorSetShiftType.

Lawrence

Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

[Klaij, Christiaan]

We also faced this problem in our code. So I've added:

CALL 
PetscOptionsSetValue(PETSC_NULL_OBJECT,"-sub_pc_factor_shift_type","nonzero",ierr)

since there seems to be no setter function for this (correct me
if I'm wrong). Then everythings fine again.

Out of curiosity, what was the reason to change the default
behaviour? It seems to have really reduced the robustness of
PCBJACOBI.

> Message: 2
> Date: Fri, 11 Aug 2017 12:28:43 -0500
> From: Barry Smith <bsm...@mcs.anl.gov>
> To: Gaetan Kenway <gaet...@gmail.com>
> Cc: petsc-users <petsc-users@mcs.anl.gov>
> Subject: Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and
> 3.7.x?
> Message-ID: <e09f4c23-527d-4f48-a2e9-ba6b338d3...@mcs.anl.gov>
> Content-Type: text/plain; charset="us-ascii"
>
>
>Run with the additional option
>
>  -sub_pc_factor_shift_type nonzero
>
> does this resolve the problem. We changed the default behavior for ILU when 
> it detects "zero" pivots.
>
> Please let us know if this resolves the problem and we'll update the changes 
> file.
>
>Barry


dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
MARIN | T +31 317 49 33 44 | mailto:c.kl...@marin.nl | http://www.marin.nl

MARIN news: 
http://www.marin.nl/web/News/News-items/Improved-modelling-of-sheet-cavitation-dynamics-on-Delft-Twistll-Hydrofoil-1.htm

Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

[Gaetan Kenway]

Interesting. The main thing is it's now sorted out and then solver is back
in production.

Thanks for your help

On Fri, Aug 11, 2017 at 1:05 PM, Barry Smith  wrote:

>
> > On Aug 11, 2017, at 2:43 PM, Gaetan Kenway  wrote:
> >
> > Huh. That's odd then. I was actually bisecting the petsc releases to
> narrow it down...I knew 3.3 was ok and 3.7 was not. So I tried 3.5 which
> was ok, and then 3.6 which was ok as well, leading to me to conclude the
> difference was between 3.6 and 3.7.
>
>Other people have also reported only seeing the problem with later
> versions. I think it is related to the default tolerances and it is just
> "pure luck" that it didn't need a shift with the intermediate versions. ILU
> with nearly zero pivots is a fragile issue.
>
>Barry
>
> >
> > On Fri, Aug 11, 2017 at 12:03 PM, Barry Smith 
> wrote:
> >
> >Thanks for confirming this. The change was actually in the 3.4
> release. I have updated the 3.4 changes file to include this change in both
> the maint and master branches.
> >
> > Barry
> >
> > > On Aug 11, 2017, at 12:47 PM, Gaetan Kenway  wrote:
> > >
> > > OK, so that was certainly it.  I vaguely recall reading something
> about this on the mailing list at one point in time, but couldn't find
> anything.
> > > I would definitely put something on the 3.7 change doc since I looked
> there first to see if anything stuck out.
> > >
> > > Thanks!
> > >
> > > On Fri, Aug 11, 2017 at 10:28 AM, Barry Smith 
> wrote:
> > >
> > >Run with the additional option
> > >
> > >  -sub_pc_factor_shift_type nonzero
> > >
> > > does this resolve the problem. We changed the default behavior for ILU
> when it detects "zero" pivots.
> > >
> > > Please let us know if this resolves the problem and we'll update the
> changes file.
> > >
> > >Barry
> > >
> > >
> > >
> > > > On Aug 11, 2017, at 12:14 PM, Gaetan Kenway 
> wrote:
> > > >
> > > > Hi All
> > > >
> > > > I'm in the process of updating a code that uses PETSc for solving
> linear systems for an unstructured CFD code. As of recently, it was using
> an ancient version (3.3). However, when I updated it up to 3.7.6 I ended up
> running into issues with one of the KSP solves. The remainder of the code
> is identical,
> > > > I've tracked the issue down to occurring between version 3.6.4 and
> version 3.7.0 . The same issue is present on the most recent version 3.7.6.
> > > >
> > > > Specifically the issue is that on the second iteration, on the 3.7
> version the KSP kicks out with KSP converged reason of -11 or
> KSP_DIVERGED_PCSETUP_FAILED . After that the two runs differ.  The KSPView
> for each of the two are given below which are identical, up to small
> formatting changes. There is still more I can track down, but I thought I
> would ask if someone knows what might have changed between these two
> versions which could save me a lot of time.
> > > >
> > > > Thanks,
> > > > Gaetan
> > > >
> > > > 3.6 KSP View:
> > > > KSP Object: 8 MPI processes
> > > >   type: gmres
> > > > GMRES: restart=3, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> > > > GMRES: happy breakdown tolerance 1e-30
> > > >   maximum iterations=3
> > > >   using preconditioner applied to right hand side for initial guess
> > > >   tolerances:  relative=1e-08, absolute=1e-20, divergence=1e+15
> > > >   left preconditioning
> > > >   using nonzero initial guess
> > > >   using PRECONDITIONED norm type for convergence test
> > > > PC Object: 8 MPI processes
> > > >   type: bjacobi
> > > > block Jacobi: number of blocks = 8
> > > > Local solve is same for all blocks, in the following KSP and PC
> objects:
> > > >   KSP Object:  (sub_)   1 MPI processes
> > > > type: preonly
> > > > maximum iterations=1, initial guess is zero
> > > > tolerances:  relative=1e-05, absolute=1e-50, divergence=1
> > > > left preconditioning
> > > > using NONE norm type for convergence test
> > > >   PC Object:  (sub_)   1 MPI processes
> > > > type: ilu
> > > >   ILU: out-of-place factorization
> > > >   0 levels of fill
> > > >   tolerance for zero pivot 2.22045e-14
> > > >   matrix ordering: natural
> > > >   factor fill ratio given 1, needed 1
> > > > Factored matrix follows:
> > > >   Mat Object:   1 MPI processes
> > > > type: seqaij
> > > > rows=46439, cols=46439
> > > > package used to perform factorization: petsc
> > > > total: nonzeros=502615, allocated nonzeros=502615
> > > > total number of mallocs used during MatSetValues calls =0
> > > >   not using I-node routines
> > > > linear system matrix = precond matrix:
> > > > Mat Object: 1 MPI processes
> > > >   type: seqaij
> > > >   rows=46439, cols=46439
> > > >   total: 

Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

[Barry Smith]

> On Aug 11, 2017, at 2:43 PM, Gaetan Kenway  wrote:
> 
> Huh. That's odd then. I was actually bisecting the petsc releases to narrow 
> it down...I knew 3.3 was ok and 3.7 was not. So I tried 3.5 which was ok, and 
> then 3.6 which was ok as well, leading to me to conclude the difference was 
> between 3.6 and 3.7. 

   Other people have also reported only seeing the problem with later versions. 
I think it is related to the default tolerances and it is just "pure luck" that 
it didn't need a shift with the intermediate versions. ILU with nearly zero 
pivots is a fragile issue.

   Barry

> 
> On Fri, Aug 11, 2017 at 12:03 PM, Barry Smith  wrote:
> 
>Thanks for confirming this. The change was actually in the 3.4 release. I 
> have updated the 3.4 changes file to include this change in both the maint 
> and master branches.
> 
> Barry
> 
> > On Aug 11, 2017, at 12:47 PM, Gaetan Kenway  wrote:
> >
> > OK, so that was certainly it.  I vaguely recall reading something about 
> > this on the mailing list at one point in time, but couldn't find anything.
> > I would definitely put something on the 3.7 change doc since I looked there 
> > first to see if anything stuck out.
> >
> > Thanks!
> >
> > On Fri, Aug 11, 2017 at 10:28 AM, Barry Smith  wrote:
> >
> >Run with the additional option
> >
> >  -sub_pc_factor_shift_type nonzero
> >
> > does this resolve the problem. We changed the default behavior for ILU when 
> > it detects "zero" pivots.
> >
> > Please let us know if this resolves the problem and we'll update the 
> > changes file.
> >
> >Barry
> >
> >
> >
> > > On Aug 11, 2017, at 12:14 PM, Gaetan Kenway  wrote:
> > >
> > > Hi All
> > >
> > > I'm in the process of updating a code that uses PETSc for solving linear 
> > > systems for an unstructured CFD code. As of recently, it was using an 
> > > ancient version (3.3). However, when I updated it up to 3.7.6 I ended up 
> > > running into issues with one of the KSP solves. The remainder of the code 
> > > is identical,
> > > I've tracked the issue down to occurring between version 3.6.4 and 
> > > version 3.7.0 . The same issue is present on the most recent version 
> > > 3.7.6.
> > >
> > > Specifically the issue is that on the second iteration, on the 3.7 
> > > version the KSP kicks out with KSP converged reason of -11 or  
> > > KSP_DIVERGED_PCSETUP_FAILED . After that the two runs differ.  The 
> > > KSPView for each of the two are given below which are identical, up to 
> > > small formatting changes. There is still more I can track down, but I 
> > > thought I would ask if someone knows what might have changed between 
> > > these two versions which could save me a lot of time.
> > >
> > > Thanks,
> > > Gaetan
> > >
> > > 3.6 KSP View:
> > > KSP Object: 8 MPI processes
> > >   type: gmres
> > > GMRES: restart=3, using Classical (unmodified) Gram-Schmidt 
> > > Orthogonalization with no iterative refinement
> > > GMRES: happy breakdown tolerance 1e-30
> > >   maximum iterations=3
> > >   using preconditioner applied to right hand side for initial guess
> > >   tolerances:  relative=1e-08, absolute=1e-20, divergence=1e+15
> > >   left preconditioning
> > >   using nonzero initial guess
> > >   using PRECONDITIONED norm type for convergence test
> > > PC Object: 8 MPI processes
> > >   type: bjacobi
> > > block Jacobi: number of blocks = 8
> > > Local solve is same for all blocks, in the following KSP and PC 
> > > objects:
> > >   KSP Object:  (sub_)   1 MPI processes
> > > type: preonly
> > > maximum iterations=1, initial guess is zero
> > > tolerances:  relative=1e-05, absolute=1e-50, divergence=1
> > > left preconditioning
> > > using NONE norm type for convergence test
> > >   PC Object:  (sub_)   1 MPI processes
> > > type: ilu
> > >   ILU: out-of-place factorization
> > >   0 levels of fill
> > >   tolerance for zero pivot 2.22045e-14
> > >   matrix ordering: natural
> > >   factor fill ratio given 1, needed 1
> > > Factored matrix follows:
> > >   Mat Object:   1 MPI processes
> > > type: seqaij
> > > rows=46439, cols=46439
> > > package used to perform factorization: petsc
> > > total: nonzeros=502615, allocated nonzeros=502615
> > > total number of mallocs used during MatSetValues calls =0
> > >   not using I-node routines
> > > linear system matrix = precond matrix:
> > > Mat Object: 1 MPI processes
> > >   type: seqaij
> > >   rows=46439, cols=46439
> > >   total: nonzeros=502615, allocated nonzeros=504081
> > >   total number of mallocs used during MatSetValues calls =0
> > > not using I-node routines
> > >   linear system matrix = precond matrix:
> > >   Mat Object:   8 MPI processes
> > > type: mpiaij
> > > rows=368656, 

Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

[Gaetan Kenway]

Huh. That's odd then. I was actually bisecting the petsc releases to narrow
it down...I knew 3.3 was ok and 3.7 was not. So I tried 3.5 which was ok,
and then 3.6 which was ok as well, leading to me to conclude the difference
was between 3.6 and 3.7.

On Fri, Aug 11, 2017 at 12:03 PM, Barry Smith  wrote:

>
>Thanks for confirming this. The change was actually in the 3.4 release.
> I have updated the 3.4 changes file to include this change in both the
> maint and master branches.
>
> Barry
>
> > On Aug 11, 2017, at 12:47 PM, Gaetan Kenway  wrote:
> >
> > OK, so that was certainly it.  I vaguely recall reading something about
> this on the mailing list at one point in time, but couldn't find anything.
> > I would definitely put something on the 3.7 change doc since I looked
> there first to see if anything stuck out.
> >
> > Thanks!
> >
> > On Fri, Aug 11, 2017 at 10:28 AM, Barry Smith 
> wrote:
> >
> >Run with the additional option
> >
> >  -sub_pc_factor_shift_type nonzero
> >
> > does this resolve the problem. We changed the default behavior for ILU
> when it detects "zero" pivots.
> >
> > Please let us know if this resolves the problem and we'll update the
> changes file.
> >
> >Barry
> >
> >
> >
> > > On Aug 11, 2017, at 12:14 PM, Gaetan Kenway  wrote:
> > >
> > > Hi All
> > >
> > > I'm in the process of updating a code that uses PETSc for solving
> linear systems for an unstructured CFD code. As of recently, it was using
> an ancient version (3.3). However, when I updated it up to 3.7.6 I ended up
> running into issues with one of the KSP solves. The remainder of the code
> is identical,
> > > I've tracked the issue down to occurring between version 3.6.4 and
> version 3.7.0 . The same issue is present on the most recent version 3.7.6.
> > >
> > > Specifically the issue is that on the second iteration, on the 3.7
> version the KSP kicks out with KSP converged reason of -11 or
> KSP_DIVERGED_PCSETUP_FAILED . After that the two runs differ.  The KSPView
> for each of the two are given below which are identical, up to small
> formatting changes. There is still more I can track down, but I thought I
> would ask if someone knows what might have changed between these two
> versions which could save me a lot of time.
> > >
> > > Thanks,
> > > Gaetan
> > >
> > > 3.6 KSP View:
> > > KSP Object: 8 MPI processes
> > >   type: gmres
> > > GMRES: restart=3, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> > > GMRES: happy breakdown tolerance 1e-30
> > >   maximum iterations=3
> > >   using preconditioner applied to right hand side for initial guess
> > >   tolerances:  relative=1e-08, absolute=1e-20, divergence=1e+15
> > >   left preconditioning
> > >   using nonzero initial guess
> > >   using PRECONDITIONED norm type for convergence test
> > > PC Object: 8 MPI processes
> > >   type: bjacobi
> > > block Jacobi: number of blocks = 8
> > > Local solve is same for all blocks, in the following KSP and PC
> objects:
> > >   KSP Object:  (sub_)   1 MPI processes
> > > type: preonly
> > > maximum iterations=1, initial guess is zero
> > > tolerances:  relative=1e-05, absolute=1e-50, divergence=1
> > > left preconditioning
> > > using NONE norm type for convergence test
> > >   PC Object:  (sub_)   1 MPI processes
> > > type: ilu
> > >   ILU: out-of-place factorization
> > >   0 levels of fill
> > >   tolerance for zero pivot 2.22045e-14
> > >   matrix ordering: natural
> > >   factor fill ratio given 1, needed 1
> > > Factored matrix follows:
> > >   Mat Object:   1 MPI processes
> > > type: seqaij
> > > rows=46439, cols=46439
> > > package used to perform factorization: petsc
> > > total: nonzeros=502615, allocated nonzeros=502615
> > > total number of mallocs used during MatSetValues calls =0
> > >   not using I-node routines
> > > linear system matrix = precond matrix:
> > > Mat Object: 1 MPI processes
> > >   type: seqaij
> > >   rows=46439, cols=46439
> > >   total: nonzeros=502615, allocated nonzeros=504081
> > >   total number of mallocs used during MatSetValues calls =0
> > > not using I-node routines
> > >   linear system matrix = precond matrix:
> > >   Mat Object:   8 MPI processes
> > > type: mpiaij
> > > rows=368656, cols=368656
> > > total: nonzeros=4.63682e+06, allocated nonzeros=4.64417e+06
> > > total number of mallocs used during MatSetValues calls =0
> > >   not using I-node (on process 0) routines
> > > 
> > > reason,its: 2 3 0.001 1e-20
> > >
> > >
> > > Petsc 3.7 KSP View
> > > KSP Object: 8 MPI processes
> > >   type: gmres
> > > GMRES: restart=3, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> > > 

Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

[Barry Smith]

   Thanks for confirming this. The change was actually in the 3.4 release. I 
have updated the 3.4 changes file to include this change in both the maint and 
master branches.

Barry

> On Aug 11, 2017, at 12:47 PM, Gaetan Kenway  wrote:
> 
> OK, so that was certainly it.  I vaguely recall reading something about this 
> on the mailing list at one point in time, but couldn't find anything. 
> I would definitely put something on the 3.7 change doc since I looked there 
> first to see if anything stuck out. 
> 
> Thanks!
> 
> On Fri, Aug 11, 2017 at 10:28 AM, Barry Smith  wrote:
> 
>Run with the additional option
> 
>  -sub_pc_factor_shift_type nonzero
> 
> does this resolve the problem. We changed the default behavior for ILU when 
> it detects "zero" pivots.
> 
> Please let us know if this resolves the problem and we'll update the changes 
> file.
> 
>Barry
> 
> 
> 
> > On Aug 11, 2017, at 12:14 PM, Gaetan Kenway  wrote:
> >
> > Hi All
> >
> > I'm in the process of updating a code that uses PETSc for solving linear 
> > systems for an unstructured CFD code. As of recently, it was using an 
> > ancient version (3.3). However, when I updated it up to 3.7.6 I ended up 
> > running into issues with one of the KSP solves. The remainder of the code 
> > is identical,
> > I've tracked the issue down to occurring between version 3.6.4 and version 
> > 3.7.0 . The same issue is present on the most recent version 3.7.6.
> >
> > Specifically the issue is that on the second iteration, on the 3.7 version 
> > the KSP kicks out with KSP converged reason of -11 or  
> > KSP_DIVERGED_PCSETUP_FAILED . After that the two runs differ.  The KSPView 
> > for each of the two are given below which are identical, up to small 
> > formatting changes. There is still more I can track down, but I thought I 
> > would ask if someone knows what might have changed between these two 
> > versions which could save me a lot of time.
> >
> > Thanks,
> > Gaetan
> >
> > 3.6 KSP View:
> > KSP Object: 8 MPI processes
> >   type: gmres
> > GMRES: restart=3, using Classical (unmodified) Gram-Schmidt 
> > Orthogonalization with no iterative refinement
> > GMRES: happy breakdown tolerance 1e-30
> >   maximum iterations=3
> >   using preconditioner applied to right hand side for initial guess
> >   tolerances:  relative=1e-08, absolute=1e-20, divergence=1e+15
> >   left preconditioning
> >   using nonzero initial guess
> >   using PRECONDITIONED norm type for convergence test
> > PC Object: 8 MPI processes
> >   type: bjacobi
> > block Jacobi: number of blocks = 8
> > Local solve is same for all blocks, in the following KSP and PC objects:
> >   KSP Object:  (sub_)   1 MPI processes
> > type: preonly
> > maximum iterations=1, initial guess is zero
> > tolerances:  relative=1e-05, absolute=1e-50, divergence=1
> > left preconditioning
> > using NONE norm type for convergence test
> >   PC Object:  (sub_)   1 MPI processes
> > type: ilu
> >   ILU: out-of-place factorization
> >   0 levels of fill
> >   tolerance for zero pivot 2.22045e-14
> >   matrix ordering: natural
> >   factor fill ratio given 1, needed 1
> > Factored matrix follows:
> >   Mat Object:   1 MPI processes
> > type: seqaij
> > rows=46439, cols=46439
> > package used to perform factorization: petsc
> > total: nonzeros=502615, allocated nonzeros=502615
> > total number of mallocs used during MatSetValues calls =0
> >   not using I-node routines
> > linear system matrix = precond matrix:
> > Mat Object: 1 MPI processes
> >   type: seqaij
> >   rows=46439, cols=46439
> >   total: nonzeros=502615, allocated nonzeros=504081
> >   total number of mallocs used during MatSetValues calls =0
> > not using I-node routines
> >   linear system matrix = precond matrix:
> >   Mat Object:   8 MPI processes
> > type: mpiaij
> > rows=368656, cols=368656
> > total: nonzeros=4.63682e+06, allocated nonzeros=4.64417e+06
> > total number of mallocs used during MatSetValues calls =0
> >   not using I-node (on process 0) routines
> > 
> > reason,its: 2 3 0.001 1e-20
> >
> >
> > Petsc 3.7 KSP View
> > KSP Object: 8 MPI processes
> >   type: gmres
> > GMRES: restart=3, using Classical (unmodified) Gram-Schmidt 
> > Orthogonalization with no iterative refinement
> > GMRES: happy breakdown tolerance 1e-30
> >   maximum iterations=3
> >   using preconditioner applied to right hand side for initial guess
> >   tolerances:  relative=1e-08, absolute=1e-20, divergence=1e+15
> >   left preconditioning
> >   using nonzero initial guess
> >   using PRECONDITIONED norm type for convergence test
> > PC Object: 8 MPI processes
> >   type: bjacobi
> > block Jacobi: number of blocks = 8
> > Local solve is same for all blocks, 

Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

[Gaetan Kenway]

OK, so that was certainly it.  I vaguely recall reading something about
this on the mailing list at one point in time, but couldn't find anything.
I would definitely put something on the 3.7 change doc since I looked there
first to see if anything stuck out.

Thanks!

On Fri, Aug 11, 2017 at 10:28 AM, Barry Smith  wrote:

>
>Run with the additional option
>
>  -sub_pc_factor_shift_type nonzero
>
> does this resolve the problem. We changed the default behavior for ILU
> when it detects "zero" pivots.
>
> Please let us know if this resolves the problem and we'll update the
> changes file.
>
>Barry
>
>
>
> > On Aug 11, 2017, at 12:14 PM, Gaetan Kenway  wrote:
> >
> > Hi All
> >
> > I'm in the process of updating a code that uses PETSc for solving linear
> systems for an unstructured CFD code. As of recently, it was using an
> ancient version (3.3). However, when I updated it up to 3.7.6 I ended up
> running into issues with one of the KSP solves. The remainder of the code
> is identical,
> > I've tracked the issue down to occurring between version 3.6.4 and
> version 3.7.0 . The same issue is present on the most recent version 3.7.6.
> >
> > Specifically the issue is that on the second iteration, on the 3.7
> version the KSP kicks out with KSP converged reason of -11 or
> KSP_DIVERGED_PCSETUP_FAILED . After that the two runs differ.  The KSPView
> for each of the two are given below which are identical, up to small
> formatting changes. There is still more I can track down, but I thought I
> would ask if someone knows what might have changed between these two
> versions which could save me a lot of time.
> >
> > Thanks,
> > Gaetan
> >
> > 3.6 KSP View:
> > KSP Object: 8 MPI processes
> >   type: gmres
> > GMRES: restart=3, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> > GMRES: happy breakdown tolerance 1e-30
> >   maximum iterations=3
> >   using preconditioner applied to right hand side for initial guess
> >   tolerances:  relative=1e-08, absolute=1e-20, divergence=1e+15
> >   left preconditioning
> >   using nonzero initial guess
> >   using PRECONDITIONED norm type for convergence test
> > PC Object: 8 MPI processes
> >   type: bjacobi
> > block Jacobi: number of blocks = 8
> > Local solve is same for all blocks, in the following KSP and PC
> objects:
> >   KSP Object:  (sub_)   1 MPI processes
> > type: preonly
> > maximum iterations=1, initial guess is zero
> > tolerances:  relative=1e-05, absolute=1e-50, divergence=1
> > left preconditioning
> > using NONE norm type for convergence test
> >   PC Object:  (sub_)   1 MPI processes
> > type: ilu
> >   ILU: out-of-place factorization
> >   0 levels of fill
> >   tolerance for zero pivot 2.22045e-14
> >   matrix ordering: natural
> >   factor fill ratio given 1, needed 1
> > Factored matrix follows:
> >   Mat Object:   1 MPI processes
> > type: seqaij
> > rows=46439, cols=46439
> > package used to perform factorization: petsc
> > total: nonzeros=502615, allocated nonzeros=502615
> > total number of mallocs used during MatSetValues calls =0
> >   not using I-node routines
> > linear system matrix = precond matrix:
> > Mat Object: 1 MPI processes
> >   type: seqaij
> >   rows=46439, cols=46439
> >   total: nonzeros=502615, allocated nonzeros=504081
> >   total number of mallocs used during MatSetValues calls =0
> > not using I-node routines
> >   linear system matrix = precond matrix:
> >   Mat Object:   8 MPI processes
> > type: mpiaij
> > rows=368656, cols=368656
> > total: nonzeros=4.63682e+06, allocated nonzeros=4.64417e+06
> > total number of mallocs used during MatSetValues calls =0
> >   not using I-node (on process 0) routines
> > 
> > reason,its: 2 3 0.001 1e-20
> >
> >
> > Petsc 3.7 KSP View
> > KSP Object: 8 MPI processes
> >   type: gmres
> > GMRES: restart=3, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> > GMRES: happy breakdown tolerance 1e-30
> >   maximum iterations=3
> >   using preconditioner applied to right hand side for initial guess
> >   tolerances:  relative=1e-08, absolute=1e-20, divergence=1e+15
> >   left preconditioning
> >   using nonzero initial guess
> >   using PRECONDITIONED norm type for convergence test
> > PC Object: 8 MPI processes
> >   type: bjacobi
> > block Jacobi: number of blocks = 8
> > Local solve is same for all blocks, in the following KSP and PC
> objects:
> >   KSP Object:  (sub_)   1 MPI processes
> > type: preonly
> > maximum iterations=1, initial guess is zero
> > tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
> > left preconditioning
> > using NONE norm type for convergence test
> >   PC Object:  

Re: [petsc-users] Petsc ILU PC Change between 3.6.4 and 3.7.x?

[Barry Smith]

   Run with the additional option

 -sub_pc_factor_shift_type nonzero 

does this resolve the problem. We changed the default behavior for ILU when it 
detects "zero" pivots.

Please let us know if this resolves the problem and we'll update the changes 
file.

   Barry



> On Aug 11, 2017, at 12:14 PM, Gaetan Kenway  wrote:
> 
> Hi All
> 
> I'm in the process of updating a code that uses PETSc for solving linear 
> systems for an unstructured CFD code. As of recently, it was using an ancient 
> version (3.3). However, when I updated it up to 3.7.6 I ended up running into 
> issues with one of the KSP solves. The remainder of the code is identical,
> I've tracked the issue down to occurring between version 3.6.4 and version 
> 3.7.0 . The same issue is present on the most recent version 3.7.6.
> 
> Specifically the issue is that on the second iteration, on the 3.7 version 
> the KSP kicks out with KSP converged reason of -11 or  
> KSP_DIVERGED_PCSETUP_FAILED . After that the two runs differ.  The KSPView 
> for each of the two are given below which are identical, up to small 
> formatting changes. There is still more I can track down, but I thought I 
> would ask if someone knows what might have changed between these two versions 
> which could save me a lot of time. 
> 
> Thanks,
> Gaetan
> 
> 3.6 KSP View:
> KSP Object: 8 MPI processes
>   type: gmres
> GMRES: restart=3, using Classical (unmodified) Gram-Schmidt 
> Orthogonalization with no iterative refinement
> GMRES: happy breakdown tolerance 1e-30
>   maximum iterations=3
>   using preconditioner applied to right hand side for initial guess
>   tolerances:  relative=1e-08, absolute=1e-20, divergence=1e+15
>   left preconditioning
>   using nonzero initial guess
>   using PRECONDITIONED norm type for convergence test
> PC Object: 8 MPI processes
>   type: bjacobi
> block Jacobi: number of blocks = 8
> Local solve is same for all blocks, in the following KSP and PC objects:
>   KSP Object:  (sub_)   1 MPI processes
> type: preonly
> maximum iterations=1, initial guess is zero
> tolerances:  relative=1e-05, absolute=1e-50, divergence=1
> left preconditioning
> using NONE norm type for convergence test
>   PC Object:  (sub_)   1 MPI processes
> type: ilu
>   ILU: out-of-place factorization
>   0 levels of fill
>   tolerance for zero pivot 2.22045e-14
>   matrix ordering: natural
>   factor fill ratio given 1, needed 1
> Factored matrix follows:
>   Mat Object:   1 MPI processes
> type: seqaij
> rows=46439, cols=46439
> package used to perform factorization: petsc
> total: nonzeros=502615, allocated nonzeros=502615
> total number of mallocs used during MatSetValues calls =0
>   not using I-node routines
> linear system matrix = precond matrix:
> Mat Object: 1 MPI processes
>   type: seqaij
>   rows=46439, cols=46439
>   total: nonzeros=502615, allocated nonzeros=504081
>   total number of mallocs used during MatSetValues calls =0
> not using I-node routines
>   linear system matrix = precond matrix:
>   Mat Object:   8 MPI processes
> type: mpiaij
> rows=368656, cols=368656
> total: nonzeros=4.63682e+06, allocated nonzeros=4.64417e+06
> total number of mallocs used during MatSetValues calls =0
>   not using I-node (on process 0) routines
> 
> reason,its: 2 3 0.001 1e-20
> 
> 
> Petsc 3.7 KSP View
> KSP Object: 8 MPI processes
>   type: gmres
> GMRES: restart=3, using Classical (unmodified) Gram-Schmidt 
> Orthogonalization with no iterative refinement
> GMRES: happy breakdown tolerance 1e-30
>   maximum iterations=3
>   using preconditioner applied to right hand side for initial guess
>   tolerances:  relative=1e-08, absolute=1e-20, divergence=1e+15
>   left preconditioning
>   using nonzero initial guess
>   using PRECONDITIONED norm type for convergence test
> PC Object: 8 MPI processes
>   type: bjacobi
> block Jacobi: number of blocks = 8
> Local solve is same for all blocks, in the following KSP and PC objects:
>   KSP Object:  (sub_)   1 MPI processes
> type: preonly
> maximum iterations=1, initial guess is zero
> tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
> left preconditioning
> using NONE norm type for convergence test
>   PC Object:  (sub_)   1 MPI processes
> type: ilu
>   ILU: out-of-place factorization
>   0 levels of fill
>   tolerance for zero pivot 2.22045e-14
>   matrix ordering: natural
>   factor fill ratio given 1., needed 1.
> Factored matrix follows:
>   Mat Object:   1 MPI processes
> type: seqaij
> rows=46439, cols=46439
> package used to perform factorization: petsc
> total: nonzeros=502615, allocated nonzeros=502615
> total