You are referring to
8 KSP Residual norm 8.464122393957e-07
Residual norms for fieldsplit_p_lsc_ solve.
0 KSP Residual norm 1.000000000000e+00
1 KSP Residual norm 1.627170127450e-05
2 KSP Residual norm 1.448537328052e-05
3 KSP Residual norm 1.309526985856e-05
4 KSP Residual norm 1.308667010157e-05
5 KSP Residual norm 1.254021842629e-05
6 KSP Residual norm 9.490928295823e-06
where because the default rtol is 1.e-5 it takes 6 iterations, mostly to get
the last little bit?
The thing I notice is that "Residual norms for fieldsplit_p_ solve.:" are
converging pretty slowly, around 1 digit per iteration. That tells me that the
inner inner solve is "oversolving", that is giving more accuracy than is likely
needed. Of course, when it takes 1 iteration there is no way for you to back it
off to take 1/2 an iteration :-).
Some things to consider:
1) do not depend on the rtol at all but just set that inner inner solve to do a
single iteration; clearly in this particular case this would work fine. (you
could even turn off norm calculations in this inner solve, just use Richardson
in this inner inner solve, to save a tiny bit of time), of course, only
experimentation would tell if that always works for your family of problems.
2) or just use an rtol of say 1.e-3 in that inner inner solve. Likely that is
ok given the relative slow convergence of the fieldsplit_p_ solve.
Now if the "Residual norms for fieldsplit_p_ solve.:" was dropping say 4 or
more digits each iteration, then I would not generally back off the inner inner
solve since likely in that case it would not be oversolving.
I don't think the scaling factor as part of the FGMRES algorithm is relevant
here.
Barry
> On Jul 6, 2023, at 3:09 PM, Alexander Lindsay <[email protected]>
> wrote:
>
> Hi all, I've found that having fgmres as an outer solve and the initial
> residual scaling that comes with it can cause difficulties for inner solves
> as I'm approaching convergence, presumably because I'm running out of
> precision. This is the kind of thing I would normally set an absolute
> tolerance for, but it feels a little silly to set `-ksp_atol 1e-4` when at
> the beginning of solves there is no problem reaching that tolerance. Here is
> an example solve history. You can see that the fieldsplit_p_lsc_solve
> converges in a single iteration until the very end where it's clearly
> struggling to hit the rtol of 1e-5. Is the scaling of the fgmres residual
> important to the algorithm? git blame showed that the comment regarding
> "scale VEC_VV (the initial residual)" was last hit in 1999 (and it existed
> before that time), see line 125 of fgmres.c
>
> 0 KSP Residual norm 2.586864431320e-06
> Residual norms for fieldsplit_p_ solve.
> 0 KSP Residual norm 4.692593309369e-02
> Residual norms for fieldsplit_p_lsc_ solve.
> 0 KSP Residual norm 1.000000000000e+00
> 1 KSP Residual norm 4.048060575312e-06
> Residual norms for fieldsplit_p_lsc_ solve.
> 0 KSP Residual norm 4.951804839158e+01
> 1 KSP Residual norm 9.672178594775e-05
> 1 KSP Residual norm 2.076409133334e-02
> Residual norms for fieldsplit_p_lsc_ solve.
> 0 KSP Residual norm 1.000000000000e+00
> 1 KSP Residual norm 1.346618231753e-06
> Residual norms for fieldsplit_p_lsc_ solve.
> 0 KSP Residual norm 8.119436471232e+01
> 1 KSP Residual norm 1.233237431511e-04
> 2 KSP Residual norm 8.753379232559e-03
> Residual norms for fieldsplit_p_lsc_ solve.
> 0 KSP Residual norm 1.000000000000e+00
> 1 KSP Residual norm 1.023335686682e-06
> Residual norms for fieldsplit_p_lsc_ solve.
> 0 KSP Residual norm 1.095058035930e+02
> 1 KSP Residual norm 8.452076098586e-05
> 3 KSP Residual norm 1.589938296474e-03
> Residual norms for fieldsplit_p_lsc_ solve.
> 0 KSP Residual norm 1.000000000000e+00
> 1 KSP Residual norm 7.680981295089e-06
> Residual norms for fieldsplit_p_lsc_ solve.
> 0 KSP Residual norm 7.768759831332e+01
> 1 KSP Residual norm 5.120395415891e-04
> 4 KSP Residual norm 4.191115394200e-04
> Residual norms for fieldsplit_p_lsc_ solve.
> 0 KSP Residual norm 1.000000000000e+00
> 1 KSP Residual norm 8.815992679344e-07
> Residual norms for fieldsplit_p_lsc_ solve.
> 0 KSP Residual norm 5.245482238296e+01
> 1 KSP Residual norm 8.672309412797e-05
> 5 KSP Residual norm 1.053200310197e-04
> Residual norms for fieldsplit_p_lsc_ solve.
> 0 KSP Residual norm 1.000000000000e+00
> 1 KSP Residual norm 1.450184043299e-06
> Residual norms for fieldsplit_p_lsc_ solve.
> 0 KSP Residual norm 9.058043325461e+01
> 1 KSP Residual norm 1.643116692318e-04
> 6 KSP Residual norm 2.315708746597e-05
> Residual norms for fieldsplit_p_lsc_ solve.
> 0 KSP Residual norm 1.000000000000e+00
> 1 KSP Residual norm 1.406236994541e-06
> Residual norms for fieldsplit_p_lsc_ solve.
> 0 KSP Residual norm 5.158957906917e+01
> 1 KSP Residual norm 8.910977751230e-05
> 7 KSP Residual norm 2.378083073391e-06
> Residual norms for fieldsplit_p_lsc_ solve.
> 0 KSP Residual norm 1.000000000000e+00
> 1 KSP Residual norm 7.981173164359e-06
> Residual norms for fieldsplit_p_lsc_ solve.
> 0 KSP Residual norm 7.271546763656e+01
> 1 KSP Residual norm 2.211194524633e-04
> 8 KSP Residual norm 8.464122393957e-07
> Residual norms for fieldsplit_p_lsc_ solve.
> 0 KSP Residual norm 1.000000000000e+00
> 1 KSP Residual norm 1.627170127450e-05
> 2 KSP Residual norm 1.448537328052e-05
> 3 KSP Residual norm 1.309526985856e-05
> 4 KSP Residual norm 1.308667010157e-05
> 5 KSP Residual norm 1.254021842629e-05
> 6 KSP Residual norm 9.490928295823e-06
> Residual norms for fieldsplit_p_lsc_ solve.
> 0 KSP Residual norm 7.084510697783e+01
> 1 KSP Residual norm 1.444782053473e-04
> 9 KSP Residual norm 3.334507715334e-07
> Linear fieldsplit_p_ solve converged due to CONVERGED_RTOL iterations 9
