I first set the option -ksp_converged_reason and obtained the message
$ ./ex1
Residual norms for fieldsplit_PA_ solve.
0 KSP unpreconditioned resid norm 2.288897733893e+03 true resid norm
2.288897733893e+03 ||r(i)||/||b|| 1.000000000000e+00
Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0
PCSETUP_FAILED due to SUBPC_ERROR
so it is failing in building the preconditioner since it didn't even work for
the first fieldsplit I guessed that it failed in the gamg on the first
fieldsplit so ran with
-fieldsplit_PA_ksp_error_if_not_converged
and got
$ ./ex1
Residual norms for fieldsplit_PA_ solve.
0 KSP unpreconditioned resid norm 2.288897733893e+03 true resid norm
2.288897733893e+03 ||r(i)||/||b|| 1.000000000000e+00
Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0
PCSETUP_FAILED due to SUBPC_ERROR
~/Src/petsc/test-dir (master=) arch-master-basic
$ ./ex1 -fieldsplit_PA_ksp_error_if_not_converged
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR:
[0]PETSC ERROR: KSPSolve has not converged
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.7.3-1182-g00a02d5 GIT Date:
2016-08-16 15:09:17 -0500
[0]PETSC ERROR: ./ex1 on a arch-master-basic named Barrys-MacBook-Pro.local by
barrysmith Wed Aug 17 00:35:16 2016
[0]PETSC ERROR: Configure options --with-mpi-dir=/Users/barrysmith/libraries
[0]PETSC ERROR: #1 KSPSolve() line 850 in
/Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #2 PCGAMGOptProlongator_AGG() line 1221 in
/Users/barrysmith/Src/petsc/src/ksp/pc/impls/gamg/agg.c
looking at the code and seeing the options prefix for this KSP solve I ran with
-fieldsplit_PA_ksp_error_if_not_converged
-fieldsplit_PA_gamg_est_ksp_monitor_true_residual
and got
$ ./ex1 -fieldsplit_PA_ksp_error_if_not_converged
-fieldsplit_PA_gamg_est_ksp_monitor_true_residual
Residual norms for fieldsplit_PA_gamg_est_ solve.
0 KSP none resid norm 7.030417576826e+07 true resid norm 1.006594247197e+01
||r(i)||/||b|| 1.000000000000e+00
1 KSP none resid norm 6.979279406029e+07 true resid norm 1.150138107009e+01
||r(i)||/||b|| 1.142603497100e+00
2 KSP none resid norm 6.979246564783e+07 true resid norm 6.970771666727e+03
||r(i)||/||b|| 6.925105807166e+02
3 KSP none resid norm 6.978033367036e+07 true resid norm 9.958555706490e+02
||r(i)||/||b|| 9.893316730368e+01
4 KSP none resid norm 6.977995917588e+07 true resid norm 1.095475380870e+04
||r(i)||/||b|| 1.088298869103e+03
5 KSP none resid norm 6.954940040289e+07 true resid norm 2.182804459638e+04
||r(i)||/||b|| 2.168504802919e+03
6 KSP none resid norm 6.905975832912e+07 true resid norm 4.557801389945e+04
||r(i)||/||b|| 4.527943014412e+03
7 KSP none resid norm 6.905788649989e+07 true resid norm 4.476060162996e+04
||r(i)||/||b|| 4.446737278162e+03
8 KSP none resid norm 5.464732207984e+07 true resid norm 7.801211607942e+04
||r(i)||/||b|| 7.750105496491e+03
9 KSP none resid norm 5.393328767072e+07 true resid norm 8.529925739695e+04
||r(i)||/||b|| 8.474045787013e+03
10 KSP none resid norm 5.294387823310e+07 true resid norm 8.380999411358e+04
||r(i)||/||b|| 8.326095082197e+03
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR:
[0]PETSC ERROR: KSPSolve has not converged
the smoother is simply not working AT ALL on your first sub matrix so I ran
with -fieldsplit_PA_gamg_est_ksp_pmat_view and loaded the resulting matrix from
the file binaryoutput into matlab and checked its eigenvalues
>> e = eig(full(a))
e =
5.5260 +85.4938i
5.5260 -85.4938i
-11.6673 + 0.0000i
-6.5409 + 0.0000i
6.5240 + 0.0000i
-2.5377 + 2.0951i
-2.5377 - 2.0951i
3.0712 + 0.0000i
-0.5365 + 0.9521i
-0.5365 - 0.9521i
1.0710 + 0.0000i
-0.9334 + 0.0000i
0.7608 + 0.0000i
0.4337 + 0.0000i
-0.4558 + 0.0000i
-0.4011 + 0.0000i
0.1212 + 0.0000i
0.0327 + 0.0338i
0.0327 - 0.0338i
-0.0426 + 0.0000i
-0.0360 + 0.0334i
-0.0360 - 0.0334i
0.0308 + 0.0000i
-0.0003 + 0.0327i
-0.0003 - 0.0327i
-0.0323 + 0.0000i
0.0120 + 0.0185i
0.0120 - 0.0185i
-0.0256 + 0.0000i
-0.0225 + 0.0000i
0.0152 + 0.0000i
-0.0083 + 0.0125i
-0.0083 - 0.0125i
-0.0177 + 0.0000i
-0.0175 + 0.0000i
-0.0177 + 0.0000i
-0.0158 + 0.0000i
-0.0176 + 0.0000i
-0.0136 + 0.0038i
-0.0136 - 0.0038i
0.0125 + 0.0000i
-0.0080 + 0.0069i
-0.0080 - 0.0069i
0.0066 + 0.0075i
0.0066 - 0.0075i
0.0097 + 0.0000i
0.0039 + 0.0085i
0.0039 - 0.0085i
0.0070 + 0.0000i
-0.0095 + 0.0011i
-0.0095 - 0.0011i
-0.0064 + 0.0000i
0.0024 + 0.0036i
0.0024 - 0.0036i
0.0042 + 0.0000i
0.0042 - 0.0000i
0.0040 + 0.0000i
-0.0035 + 0.0021i
-0.0035 - 0.0021i
-0.0004 + 0.0038i
-0.0004 - 0.0038i
-0.0040 + 0.0000i
-0.0037 + 0.0000i
-0.0036 + 0.0000i
-0.0037 + 0.0000i
-0.0033 + 0.0000i
-0.0033 + 0.0000i
-0.0033 + 0.0000i
-0.0029 + 0.0000i
0.0015 + 0.0014i
0.0015 - 0.0014i
0.0004 + 0.0022i
0.0004 - 0.0022i
0.0003 + 0.0024i
0.0003 - 0.0024i
-0.0012 + 0.0018i
-0.0012 - 0.0018i
0.0003 + 0.0023i
0.0003 - 0.0023i
-0.0022 + 0.0000i
-0.0024 + 0.0000i
-0.0024 - 0.0000i
-0.0021 + 0.0000i
0.0010 + 0.0010i
0.0010 - 0.0010i
0.0009 + 0.0009i
0.0009 - 0.0009i
-0.0002 + 0.0014i
-0.0002 - 0.0014i
-0.0013 + 0.0011i
-0.0013 - 0.0011i
-0.0011 + 0.0012i
-0.0011 - 0.0012i
-0.0014 + 0.0008i
-0.0014 - 0.0008i
-0.0012 + 0.0008i
-0.0012 - 0.0008i
-0.0014 + 0.0004i
-0.0014 - 0.0004i
-0.0005 + 0.0011i
-0.0005 - 0.0011i
0.0007 + 0.0009i
0.0007 - 0.0009i
-0.0010 + 0.0001i
-0.0010 - 0.0001i
0.0008 + 0.0006i
0.0008 - 0.0006i
0.0011 + 0.0001i
0.0011 - 0.0001i
0.0010 + 0.0001i
0.0010 - 0.0001i
0.0010 + 0.0000i
0.0010 - 0.0000i
0.0003 + 0.0008i
0.0003 - 0.0008i
-0.0001 + 0.0009i
-0.0001 - 0.0009i
-0.0008 + 0.0000i
-0.0008 + 0.0005i
-0.0008 - 0.0005i
-0.0004 + 0.0007i
-0.0004 - 0.0007i
-0.0008 + 0.0002i
-0.0008 - 0.0002i
-0.0000 + 0.0008i
-0.0000 - 0.0008i
0.0008 + 0.0000i
0.0008 - 0.0000i
-0.0008 + 0.0000i
-0.0007 + 0.0000i
-0.0004 + 0.0005i
-0.0004 - 0.0005i
0.0005 + 0.0003i
0.0005 - 0.0003i
0.0006 + 0.0001i
0.0006 - 0.0001i
0.0006 + 0.0000i
-0.0005 + 0.0003i
-0.0005 - 0.0003i
0.0003 + 0.0005i
0.0003 - 0.0005i
-0.0001 + 0.0005i
-0.0001 - 0.0005i
0.0000 + 0.0006i
0.0000 - 0.0006i
-0.0004 + 0.0000i
-0.0004 + 0.0000i
0.0003 + 0.0003i
0.0003 - 0.0003i
0.0003 + 0.0002i
0.0003 - 0.0002i
-0.0004 + 0.0000i
-0.0000 + 0.0004i
-0.0000 - 0.0004i
-0.0002 + 0.0003i
-0.0002 - 0.0003i
-0.0001 + 0.0003i
-0.0001 - 0.0003i
0.0003 + 0.0001i
0.0003 - 0.0001i
-0.0003 + 0.0000i
-0.0003 - 0.0000i
0.0001 + 0.0002i
0.0001 - 0.0002i
0.0002 + 0.0000i
0.0002 + 0.0001i
0.0002 - 0.0001i
0.0001 + 0.0002i
0.0001 - 0.0002i
0.0002 + 0.0000i
-0.0002 + 0.0000i
-0.0002 - 0.0000i
0.0001 + 0.0000i
0.0000 + 0.0001i
0.0000 - 0.0001i
0.0001 + 0.0001i
0.0001 - 0.0001i
-0.0001 + 0.0000i
-0.0001 + 0.0000i
-0.0001 - 0.0000i
0.0001 + 0.0001i
0.0001 - 0.0001i
0.0001 + 0.0000i
0.0001 - 0.0000i
-0.0001 + 0.0001i
-0.0001 - 0.0001i
0.0001 + 0.0001i
0.0001 - 0.0001i
0.0001 + 0.0000i
0.0000 + 0.0001i
0.0000 - 0.0001i
0.0000 + 0.0001i
0.0000 - 0.0001i
-0.0000 + 0.0001i
-0.0000 - 0.0001i
-0.0001 + 0.0000i
-0.0001 - 0.0000i
-0.0001 + 0.0000i
-0.0001 - 0.0000i
-0.0001 + 0.0000i
-0.0001 - 0.0000i
0.0001 + 0.0000i
0.0001 - 0.0000i
-0.0001 + 0.0000i
-0.0001 - 0.0000i
-0.0001 + 0.0000i
0.0000 + 0.0001i
0.0000 - 0.0001i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
0.0000 + 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 + 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
-0.0000 + 0.0000i
-0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
-0.0000 + 0.0000i
0.0000 + 0.0000i
-0.0000 + 0.0000i
0.0000 + 0.0000i
-0.0000 + 0.0000i
-0.0000 + 0.0000i
0.0000 + 0.0000i
-0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
The first matrix you are handing off to GAMG is totally inappropriate for GAMG
it has a huge range of eigenvalue scales both positive and negative and many
zero eigenvectors. The smoother GAMG uses does not work on the matrix nor will
GAMG work on the matrix.
Hence the entire nested fieldsplit preconditioner cannot work on the entire
matrix since the inner GAMG preconditioner cannot work on the inner matrix, in
fact probably next to no preconditioner will work on the inner matrix.
You need to go back and make sure that either 1) the matrices being generated
are what they should be and 2) your assumption that GAMG would work on the
first submatrix makes sense
Barry
> On Aug 16, 2016, at 5:21 PM, Safin, Artur <[email protected]> wrote:
>
> Barry, Mark,
>
> Apologies for taking a while to respond.
>
>> Are you saying it works for a while but fails when the problem is large, or
>> that it never works with fieldsplit_1? And how many processors are you using?
>
> All of these are serial runs. The preconditioner always fails for large
> problems; but I also found an example where it fails for a fairly small
> matrix. fieldsplit_1 is the one that fails, although fieldpslit_0 always
> works.
>
>> This should be fixed in the master branch
>
> I tried the developmental version (from today), but the same bug persists.
>
>
>
> I have some code that reproduces the example. The index sets that I use are
> non-overlapping; I also checked with matlab that each fieldsplit submatrix is
> non-singular.
> The issue is with fieldsplit_TA: if I do not use -fieldsplit_TA-pc_type_gamg,
> then the solver converges. Otherwise, it crashes with the same message.
>
> Artur
>
>
> <A> <A.info><ex.c> <run.sh>
