Roy Stogner <royst...@ices.utexas.edu> writes: >> Interesting. Can you include the full unpreconditioned output? And >> please compare with and without -ksp_gmres_modifiedgramschmidt. > > From "./main-opt -ksp_monitor_true_residual -ksp_norm_type unpreconditioned": > 0 KSP unpreconditioned resid norm 1.137340546775e-01 true resid norm > 1.137340546775e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP unpreconditioned resid norm 1.851221771429e-02 true resid norm > 1.850940946785e-02 ||r(i)||/||b|| 1.627428963148e-01 > 2 KSP unpreconditioned resid norm 8.058732139350e-04 true resid norm > 8.487837216519e-04 ||r(i)||/||b|| 7.462881052281e-03 > 3 KSP unpreconditioned resid norm 1.410067346760e-04 true resid norm > 3.750759001371e-04 ||r(i)||/||b|| 3.297832836441e-03 > 4 KSP unpreconditioned resid norm 3.036297998627e-06 true resid norm > 3.296862054294e-04 ||r(i)||/||b|| 2.898746609924e-03 > 5 KSP unpreconditioned resid norm 1.628004374290e-07 true resid norm > 4.048903454605e-04 ||r(i)||/||b|| 3.559974596954e-03 > 6 KSP unpreconditioned resid norm 6.839537794328e-09 true resid norm > 2.859920502301e-04 ||r(i)||/||b|| 2.514568315013e-03 > 7 KSP unpreconditioned resid norm 3.580827398197e-10 true resid norm > 3.424888565589e-04 ||r(i)||/||b|| 3.011313168514e-03 > 8 KSP unpreconditioned resid norm 1.371032175862e-11 true resid norm > 3.818671006476e-04 ||r(i)||/||b|| 3.357544068312e-03 > 9 KSP unpreconditioned resid norm 1.130911552385e-12 true resid norm > 2.475276237056e-04 ||r(i)||/||b|| 2.176372102511e-03 > 10 KSP unpreconditioned resid norm 4.683505266751e-14 true resid norm > 3.820485023471e-04 ||r(i)||/||b|| 3.359139032108e-03 > And after adding "-ksp_gmres_modifiedgramschmidt" too: > 0 KSP unpreconditioned resid norm 1.137340546775e-01 true resid norm > 1.137340546775e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP unpreconditioned resid norm 1.851221771429e-02 true resid norm > 1.850940946785e-02 ||r(i)||/||b|| 1.627428963148e-01 > 2 KSP unpreconditioned resid norm 8.058732139350e-04 true resid norm > 8.290754235563e-04 ||r(i)||/||b|| 7.289596998076e-03 > 3 KSP unpreconditioned resid norm 1.409277416992e-04 true resid norm > 3.366737464418e-04 ||r(i)||/||b|| 2.960184154134e-03 > 4 KSP unpreconditioned resid norm 3.016318237424e-06 true resid norm > 3.904608063041e-04 ||r(i)||/||b|| 3.433103720879e-03 > 5 KSP unpreconditioned resid norm 1.684322222043e-07 true resid norm > 1.987083874964e-04 ||r(i)||/||b|| 1.747131833643e-03 > 6 KSP unpreconditioned resid norm 7.043704865544e-09 true resid norm > 3.021936047142e-04 ||r(i)||/||b|| 2.657019531846e-03 > 7 KSP unpreconditioned resid norm 3.538690849000e-10 true resid norm > 3.937474047247e-04 ||r(i)||/||b|| 3.462000944582e-03 > 8 KSP unpreconditioned resid norm 2.298664686412e-11 true resid norm > 2.516665836236e-04 ||r(i)||/||b|| 2.212763664649e-03 > 9 KSP unpreconditioned resid norm 8.918329988448e-13 true resid norm > 3.883206450876e-04 ||r(i)||/||b|| 3.414286479003e-03 > 10 KSP unpreconditioned resid norm 3.074816936477e-14 true resid norm > 2.987382118533e-04 ||r(i)||/||b|| 2.626638192935e-03
This seems to be a peculiar linear algebra phenomenon, seemingly due to a bad shift strategy for the indefinite matrix (see below). FGMRES, GCR, and and BiCG manage anyway while GMRES and BiCGStab do not. >> How big is this matrix? Can you write it with "-ksp_view_mat binary" >> and send it to me (or post somewhere)? > > Not even 100x100; it's a very-well-simplified version of the real > problem. > > http://users.ices.utexas.edu/~roystgnr/binaryoutput src/ksp/ksp/examples/tutorials$ ./ex10 -f binaryoutput -rhs [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot [0]PETSC ERROR: Zero pivot row 4 value 0 tolerance 2.22045e-14 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.5.2-939-geeb4e01 GIT Date: 2014-11-05 15:13:20 -0600 [0]PETSC ERROR: ./ex10 on a mpich-g named batura by jed Fri Jan 9 11:43:38 2015 [0]PETSC ERROR: Configure options --download-chaco --download-ctetgen --download-generator --download-hypre --download-ml --download-sundials --download-superlu --download-superlu_dist --download-triangle --with-c2html --with-exodusii --with-hdf5 --with-lgrind --with-metis --with-mpi-dir=/home/jed/usr/ccache/mpich/ --with-netcdf --with-parmetis --with-sowing --with-suitesparse --with-x -PETSC_ARCH=mpich-g COPTFLAGS="-Og -g" [0]PETSC ERROR: #1 MatPivotCheck_none() line 634 in /home/jed/petsc/include/petsc-private/matimpl.h [0]PETSC ERROR: #2 MatPivotCheck() line 653 in /home/jed/petsc/include/petsc-private/matimpl.h [0]PETSC ERROR: #3 MatLUFactorNumeric_SeqAIJ_Inode() line 1424 in /home/jed/petsc/src/mat/impls/aij/seq/inode.c [0]PETSC ERROR: #4 MatLUFactorNumeric() line 2930 in /home/jed/petsc/src/mat/interface/matrix.c [0]PETSC ERROR: #5 PCSetUp_ILU() line 232 in /home/jed/petsc/src/ksp/pc/impls/factor/ilu/ilu.c [0]PETSC ERROR: #6 PCSetUp() line 902 in /home/jed/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #7 KSPSetUp() line 306 in /home/jed/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #8 main() line 312 in /home/jed/petsc/src/ksp/ksp/examples/tutorials/ex10.c [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -f binaryoutput [0]PETSC ERROR: -malloc_test [0]PETSC ERROR: -rhs [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-ma...@mcs.anl.gov---------- $ ./ex10 -f binaryoutput -rhs -pc_type lu [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot [0]PETSC ERROR: Zero pivot row 12 value 0 tolerance 2.22045e-14 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.5.2-939-geeb4e01 GIT Date: 2014-11-05 15:13:20 -0600 [0]PETSC ERROR: ./ex10 on a mpich-g named batura by jed Fri Jan 9 11:44:29 2015 [0]PETSC ERROR: Configure options --download-chaco --download-ctetgen --download-generator --download-hypre --download-ml --download-sundials --download-superlu --download-superlu_dist --download-triangle --with-c2html --with-exodusii --with-hdf5 --with-lgrind --with-metis --with-mpi-dir=/home/jed/usr/ccache/mpich/ --with-netcdf --with-parmetis --with-sowing --with-suitesparse --with-x -PETSC_ARCH=mpich-g COPTFLAGS="-Og -g" [0]PETSC ERROR: #1 MatPivotCheck_none() line 634 in /home/jed/petsc/include/petsc-private/matimpl.h [0]PETSC ERROR: #2 MatPivotCheck() line 653 in /home/jed/petsc/include/petsc-private/matimpl.h [0]PETSC ERROR: #3 MatLUFactorNumeric_SeqAIJ_Inode() line 1424 in /home/jed/petsc/src/mat/impls/aij/seq/inode.c [0]PETSC ERROR: #4 MatLUFactorNumeric() line 2930 in /home/jed/petsc/src/mat/interface/matrix.c [0]PETSC ERROR: #5 PCSetUp_LU() line 152 in /home/jed/petsc/src/ksp/pc/impls/factor/lu/lu.c [0]PETSC ERROR: #6 PCSetUp() line 902 in /home/jed/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #7 KSPSetUp() line 306 in /home/jed/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #8 main() line 312 in /home/jed/petsc/src/ksp/ksp/examples/tutorials/ex10.c [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -f binaryoutput [0]PETSC ERROR: -malloc_test [0]PETSC ERROR: -pc_type lu [0]PETSC ERROR: -rhs [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-ma...@mcs.anl.gov---------- A "positive definite" shift works, but converges very slowly. $ ./ex10 -f binaryoutput -rhs -ksp_converged_reason -ksp_monitor_true_residual -ksp_rtol 1e-10 -pc_factor_shift_type positive_definite 0 KSP preconditioned resid norm 5.126361879870e+01 true resid norm 9.899494936612e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.742253255361e+01 true resid norm 9.945020941036e+00 ||r(i)||/||b|| 1.004598820921e+00 2 KSP preconditioned resid norm 1.656435081546e+01 true resid norm 1.017851736308e+01 ||r(i)||/||b|| 1.028185521408e+00 3 KSP preconditioned resid norm 1.381919181644e+01 true resid norm 1.317793534482e+01 ||r(i)||/||b|| 1.331172492052e+00 [...] 92 KSP preconditioned resid norm 1.209846782924e-08 true resid norm 7.516470758142e-09 ||r(i)||/||b|| 7.592782062390e-10 93 KSP preconditioned resid norm 1.010690565657e-08 true resid norm 5.111314393418e-09 ||r(i)||/||b|| 5.163207240517e-10 94 KSP preconditioned resid norm 6.900556652393e-09 true resid norm 3.274216395272e-09 ||r(i)||/||b|| 3.307458023098e-10 95 KSP preconditioned resid norm 4.665959835776e-09 true resid norm 3.435941544160e-09 ||r(i)||/||b|| 3.470825093766e-10 Linear solve converged due to CONVERGED_RTOL iterations 95 Number of iterations = 95 Residual norm 3.43594e-09 The "nonzero" shift does not: $ ./ex10 -f binaryoutput -rhs -ksp_converged_reason -ksp_monitor_true_residual -ksp_rtol 1e-10 -pc_factor_shift_type nonzero 0 KSP preconditioned resid norm 7.686671891138e+01 true resid norm 9.899494936612e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.125291740842e+01 true resid norm 2.677042946539e+00 ||r(i)||/||b|| 2.704221744322e-01 2 KSP preconditioned resid norm 4.169562114751e-01 true resid norm 1.053544866606e-01 ||r(i)||/||b|| 1.064241027802e-02 3 KSP preconditioned resid norm 3.884939867076e-02 true resid norm 2.597689725056e-02 ||r(i)||/||b|| 2.624062885722e-03 4 KSP preconditioned resid norm 1.870022102589e-03 true resid norm 2.366328914521e-02 ||r(i)||/||b|| 2.390353174251e-03 5 KSP preconditioned resid norm 1.006640708018e-04 true resid norm 2.362168570871e-02 ||r(i)||/||b|| 2.386150592527e-03 6 KSP preconditioned resid norm 6.695456376917e-06 true resid norm 2.362181124027e-02 ||r(i)||/||b|| 2.386163273129e-03 7 KSP preconditioned resid norm 2.568374375496e-07 true resid norm 2.362165514116e-02 ||r(i)||/||b|| 2.386147504737e-03 8 KSP preconditioned resid norm 1.798370685705e-08 true resid norm 2.362165522405e-02 ||r(i)||/||b|| 2.386147513111e-03 9 KSP preconditioned resid norm 8.573160013296e-10 true resid norm 2.362165478290e-02 ||r(i)||/||b|| 2.386147468548e-03 Linear solve converged due to CONVERGED_RTOL iterations 9 Number of iterations = 9 Residual norm 0.0236217 Same story with "inblocks", but convergence appears to be fast if you use FGMRES, GCR, BiCG, or CGS. Now back to the failing case, I'm finding that if I write out the explicit preconditioned operator, then read that in with ex10 and execute with no preconditioner, I get nice convergence with either right or left-preconditioned GMRES. (See reproducible test case below. Now that's weird and I don't have an explanation. I've had this email half-written for a while and have tinkered with the code trying to explain it, but haven't had an extended block of time so I figure I should post this in case anyone else has ideas. The next thing I have in mind is to create the same RHS vector (after preconditioning) and compare the iterates which should be identical (though the true residual won't because the operator used to compute the true residual is different). $ ./ex10 -f binaryoutput -rhs -ksp_converged_reason -ksp_monitor_true_residual -ksp_rtol 1e-16 -pc_factor_shift_type nonzero -ksp_norm_type preconditioned -ksp_type gmres -ksp_view_preconditioned_operator_explicit binary:pcilu-left 0 KSP preconditioned resid norm 7.686671891138e+01 true resid norm 9.899494936612e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.125291740842e+01 true resid norm 2.677042946539e+00 ||r(i)||/||b|| 2.704221744322e-01 2 KSP preconditioned resid norm 4.169562114751e-01 true resid norm 1.053544866606e-01 ||r(i)||/||b|| 1.064241027802e-02 3 KSP preconditioned resid norm 3.884939867076e-02 true resid norm 2.597689725056e-02 ||r(i)||/||b|| 2.624062885722e-03 4 KSP preconditioned resid norm 1.870022102589e-03 true resid norm 2.366328914521e-02 ||r(i)||/||b|| 2.390353174251e-03 5 KSP preconditioned resid norm 1.006640708018e-04 true resid norm 2.362168570871e-02 ||r(i)||/||b|| 2.386150592527e-03 6 KSP preconditioned resid norm 6.695456376917e-06 true resid norm 2.362181124027e-02 ||r(i)||/||b|| 2.386163273129e-03 7 KSP preconditioned resid norm 2.568374375496e-07 true resid norm 2.362165514116e-02 ||r(i)||/||b|| 2.386147504737e-03 8 KSP preconditioned resid norm 1.798370685705e-08 true resid norm 2.362165522405e-02 ||r(i)||/||b|| 2.386147513111e-03 9 KSP preconditioned resid norm 8.573160013296e-10 true resid norm 2.362165478290e-02 ||r(i)||/||b|| 2.386147468548e-03 10 KSP preconditioned resid norm 4.225844037925e-11 true resid norm 2.362165479751e-02 ||r(i)||/||b|| 2.386147470024e-03 11 KSP preconditioned resid norm 1.874852209366e-12 true resid norm 2.362165479842e-02 ||r(i)||/||b|| 2.386147470116e-03 12 KSP preconditioned resid norm 6.799500125012e-14 true resid norm 2.362165479837e-02 ||r(i)||/||b|| 2.386147470110e-03 13 KSP preconditioned resid norm 7.912052053829e-15 true resid norm 2.362165479837e-02 ||r(i)||/||b|| 2.386147470110e-03 14 KSP preconditioned resid norm 5.551931114634e-15 true resid norm 2.362165479837e-02 ||r(i)||/||b|| 2.386147470110e-03 Linear solve converged due to CONVERGED_RTOL iterations 14 Number of iterations = 14 Residual norm 0.0236217 $ ./ex10 -f pcilu-left -rhs -ksp_converged_reason -ksp_monitor_true_residual -ksp_rtol 1e-16 -pc_type none -ksp_norm_type preconditioned -ksp_type gmres 0 KSP preconditioned resid norm 9.899494936612e+00 true resid norm 9.899494936612e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.184329742133e+00 true resid norm 2.184329742133e+00 ||r(i)||/||b|| 2.206506247157e-01 2 KSP preconditioned resid norm 1.849697585610e-01 true resid norm 1.849697585610e-01 ||r(i)||/||b|| 1.868476722756e-02 3 KSP preconditioned resid norm 7.233163029549e-03 true resid norm 7.233163029549e-03 ||r(i)||/||b|| 7.306598039460e-04 4 KSP preconditioned resid norm 3.819911952092e-04 true resid norm 3.819911952090e-04 ||r(i)||/||b|| 3.858693778369e-05 5 KSP preconditioned resid norm 1.940407608184e-05 true resid norm 1.940407608155e-05 ||r(i)||/||b|| 1.960107682846e-06 6 KSP preconditioned resid norm 1.446966001367e-06 true resid norm 1.446966000978e-06 ||r(i)||/||b|| 1.461656387769e-07 7 KSP preconditioned resid norm 4.828587794298e-08 true resid norm 4.828587816553e-08 ||r(i)||/||b|| 4.877610269485e-09 8 KSP preconditioned resid norm 3.454073475625e-09 true resid norm 3.454073609875e-09 ||r(i)||/||b|| 3.489141246086e-10 9 KSP preconditioned resid norm 2.149824726703e-10 true resid norm 2.149827018270e-10 ||r(i)||/||b|| 2.171653232853e-11 10 KSP preconditioned resid norm 9.143524324851e-12 true resid norm 9.143698087696e-12 ||r(i)||/||b|| 9.236529889904e-13 11 KSP preconditioned resid norm 1.841531740366e-13 true resid norm 1.838833676326e-13 ||r(i)||/||b|| 1.857502517149e-14 12 KSP preconditioned resid norm 9.365474613992e-15 true resid norm 9.795791624425e-15 ||r(i)||/||b|| 9.895243835316e-16 13 KSP preconditioned resid norm 3.696199042068e-15 true resid norm 4.825320989910e-15 ||r(i)||/||b|| 4.874310276239e-16 14 KSP preconditioned resid norm 2.770629172379e-15 true resid norm 5.677355330545e-15 ||r(i)||/||b|| 5.734994933477e-16 15 KSP preconditioned resid norm 2.303066887546e-15 true resid norm 5.583218758734e-15 ||r(i)||/||b|| 5.639902635926e-16 16 KSP preconditioned resid norm 2.012645615368e-15 true resid norm 5.463831520276e-15 ||r(i)||/||b|| 5.519303313211e-16 17 KSP preconditioned resid norm 1.810217685689e-15 true resid norm 4.363896801037e-15 ||r(i)||/||b|| 4.408201457731e-16 18 KSP preconditioned resid norm 1.658676365430e-15 true resid norm 5.310534564585e-15 ||r(i)||/||b|| 5.364450003348e-16 19 KSP preconditioned resid norm 1.539766535244e-15 true resid norm 5.279107563524e-15 ||r(i)||/||b|| 5.332703938259e-16 20 KSP preconditioned resid norm 1.443237688687e-15 true resid norm 5.579906258437e-15 ||r(i)||/||b|| 5.636556505323e-16 21 KSP preconditioned resid norm 1.362848936052e-15 true resid norm 4.342661167469e-15 ||r(i)||/||b|| 4.386750228446e-16 22 KSP preconditioned resid norm 1.294551799862e-15 true resid norm 5.338314359525e-15 ||r(i)||/||b|| 5.392511833894e-16 23 KSP preconditioned resid norm 1.235590344831e-15 true resid norm 5.297753476103e-15 ||r(i)||/||b|| 5.351539154296e-16 24 KSP preconditioned resid norm 1.184014779259e-15 true resid norm 5.576591790512e-15 ||r(i)||/||b|| 5.633208387115e-16 25 KSP preconditioned resid norm 1.138401595244e-15 true resid norm 4.325597599836e-15 ||r(i)||/||b|| 4.369513422184e-16 26 KSP preconditioned resid norm 1.097683968914e-15 true resid norm 5.338314359525e-15 ||r(i)||/||b|| 5.392511833894e-16 27 KSP preconditioned resid norm 1.061044509051e-15 true resid norm 5.260395559362e-15 ||r(i)||/||b|| 5.313801959641e-16 28 KSP preconditioned resid norm 1.027844958802e-15 true resid norm 5.482973286448e-15 ||r(i)||/||b|| 5.538639417018e-16 29 KSP preconditioned resid norm 9.975786885717e-16 true resid norm 4.325597599836e-15 ||r(i)||/||b|| 4.369513422184e-16 30 KSP preconditioned resid norm 9.698377326135e-16 true resid norm 5.338314359525e-15 ||r(i)||/||b|| 5.392511833894e-16 Linear solve converged due to CONVERGED_RTOL iterations 30 Number of iterations = 30 Residual norm < 1.e-12
signature.asc
Description: PGP signature
------------------------------------------------------------------------------ New Year. New Location. New Benefits. New Data Center in Ashburn, VA. GigeNET is offering a free month of service with a new server in Ashburn. Choose from 2 high performing configs, both with 100TB of bandwidth. Higher redundancy.Lower latency.Increased capacity.Completely compliant. http://p.sf.net/sfu/gigenet
_______________________________________________ Libmesh-devel mailing list Libmesh-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/libmesh-devel
