> On Sep 8, 2015, at 8:32 PM, Gideon Simpson <[email protected]> wrote: > > The message was unclear. I assumed that you meant if I went up to 3.6.1, > this would be corrected. What file am I editing and in which directory?
src/snes/interface/snesj.c function SNESComputeJacobianDefault > > > -gideon > >> On Sep 8, 2015, at 9:27 PM, Barry Smith <[email protected]> wrote: >> >> >> Did you add the line I told you to? >> >> >>> On Sep 8, 2015, at 8:23 PM, Gideon Simpson <[email protected]> wrote: >>> >>> I just installed a fresh 3.6.1 pull, compiled against it, and I’m still >>> having an error. >>> 0 SNES Function norm 1.421454390131e-02 >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Argument out of range >>> [0]PETSC ERROR: New nonzero at (3,0) caused a malloc >>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn >>> off this check >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for >>> trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown >>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air >>> by gideon Tue Sep 8 21:21:11 2015 >>> [0]PETSC ERROR: Configure options --download-mpich=yes >>> --download-suitesparse=yes --download-superlu=yes >>> --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes >>> --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes >>> --download-parmetis=yes --download-scalapack=yes >>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 485 in >>> /opt/petsc/src/mat/impls/aij/seq/aij.c >>> [0]PETSC ERROR: #2 MatSetValues() line 1173 in >>> /opt/petsc/src/mat/interface/matrix.c >>> [0]PETSC ERROR: #3 SNESComputeJacobianDefault() line 121 in >>> /opt/petsc/src/snes/interface/snesj.c >>> [0]PETSC ERROR: #4 SNESComputeJacobian() line 2232 in >>> /opt/petsc/src/snes/interface/snes.c >>> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 231 in >>> /opt/petsc/src/snes/impls/ls/ls.c >>> [0]PETSC ERROR: #6 SNESSolve() line 3906 in >>> /opt/petsc/src/snes/interface/snes.c >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Object is in wrong state >>> [0]PETSC ERROR: Vec is locked read only, argument # 1 >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for >>> trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown >>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air >>> by gideon Tue Sep 8 21:21:11 2015 >>> [0]PETSC ERROR: Configure options --download-mpich=yes >>> --download-suitesparse=yes --download-superlu=yes >>> --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes >>> --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes >>> --download-parmetis=yes --download-scalapack=yes >>> [0]PETSC ERROR: #7 VecGetArray() line 1646 in >>> /opt/petsc/src/vec/vec/interface/rvector.c >>> >>> >>> >>> >>> -gideon >>> >>>> On Sep 8, 2015, at 8:56 PM, Barry Smith <[email protected]> wrote: >>>> >>>> >>>> This is our fault. You need to add a line >>>> >>>> ierr = MatSetOption(B,MAT_NEW_NONZERO_ALLOCATION_ERR, >>>> PETSC_FALSE);CHKERRQ(ierr); >>>> >>>> after >>>> >>>> PetscFunctionBegin; >>>> >>>> in SNESComputeJacobianDefault() >>>> >>>> then do >>>> >>>> make gnumake >>>> >>>> in that directory. >>>> >>>> You really should update to PETSc 3.6 >>>> (http://www.mcs.anl.gov/petsc/download/index.html note it is best to >>>> obtain with git) It will make both your life and our lives easier and >>>> there is no downside. >>>> >>>> Barry >>>> >>>>> On Sep 8, 2015, at 6:24 PM, Gideon Simpson <[email protected]> >>>>> wrote: >>>>> >>>>> When I use that flag, I get this error: >>>>> >>>>> 0 SNES Function norm 1.421454390131e-02 >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Argument out of range >>>>> [0]PETSC ERROR: New nonzero at (3,0) caused a malloc >>>>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn >>>>> off this check >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 >>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named >>>>> gs_air by gideon Tue Sep 8 19:24:08 2015 >>>>> [0]PETSC ERROR: Configure options --download-mpich=yes >>>>> --download-suitesparse=yes --download-superlu=yes >>>>> --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes >>>>> --with-cxx=clang++ --with-cc=clang --with-fc=gfortran >>>>> --download-metis=yes --download-parmetis=yes --download-scalapack=yes >>>>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 487 in >>>>> /opt/petsc-3.5.4/src/mat/impls/aij/seq/aij.c >>>>> [0]PETSC ERROR: #2 MatSetValues() line 1135 in >>>>> /opt/petsc-3.5.4/src/mat/interface/matrix.c >>>>> [0]PETSC ERROR: #3 SNESComputeJacobianDefault() line 120 in >>>>> /opt/petsc-3.5.4/src/snes/interface/snesj.c >>>>> [0]PETSC ERROR: #4 SNESComputeJacobian() line 2193 in >>>>> /opt/petsc-3.5.4/src/snes/interface/snes.c >>>>> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 230 in >>>>> /opt/petsc-3.5.4/src/snes/impls/ls/ls.c >>>>> [0]PETSC ERROR: #6 SNESSolve() line 3743 in >>>>> /opt/petsc-3.5.4/src/snes/interface/snes.c >>>>> >>>>> -gideon >>>>> >>>>>> On Sep 8, 2015, at 7:18 PM, Barry Smith <[email protected]> wrote: >>>>>> >>>>>> >>>>>> You can run with -snes_fd it will be slow so run with only two levels of >>>>>> grid sequencing; check the convergence rate of Newton on the finer grid. >>>>>> It will hopefully be much better (also KSP should converge in one >>>>>> iteration). Send the output >>>>>> >>>>>> Barry >>>>>> >>>>>>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>> I went through my code and nothing jumps out at me as clearly wrong. >>>>>>> Is there a way to dump to disk vectors and matrices that would be >>>>>>> useful for the comparison? I imagine I’d want the jacobian that I am >>>>>>> specifying, but what would I want to compare that against in terms of >>>>>>> using -snes_mf_operator? >>>>>>> >>>>>>> >>>>>>> -gideon >>>>>>> >>>>>>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <[email protected]> wrote: >>>>>>>> >>>>>>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson >>>>>>>> <[email protected]> wrote: >>>>>>>> Just to clarify, again, during the linear solve phase, even though the >>>>>>>> PC residual norm was small (getting to 10^{-10} in 2-3 iterations) >>>>>>>> that the true residual norm could be comparatively large signified >>>>>>>> there is something wrong. The one thing i was curious about is why it >>>>>>>> seems that there is only an issue during grid sequencing. I’m not >>>>>>>> seeing that kind of behavior on the coarsest mesh. >>>>>>>> >>>>>>>> That definitely sounds like a bug in the Jacobian evaluation. Maybe >>>>>>>> you assume a certain mesh size, and when >>>>>>>> PETSc refines the DA and passes in the new one something goes wrong? >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> -gideon >>>>>>>> >>>>>>>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <[email protected]> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson >>>>>>>>>> <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>> Alright, I’ll take another look at what has been implemented in my >>>>>>>>>> Jacobian file. One thing I’m a bit unclear about is do I need these >>>>>>>>>> -snes_mf and snes_mf_operator flags if I have, coded in, >>>>>>>>>> >>>>>>>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx); >>>>>>>>> >>>>>>>>> Gideon, >>>>>>>>> >>>>>>>>> In your case the DM creates that Jacobian matrix object and >>>>>>>>> form_function_jacobian's job is just to fill it up with the correct >>>>>>>>> values for the current solution value. If you use -snes_mf_operator >>>>>>>>> then SNES uses a matrix-free multiple for the first matrix and your >>>>>>>>> Jacobian to build the preconditioner. With -snes_mf it uses a >>>>>>>>> matrix-free multiple for the first and second matrix and ignores what >>>>>>>>> you compute. Since it is matrix free in both it cannot build a >>>>>>>>> preconditioner hence the -pc_type none >>>>>>>>> >>>>>>>>> Using -snes_mf and -snes_mf_operator are a way of testing if the >>>>>>>>> Jacobian you provide is correct >>>>>>>>> >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -gideon >>>>>>>>>> >>>>>>>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <[email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> If the Jacobian you provide is correct then using -snes_mf with a >>>>>>>>>>> PC type of LU should give two iterations (or at most three) to get >>>>>>>>>>> very good (10^-12) convergence of the linear system. Here you are >>>>>>>>>>> not getting that after a few SNES iterations. So I am strongly >>>>>>>>>>> leaning to something wrong with the Jacobian you provide. >>>>>>>>>>> -snes_type test unfortunately doesn't catch all problems. >>>>>>>>>>> >>>>>>>>>>> You can try -snes_mf -pc_type none and see how the linear solver >>>>>>>>>>> converges. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Barry >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson >>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Off the top of my head, I can’t rule out a null space in this >>>>>>>>>>>> problem. I thought I did a pretty good job of coding the Jacobian >>>>>>>>>>>> by hand. And before I made that change to the Jacobian file, >>>>>>>>>>>> changing J for Jpre, if I ran with -snes_check_jacobian, it >>>>>>>>>>>> appeared to be satisfactory. >>>>>>>>>>>> >>>>>>>>>>>> When I run with -ksp_type fgmres, I get >>>>>>>>>>>> >>>>>>>>>>>> 0 SNES Function norm 3.857327250267e-09 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid >>>>>>>>>>>> norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid >>>>>>>>>>>> norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid >>>>>>>>>>>> norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04 >>>>>>>>>>>> 1 SNES Function norm 2.479697868412e-12 >>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE >>>>>>>>>>>> iterations 1 >>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid >>>>>>>>>>>> norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid >>>>>>>>>>>> norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06 >>>>>>>>>>>> 1 SNES Function norm 8.484696862318e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid >>>>>>>>>>>> norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid >>>>>>>>>>>> norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid >>>>>>>>>>>> norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04 >>>>>>>>>>>> 2 SNES Function norm 6.551139809777e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid >>>>>>>>>>>> norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid >>>>>>>>>>>> norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid >>>>>>>>>>>> norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04 >>>>>>>>>>>> 3 SNES Function norm 5.771483337231e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid >>>>>>>>>>>> norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid >>>>>>>>>>>> norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid >>>>>>>>>>>> norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04 >>>>>>>>>>>> 4 SNES Function norm 5.237540245670e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid >>>>>>>>>>>> norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid >>>>>>>>>>>> norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid >>>>>>>>>>>> norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04 >>>>>>>>>>>> 5 SNES Function norm 4.753058321803e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid >>>>>>>>>>>> norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid >>>>>>>>>>>> norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid >>>>>>>>>>>> norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04 >>>>>>>>>>>> 6 SNES Function norm 4.221482124411e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid >>>>>>>>>>>> norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid >>>>>>>>>>>> norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid >>>>>>>>>>>> norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04 >>>>>>>>>>>> 7 SNES Function norm 3.803620028964e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid >>>>>>>>>>>> norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid >>>>>>>>>>>> norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid >>>>>>>>>>>> norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04 >>>>>>>>>>>> 8 SNES Function norm 3.769519577570e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid >>>>>>>>>>>> norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid >>>>>>>>>>>> norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid >>>>>>>>>>>> norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03 >>>>>>>>>>>> 3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid >>>>>>>>>>>> norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03 >>>>>>>>>>>> 9 SNES Function norm 3.766389203406e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid >>>>>>>>>>>> norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid >>>>>>>>>>>> norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid >>>>>>>>>>>> norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02 >>>>>>>>>>>> 10 SNES Function norm 3.751822489648e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid >>>>>>>>>>>> norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid >>>>>>>>>>>> norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid >>>>>>>>>>>> norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03 >>>>>>>>>>>> 11 SNES Function norm 3.737189963925e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid >>>>>>>>>>>> norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid >>>>>>>>>>>> norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid >>>>>>>>>>>> norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02 >>>>>>>>>>>> 3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid >>>>>>>>>>>> norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02 >>>>>>>>>>>> 12 SNES Function norm 3.737110331594e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid >>>>>>>>>>>> norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid >>>>>>>>>>>> norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid >>>>>>>>>>>> norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04 >>>>>>>>>>>> 13 SNES Function norm 3.684175513154e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid >>>>>>>>>>>> norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid >>>>>>>>>>>> norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid >>>>>>>>>>>> norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04 >>>>>>>>>>>> 14 SNES Function norm 3.509930783933e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid >>>>>>>>>>>> norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid >>>>>>>>>>>> norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid >>>>>>>>>>>> norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05 >>>>>>>>>>>> 15 SNES Function norm 3.162121096565e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid >>>>>>>>>>>> norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid >>>>>>>>>>>> norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid >>>>>>>>>>>> norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05 >>>>>>>>>>>> 16 SNES Function norm 2.890165052818e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid >>>>>>>>>>>> norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid >>>>>>>>>>>> norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid >>>>>>>>>>>> norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04 >>>>>>>>>>>> 17 SNES Function norm 2.816929421445e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid >>>>>>>>>>>> norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid >>>>>>>>>>>> norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid >>>>>>>>>>>> norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03 >>>>>>>>>>>> 3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid >>>>>>>>>>>> norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03 >>>>>>>>>>>> 18 SNES Function norm 2.814407559738e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid >>>>>>>>>>>> norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid >>>>>>>>>>>> norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid >>>>>>>>>>>> norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04 >>>>>>>>>>>> 19 SNES Function norm 2.810261821835e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid >>>>>>>>>>>> norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid >>>>>>>>>>>> norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid >>>>>>>>>>>> norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04 >>>>>>>>>>>> 20 SNES Function norm 2.755723672077e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid >>>>>>>>>>>> norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid >>>>>>>>>>>> norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid >>>>>>>>>>>> norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02 >>>>>>>>>>>> 3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid >>>>>>>>>>>> norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02 >>>>>>>>>>>> 4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid >>>>>>>>>>>> norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02 >>>>>>>>>>>> 5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid >>>>>>>>>>>> norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02 >>>>>>>>>>>> 6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid >>>>>>>>>>>> norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02 >>>>>>>>>>>> 21 SNES Function norm 2.755720389981e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid >>>>>>>>>>>> norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid >>>>>>>>>>>> norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid >>>>>>>>>>>> norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02 >>>>>>>>>>>> 3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid >>>>>>>>>>>> norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02 >>>>>>>>>>>> 4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid >>>>>>>>>>>> norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02 >>>>>>>>>>>> 5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid >>>>>>>>>>>> norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02 >>>>>>>>>>>> 6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid >>>>>>>>>>>> norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02 >>>>>>>>>>>> 22 SNES Function norm 2.755716464881e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid >>>>>>>>>>>> norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid >>>>>>>>>>>> norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid >>>>>>>>>>>> norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02 >>>>>>>>>>>> 3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid >>>>>>>>>>>> norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02 >>>>>>>>>>>> 4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid >>>>>>>>>>>> norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02 >>>>>>>>>>>> 23 SNES Function norm 2.755713163168e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid >>>>>>>>>>>> norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid >>>>>>>>>>>> norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid >>>>>>>>>>>> norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02 >>>>>>>>>>>> 3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid >>>>>>>>>>>> norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02 >>>>>>>>>>>> 4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid >>>>>>>>>>>> norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02 >>>>>>>>>>>> 5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid >>>>>>>>>>>> norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02 >>>>>>>>>>>> 6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid >>>>>>>>>>>> norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02 >>>>>>>>>>>> 7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid >>>>>>>>>>>> norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02 >>>>>>>>>>>> 8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid >>>>>>>>>>>> norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02 >>>>>>>>>>>> 24 SNES Function norm 2.755712363532e+02 >>>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid >>>>>>>>>>>> norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid >>>>>>>>>>>> norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01 >>>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid >>>>>>>>>>>> norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01 >>>>>>>>>>>> 3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid >>>>>>>>>>>> norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01 >>>>>>>>>>>> 4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid >>>>>>>>>>>> norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01 >>>>>>>>>>>> 5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid >>>>>>>>>>>> norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01 >>>>>>>>>>>> 6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid >>>>>>>>>>>> norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01 >>>>>>>>>>>> 7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid >>>>>>>>>>>> norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01 >>>>>>>>>>>> 8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid >>>>>>>>>>>> norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01 >>>>>>>>>>>> 9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid >>>>>>>>>>>> norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01 >>>>>>>>>>>> 10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid >>>>>>>>>>>> norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01 >>>>>>>>>>>> 11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid >>>>>>>>>>>> norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01 >>>>>>>>>>>> 12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid >>>>>>>>>>>> norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01 >>>>>>>>>>>> 13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid >>>>>>>>>>>> norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01 >>>>>>>>>>>> 14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid >>>>>>>>>>>> norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01 >>>>>>>>>>>> 15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid >>>>>>>>>>>> norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01 >>>>>>>>>>>> 16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid >>>>>>>>>>>> norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01 >>>>>>>>>>>> 17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid >>>>>>>>>>>> norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01 >>>>>>>>>>>> 18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid >>>>>>>>>>>> norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01 >>>>>>>>>>>> 19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid >>>>>>>>>>>> norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01 >>>>>>>>>>>> 20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid >>>>>>>>>>>> norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01 >>>>>>>>>>>> 21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid >>>>>>>>>>>> norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01 >>>>>>>>>>>> 22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid >>>>>>>>>>>> norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01 >>>>>>>>>>>> 23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid >>>>>>>>>>>> norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01 >>>>>>>>>>>> 24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid >>>>>>>>>>>> norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01 >>>>>>>>>>>> 25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid >>>>>>>>>>>> norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01 >>>>>>>>>>>> 26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid >>>>>>>>>>>> norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01 >>>>>>>>>>>> 27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid >>>>>>>>>>>> norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01 >>>>>>>>>>>> 28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid >>>>>>>>>>>> norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00 >>>>>>>>>>>> 29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid >>>>>>>>>>>> norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01 >>>>>>>>>>>> 30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid >>>>>>>>>>>> norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00 >>>>>>>>>>>> 31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid >>>>>>>>>>>> norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00 >>>>>>>>>>>> 32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid >>>>>>>>>>>> norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01 >>>>>>>>>>>> 33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid >>>>>>>>>>>> norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01 >>>>>>>>>>>> 34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid >>>>>>>>>>>> norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00 >>>>>>>>>>>> 35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid >>>>>>>>>>>> norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00 >>>>>>>>>>>> 36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid >>>>>>>>>>>> norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00 >>>>>>>>>>>> 37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid >>>>>>>>>>>> norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00 >>>>>>>>>>>> 38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid >>>>>>>>>>>> norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00 >>>>>>>>>>>> 39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid >>>>>>>>>>>> norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00 >>>>>>>>>>>> 40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid >>>>>>>>>>>> norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00 >>>>>>>>>>>> 41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid >>>>>>>>>>>> norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00 >>>>>>>>>>>> 42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid >>>>>>>>>>>> norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00 >>>>>>>>>>>> 43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid >>>>>>>>>>>> norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00 >>>>>>>>>>>> 44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid >>>>>>>>>>>> norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00 >>>>>>>>>>>> 45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid >>>>>>>>>>>> norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00 >>>>>>>>>>>> 46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid >>>>>>>>>>>> norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00 >>>>>>>>>>>> 47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid >>>>>>>>>>>> norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00 >>>>>>>>>>>> 48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid >>>>>>>>>>>> norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00 >>>>>>>>>>>> 49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid >>>>>>>>>>>> norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00 >>>>>>>>>>>> 50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid >>>>>>>>>>>> norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00 >>>>>>>>>>>> 51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid >>>>>>>>>>>> norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00 >>>>>>>>>>>> 52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid >>>>>>>>>>>> norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00 >>>>>>>>>>>> 53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid >>>>>>>>>>>> norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00 >>>>>>>>>>>> 54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid >>>>>>>>>>>> norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00 >>>>>>>>>>>> 55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid >>>>>>>>>>>> norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00 >>>>>>>>>>>> 56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid >>>>>>>>>>>> norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00 >>>>>>>>>>>> 57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid >>>>>>>>>>>> norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00 >>>>>>>>>>>> 58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid >>>>>>>>>>>> norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00 >>>>>>>>>>>> 59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid >>>>>>>>>>>> norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00 >>>>>>>>>>>> 60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid >>>>>>>>>>>> norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00 >>>>>>>>>>>> 61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid >>>>>>>>>>>> norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00 >>>>>>>>>>>> 62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid >>>>>>>>>>>> norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00 >>>>>>>>>>>> 63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid >>>>>>>>>>>> norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00 >>>>>>>>>>>> 64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid >>>>>>>>>>>> norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00 >>>>>>>>>>>> 65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid >>>>>>>>>>>> norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00 >>>>>>>>>>>> 66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid >>>>>>>>>>>> norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00 >>>>>>>>>>>> 67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid >>>>>>>>>>>> norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00 >>>>>>>>>>>> 68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid >>>>>>>>>>>> norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00 >>>>>>>>>>>> 69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid >>>>>>>>>>>> norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00 >>>>>>>>>>>> 70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid >>>>>>>>>>>> norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01 >>>>>>>>>>>> 71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid >>>>>>>>>>>> norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01 >>>>>>>>>>>> 72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid >>>>>>>>>>>> norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01 >>>>>>>>>>>> 73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid >>>>>>>>>>>> norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01 >>>>>>>>>>>> 74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid >>>>>>>>>>>> norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01 >>>>>>>>>>>> 75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid >>>>>>>>>>>> norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01 >>>>>>>>>>>> 76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid >>>>>>>>>>>> norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01 >>>>>>>>>>>> 77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid >>>>>>>>>>>> norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01 >>>>>>>>>>>> 78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid >>>>>>>>>>>> norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01 >>>>>>>>>>>> 79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid >>>>>>>>>>>> norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01 >>>>>>>>>>>> 80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid >>>>>>>>>>>> norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01 >>>>>>>>>>>> 81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid >>>>>>>>>>>> norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01 >>>>>>>>>>>> 82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid >>>>>>>>>>>> norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01 >>>>>>>>>>>> 83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid >>>>>>>>>>>> norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01 >>>>>>>>>>>> 84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid >>>>>>>>>>>> norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01 >>>>>>>>>>>> 85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid >>>>>>>>>>>> norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01 >>>>>>>>>>>> 86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid >>>>>>>>>>>> norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01 >>>>>>>>>>>> 87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid >>>>>>>>>>>> norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01 >>>>>>>>>>>> 88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid >>>>>>>>>>>> norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01 >>>>>>>>>>>> 89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid >>>>>>>>>>>> norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01 >>>>>>>>>>>> 90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid >>>>>>>>>>>> norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01 >>>>>>>>>>>> 91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid >>>>>>>>>>>> norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01 >>>>>>>>>>>> 92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid >>>>>>>>>>>> norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01 >>>>>>>>>>>> 93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid >>>>>>>>>>>> norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01 >>>>>>>>>>>> 94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid >>>>>>>>>>>> norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01 >>>>>>>>>>>> 95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid >>>>>>>>>>>> norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01 >>>>>>>>>>>> 96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid >>>>>>>>>>>> norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01 >>>>>>>>>>>> 97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid >>>>>>>>>>>> norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01 >>>>>>>>>>>> 98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid >>>>>>>>>>>> norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02 >>>>>>>>>>>> 99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid >>>>>>>>>>>> norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02 >>>>>>>>>>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid >>>>>>>>>>>> norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02 >>>>>>>>>>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid >>>>>>>>>>>> norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02 >>>>>>>>>>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid >>>>>>>>>>>> norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02 >>>>>>>>>>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid >>>>>>>>>>>> norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02 >>>>>>>>>>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid >>>>>>>>>>>> norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02 >>>>>>>>>>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid >>>>>>>>>>>> norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02 >>>>>>>>>>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid >>>>>>>>>>>> norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02 >>>>>>>>>>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid >>>>>>>>>>>> norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02 >>>>>>>>>>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid >>>>>>>>>>>> norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02 >>>>>>>>>>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid >>>>>>>>>>>> norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02 >>>>>>>>>>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid >>>>>>>>>>>> norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02 >>>>>>>>>>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid >>>>>>>>>>>> norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02 >>>>>>>>>>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid >>>>>>>>>>>> norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02 >>>>>>>>>>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid >>>>>>>>>>>> norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02 >>>>>>>>>>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid >>>>>>>>>>>> norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02 >>>>>>>>>>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid >>>>>>>>>>>> norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02 >>>>>>>>>>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid >>>>>>>>>>>> norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02 >>>>>>>>>>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid >>>>>>>>>>>> norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02 >>>>>>>>>>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid >>>>>>>>>>>> norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02 >>>>>>>>>>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid >>>>>>>>>>>> norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02 >>>>>>>>>>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid >>>>>>>>>>>> norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02 >>>>>>>>>>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid >>>>>>>>>>>> norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02 >>>>>>>>>>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid >>>>>>>>>>>> norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02 >>>>>>>>>>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid >>>>>>>>>>>> norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02 >>>>>>>>>>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid >>>>>>>>>>>> norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02 >>>>>>>>>>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid >>>>>>>>>>>> norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02 >>>>>>>>>>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid >>>>>>>>>>>> norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02 >>>>>>>>>>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid >>>>>>>>>>>> norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02 >>>>>>>>>>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid >>>>>>>>>>>> norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02 >>>>>>>>>>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid >>>>>>>>>>>> norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02 >>>>>>>>>>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid >>>>>>>>>>>> norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02 >>>>>>>>>>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid >>>>>>>>>>>> norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02 >>>>>>>>>>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid >>>>>>>>>>>> norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02 >>>>>>>>>>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid >>>>>>>>>>>> norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02 >>>>>>>>>>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid >>>>>>>>>>>> norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02 >>>>>>>>>>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid >>>>>>>>>>>> norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02 >>>>>>>>>>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid >>>>>>>>>>>> norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02 >>>>>>>>>>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid >>>>>>>>>>>> norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02 >>>>>>>>>>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid >>>>>>>>>>>> norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02 >>>>>>>>>>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid >>>>>>>>>>>> norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02 >>>>>>>>>>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid >>>>>>>>>>>> norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02 >>>>>>>>>>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid >>>>>>>>>>>> norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02 >>>>>>>>>>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid >>>>>>>>>>>> norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02 >>>>>>>>>>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid >>>>>>>>>>>> norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02 >>>>>>>>>>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid >>>>>>>>>>>> norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02 >>>>>>>>>>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid >>>>>>>>>>>> norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02 >>>>>>>>>>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid >>>>>>>>>>>> norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02 >>>>>>>>>>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid >>>>>>>>>>>> norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02 >>>>>>>>>>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid >>>>>>>>>>>> norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02 >>>>>>>>>>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid >>>>>>>>>>>> norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02 >>>>>>>>>>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid >>>>>>>>>>>> norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02 >>>>>>>>>>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid >>>>>>>>>>>> norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02 >>>>>>>>>>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid >>>>>>>>>>>> norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02 >>>>>>>>>>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid >>>>>>>>>>>> norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02 >>>>>>>>>>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid >>>>>>>>>>>> norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02 >>>>>>>>>>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid >>>>>>>>>>>> norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02 >>>>>>>>>>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid >>>>>>>>>>>> norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02 >>>>>>>>>>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid >>>>>>>>>>>> norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02 >>>>>>>>>>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid >>>>>>>>>>>> norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02 >>>>>>>>>>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid >>>>>>>>>>>> norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02 >>>>>>>>>>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid >>>>>>>>>>>> norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02 >>>>>>>>>>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid >>>>>>>>>>>> norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02 >>>>>>>>>>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid >>>>>>>>>>>> norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02 >>>>>>>>>>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid >>>>>>>>>>>> norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02 >>>>>>>>>>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid >>>>>>>>>>>> norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02 >>>>>>>>>>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid >>>>>>>>>>>> norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02 >>>>>>>>>>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid >>>>>>>>>>>> norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02 >>>>>>>>>>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid >>>>>>>>>>>> norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02 >>>>>>>>>>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid >>>>>>>>>>>> norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02 >>>>>>>>>>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid >>>>>>>>>>>> norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02 >>>>>>>>>>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid >>>>>>>>>>>> norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02 >>>>>>>>>>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid >>>>>>>>>>>> norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02 >>>>>>>>>>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid >>>>>>>>>>>> norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02 >>>>>>>>>>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid >>>>>>>>>>>> norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02 >>>>>>>>>>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid >>>>>>>>>>>> norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02 >>>>>>>>>>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid >>>>>>>>>>>> norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02 >>>>>>>>>>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid >>>>>>>>>>>> norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02 >>>>>>>>>>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid >>>>>>>>>>>> norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02 >>>>>>>>>>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid >>>>>>>>>>>> norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02 >>>>>>>>>>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid >>>>>>>>>>>> norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02 >>>>>>>>>>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid >>>>>>>>>>>> norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02 >>>>>>>>>>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid >>>>>>>>>>>> norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02 >>>>>>>>>>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid >>>>>>>>>>>> norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02 >>>>>>>>>>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid >>>>>>>>>>>> norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02 >>>>>>>>>>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid >>>>>>>>>>>> norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02 >>>>>>>>>>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid >>>>>>>>>>>> norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02 >>>>>>>>>>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid >>>>>>>>>>>> norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03 >>>>>>>>>>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid >>>>>>>>>>>> norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03 >>>>>>>>>>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid >>>>>>>>>>>> norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03 >>>>>>>>>>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid >>>>>>>>>>>> norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03 >>>>>>>>>>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid >>>>>>>>>>>> norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03 >>>>>>>>>>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid >>>>>>>>>>>> norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03 >>>>>>>>>>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid >>>>>>>>>>>> norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03 >>>>>>>>>>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid >>>>>>>>>>>> norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03 >>>>>>>>>>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid >>>>>>>>>>>> norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03 >>>>>>>>>>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid >>>>>>>>>>>> norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03 >>>>>>>>>>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid >>>>>>>>>>>> norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03 >>>>>>>>>>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid >>>>>>>>>>>> norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03 >>>>>>>>>>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid >>>>>>>>>>>> norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03 >>>>>>>>>>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid >>>>>>>>>>>> norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03 >>>>>>>>>>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid >>>>>>>>>>>> norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03 >>>>>>>>>>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid >>>>>>>>>>>> norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03 >>>>>>>>>>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid >>>>>>>>>>>> norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03 >>>>>>>>>>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid >>>>>>>>>>>> norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03 >>>>>>>>>>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid >>>>>>>>>>>> norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03 >>>>>>>>>>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid >>>>>>>>>>>> norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03 >>>>>>>>>>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid >>>>>>>>>>>> norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03 >>>>>>>>>>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid >>>>>>>>>>>> norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03 >>>>>>>>>>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid >>>>>>>>>>>> norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04 >>>>>>>>>>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid >>>>>>>>>>>> norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04 >>>>>>>>>>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid >>>>>>>>>>>> norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04 >>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE >>>>>>>>>>>> iterations 24 >>>>>>>>>>>> >>>>>>>>>>>> -gideon >>>>>>>>>>>> >>>>>>>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Is there any chance your Jacobian may have a null space? (For >>>>>>>>>>>>> example with Neumann boundary conditions there is often a >>>>>>>>>>>>> nullspace) >>>>>>>>>>>>> >>>>>>>>>>>>> If not run with with -ksp_type fgmres and send the same output. >>>>>>>>>>>>> >>>>>>>>>>>>> Something is screwy with the linear system convergence, could be >>>>>>>>>>>>> several things including either a null space problem or that your >>>>>>>>>>>>> Jacobian that you provide is terribly wrong. >>>>>>>>>>>>> >>>>>>>>>>>>> Barry >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 >>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.076322725615e+05 true resid >>>>>>>>>>>>> norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.990965035897e+01 true resid >>>>>>>>>>>>> norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01 >>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941661213523e-05 true resid >>>>>>>>>>>>> norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson >>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Got it, fixing that, and returning to the original question, >>>>>>>>>>>>>> this is what I now get, when I use those two flags: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 0 SNES Function norm 1.132185384796e-08 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.846486104194e-09 true resid >>>>>>>>>>>>>> norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.813743504021e-14 true resid >>>>>>>>>>>>>> norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05 >>>>>>>>>>>>>> 1 SNES Function norm 2.177599365111e-12 >>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE >>>>>>>>>>>>>> iterations 1 >>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.135087050041e+01 true resid >>>>>>>>>>>>>> norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.498184175046e-04 true resid >>>>>>>>>>>>>> norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05 >>>>>>>>>>>>>> 1 SNES Function norm 8.482593852817e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.667293100105e+02 true resid >>>>>>>>>>>>>> norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.682642450862e-02 true resid >>>>>>>>>>>>>> norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.943933962588e-08 true resid >>>>>>>>>>>>>> norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04 >>>>>>>>>>>>>> 2 SNES Function norm 6.543140468549e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.538341367076e+02 true resid >>>>>>>>>>>>>> norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.294478035133e-03 true resid >>>>>>>>>>>>>> norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.879002692918e-08 true resid >>>>>>>>>>>>>> norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04 >>>>>>>>>>>>>> 3 SNES Function norm 5.766430557220e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.362249760236e+02 true resid >>>>>>>>>>>>>> norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.565721843210e-02 true resid >>>>>>>>>>>>>> norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.900043171759e-08 true resid >>>>>>>>>>>>>> norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04 >>>>>>>>>>>>>> 4 SNES Function norm 5.235211958260e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.533523981591e+02 true resid >>>>>>>>>>>>>> norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.030969535942e-02 true resid >>>>>>>>>>>>>> norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.066614364159e-08 true resid >>>>>>>>>>>>>> norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04 >>>>>>>>>>>>>> 5 SNES Function norm 4.752913229649e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.848133067820e+02 true resid >>>>>>>>>>>>>> norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.745259702307e-02 true resid >>>>>>>>>>>>>> norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.045643600156e-07 true resid >>>>>>>>>>>>>> norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04 >>>>>>>>>>>>>> 6 SNES Function norm 4.220255380391e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.549151165192e+02 true resid >>>>>>>>>>>>>> norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.794436126782e-02 true resid >>>>>>>>>>>>>> norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.890094997586e-07 true resid >>>>>>>>>>>>>> norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04 >>>>>>>>>>>>>> 7 SNES Function norm 3.805408907074e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.225410707531e+02 true resid >>>>>>>>>>>>>> norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.810631909992e-03 true resid >>>>>>>>>>>>>> norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04 >>>>>>>>>>>>>> 8 SNES Function norm 3.764619752339e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 6.256830807263e+03 true resid >>>>>>>>>>>>>> norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.034356591467e+00 true resid >>>>>>>>>>>>>> norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.014544916729e-05 true resid >>>>>>>>>>>>>> norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03 >>>>>>>>>>>>>> 9 SNES Function norm 3.761182227091e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.988264999623e+03 true resid >>>>>>>>>>>>>> norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.115167379340e-01 true resid >>>>>>>>>>>>>> norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.515961219888e-06 true resid >>>>>>>>>>>>>> norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03 >>>>>>>>>>>>>> 10 SNES Function norm 3.740017190063e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.142561025658e+03 true resid >>>>>>>>>>>>>> norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.746046512685e-02 true resid >>>>>>>>>>>>>> norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.588755510089e-06 true resid >>>>>>>>>>>>>> norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03 >>>>>>>>>>>>>> 11 SNES Function norm 3.725903477238e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.435759974780e+03 true resid >>>>>>>>>>>>>> norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 7.389097536786e-01 true resid >>>>>>>>>>>>>> norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.785754916084e-06 true resid >>>>>>>>>>>>>> norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03 >>>>>>>>>>>>>> 12 SNES Function norm 3.716162097231e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.010267719047e+02 true resid >>>>>>>>>>>>>> norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.404383077173e-01 true resid >>>>>>>>>>>>>> norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.431973894553e-07 true resid >>>>>>>>>>>>>> norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04 >>>>>>>>>>>>>> 13 SNES Function norm 3.674168632847e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.697546464095e+02 true resid >>>>>>>>>>>>>> norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.100811362173e-01 true resid >>>>>>>>>>>>>> norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.921252511737e-07 true resid >>>>>>>>>>>>>> norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04 >>>>>>>>>>>>>> 14 SNES Function norm 3.532395445266e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.455211776575e+01 true resid >>>>>>>>>>>>>> norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.681410748806e-02 true resid >>>>>>>>>>>>>> norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.060849362790e-07 true resid >>>>>>>>>>>>>> norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05 >>>>>>>>>>>>>> 15 SNES Function norm 3.182438872366e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.515566740696e+02 true resid >>>>>>>>>>>>>> norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.959786024091e-02 true resid >>>>>>>>>>>>>> norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.945498964959e-07 true resid >>>>>>>>>>>>>> norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04 >>>>>>>>>>>>>> 16 SNES Function norm 3.091759892779e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 7.834551377727e+01 true resid >>>>>>>>>>>>>> norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.992168699741e-02 true resid >>>>>>>>>>>>>> norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 3.192091055088e-07 true resid >>>>>>>>>>>>>> norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04 >>>>>>>>>>>>>> 17 SNES Function norm 2.987839504359e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 9.384044359119e+02 true resid >>>>>>>>>>>>>> norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.206851078700e-01 true resid >>>>>>>>>>>>>> norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.074373330828e-06 true resid >>>>>>>>>>>>>> norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03 >>>>>>>>>>>>>> 18 SNES Function norm 2.987073622777e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 9.576546391157e+03 true resid >>>>>>>>>>>>>> norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.670797073017e+00 true resid >>>>>>>>>>>>>> norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.278475684614e-05 true resid >>>>>>>>>>>>>> norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02 >>>>>>>>>>>>>> 19 SNES Function norm 2.987067936734e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.381986890301e+04 true resid >>>>>>>>>>>>>> norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.870471171439e+01 true resid >>>>>>>>>>>>>> norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.628164409392e-04 true resid >>>>>>>>>>>>>> norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02 >>>>>>>>>>>>>> 20 SNES Function norm 2.987067502910e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.191539882357e+04 true resid >>>>>>>>>>>>>> norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.528735548091e+01 true resid >>>>>>>>>>>>>> norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 5.890475574699e-05 true resid >>>>>>>>>>>>>> norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02 >>>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.076322725615e+05 true resid >>>>>>>>>>>>>> norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.990965035897e+01 true resid >>>>>>>>>>>>>> norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941661213523e-05 true resid >>>>>>>>>>>>>> norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01 >>>>>>>>>>>>>> 22 SNES Function norm 2.987064525262e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.312500251239e+06 true resid >>>>>>>>>>>>>> norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.194187039215e+01 true resid >>>>>>>>>>>>>> norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.998846844269e-04 true resid >>>>>>>>>>>>>> norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02 >>>>>>>>>>>>>> 23 SNES Function norm 2.987064121622e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.022854344216e+05 true resid >>>>>>>>>>>>>> norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.602644501358e+00 true resid >>>>>>>>>>>>>> norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 9.288860381060e-05 true resid >>>>>>>>>>>>>> norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01 >>>>>>>>>>>>>> 24 SNES Function norm 2.987063973426e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.876191744212e+05 true resid >>>>>>>>>>>>>> norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.701586788162e+01 true resid >>>>>>>>>>>>>> norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.533781032398e-04 true resid >>>>>>>>>>>>>> norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01 >>>>>>>>>>>>>> 25 SNES Function norm 2.987063920553e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.373321357604e+05 true resid >>>>>>>>>>>>>> norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.804687152206e+02 true resid >>>>>>>>>>>>>> norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.556287913942e-03 true resid >>>>>>>>>>>>>> norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00 >>>>>>>>>>>>>> 26 SNES Function norm 2.987063919786e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.292036277396e+05 true resid >>>>>>>>>>>>>> norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.282180994978e+01 true resid >>>>>>>>>>>>>> norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.163542587614e-04 true resid >>>>>>>>>>>>>> norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02 >>>>>>>>>>>>>> 27 SNES Function norm 2.987063393145e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.447152171843e+05 true resid >>>>>>>>>>>>>> norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.283727214797e+01 true resid >>>>>>>>>>>>>> norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 4.585761865240e-04 true resid >>>>>>>>>>>>>> norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01 >>>>>>>>>>>>>> 28 SNES Function norm 2.987063253277e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.785016507221e+05 true resid >>>>>>>>>>>>>> norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.095374466008e+01 true resid >>>>>>>>>>>>>> norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 9.617782709610e-05 true resid >>>>>>>>>>>>>> norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01 >>>>>>>>>>>>>> 29 SNES Function norm 2.987063197153e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.669977278077e+05 true resid >>>>>>>>>>>>>> norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.767902454807e+01 true resid >>>>>>>>>>>>>> norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.086718418040e-04 true resid >>>>>>>>>>>>>> norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01 >>>>>>>>>>>>>> 30 SNES Function norm 2.987063193089e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.136061088127e+05 true resid >>>>>>>>>>>>>> norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.186362823355e+02 true resid >>>>>>>>>>>>>> norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.719259676319e-04 true resid >>>>>>>>>>>>>> norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00 >>>>>>>>>>>>>> 31 SNES Function norm 2.987063192061e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.691271219294e+05 true resid >>>>>>>>>>>>>> norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.462251245099e+01 true resid >>>>>>>>>>>>>> norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.183786013104e-04 true resid >>>>>>>>>>>>>> norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01 >>>>>>>>>>>>>> 32 SNES Function norm 2.987063094492e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.124446369819e+05 true resid >>>>>>>>>>>>>> norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.973907556421e+01 true resid >>>>>>>>>>>>>> norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 7.374208523915e-04 true resid >>>>>>>>>>>>>> norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01 >>>>>>>>>>>>>> 33 SNES Function norm 2.987063089942e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.329193031679e+05 true resid >>>>>>>>>>>>>> norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.183766435760e+01 true resid >>>>>>>>>>>>>> norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941412108263e-05 true resid >>>>>>>>>>>>>> norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02 >>>>>>>>>>>>>> 34 SNES Function norm 2.987062660109e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.096972895505e+05 true resid >>>>>>>>>>>>>> norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.733316558393e+01 true resid >>>>>>>>>>>>>> norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.068548659549e-04 true resid >>>>>>>>>>>>>> norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01 >>>>>>>>>>>>>> 35 SNES Function norm 2.987062107900e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 6.499649833972e+04 true resid >>>>>>>>>>>>>> norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.096524448730e+01 true resid >>>>>>>>>>>>>> norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.801709551672e-04 true resid >>>>>>>>>>>>>> norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01 >>>>>>>>>>>>>> 36 SNES Function norm 2.987062055224e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 8.633609234208e+04 true resid >>>>>>>>>>>>>> norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.229972441154e+01 true resid >>>>>>>>>>>>>> norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 5.664496361269e-05 true resid >>>>>>>>>>>>>> norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01 >>>>>>>>>>>>>> 37 SNES Function norm 2.987061774798e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.344743984989e+05 true resid >>>>>>>>>>>>>> norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.372624802517e+01 true resid >>>>>>>>>>>>>> norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.527665232153e-04 true resid >>>>>>>>>>>>>> norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01 >>>>>>>>>>>>>> 38 SNES Function norm 2.987061715400e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.756130938265e+05 true resid >>>>>>>>>>>>>> norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.629319937995e+01 true resid >>>>>>>>>>>>>> norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.099878702234e-04 true resid >>>>>>>>>>>>>> norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01 >>>>>>>>>>>>>> 39 SNES Function norm 2.987061699634e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.326428261777e+05 true resid >>>>>>>>>>>>>> norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.756089168490e+01 true resid >>>>>>>>>>>>>> norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01 >>>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.956105557949e-04 true resid >>>>>>>>>>>>>> norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01 >>>>>>>>>>>>>> 40 SNES Function norm 2.987061630064e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.133259484912e+07 true resid >>>>>>>>>>>>>> norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.296568492777e+01 true resid >>>>>>>>>>>>>> norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00 >>>>>>>>>>>>>> 41 SNES Function norm 2.987061630064e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.124224624453e+07 true resid >>>>>>>>>>>>>> norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.219934148122e+01 true resid >>>>>>>>>>>>>> norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00 >>>>>>>>>>>>>> 42 SNES Function norm 2.987061630064e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.748707463105e+08 true resid >>>>>>>>>>>>>> norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.291843121908e+01 true resid >>>>>>>>>>>>>> norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00 >>>>>>>>>>>>>> 43 SNES Function norm 2.987061630064e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.022718794881e+09 true resid >>>>>>>>>>>>>> norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.314705152175e+01 true resid >>>>>>>>>>>>>> norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00 >>>>>>>>>>>>>> 44 SNES Function norm 2.987061630064e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.380361366230e+08 true resid >>>>>>>>>>>>>> norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.237699578864e+01 true resid >>>>>>>>>>>>>> norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00 >>>>>>>>>>>>>> 45 SNES Function norm 2.987061630064e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 6.303672864381e+08 true resid >>>>>>>>>>>>>> norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.789746869236e+01 true resid >>>>>>>>>>>>>> norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00 >>>>>>>>>>>>>> 46 SNES Function norm 2.987061630064e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.428971983722e+09 true resid >>>>>>>>>>>>>> norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.314776104627e+01 true resid >>>>>>>>>>>>>> norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00 >>>>>>>>>>>>>> 47 SNES Function norm 2.987061630064e+02 >>>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.189185292378e+09 true resid >>>>>>>>>>>>>> norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00 >>>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.191097159135e+01 true resid >>>>>>>>>>>>>> norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00 >>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH >>>>>>>>>>>>>> iterations 47 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -gideon >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <[email protected]> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This indicates that somewhere in your ComputeJacobian you are >>>>>>>>>>>>>>> setting matrix entries with the first Mat argument when you >>>>>>>>>>>>>>> should always set them with the second matrix argument. For >>>>>>>>>>>>>>> example if you have >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at >>>>>>>>>>>>>>> the end of the function you should call MatAssemblyBegin/End() >>>>>>>>>>>>>>> on jpre then on J if J is not == jpre see for example >>>>>>>>>>>>>>> src/snes/examples/tutorials/ex1.c >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a minor glitch we'll get past. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson >>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> By the way, I tried using a different petsc installation, and >>>>>>>>>>>>>>>> now, rather than the segmentation fault, I get the following >>>>>>>>>>>>>>>> error: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object >>>>>>>>>>>>>>>> type >>>>>>>>>>>>>>>> [0]PETSC ERROR: Mat type mffd >>>>>>>>>>>>>>>> [0]PETSC ERROR: See >>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for >>>>>>>>>>>>>>>> trouble shooting. >>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 >>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug >>>>>>>>>>>>>>>> named gs_air by gideon Mon Sep 7 21:32:18 2015 >>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes >>>>>>>>>>>>>>>> --download-suitesparse=yes --download-superlu=yes >>>>>>>>>>>>>>>> --download-superlu_dist=yes --download-mumps=yes >>>>>>>>>>>>>>>> --download-sprng=yes --with-cxx=clang++ --with-cc=clang >>>>>>>>>>>>>>>> --with-fc=gfortran --download-metis=yes >>>>>>>>>>>>>>>> --download-parmetis=yes --download-scalapack=yes >>>>>>>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in >>>>>>>>>>>>>>>> /opt/petsc-3.5.4/src/mat/interface/matrix.c >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -gideon >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <[email protected]> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hmm, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Ok you can try running it directly in the debugger since it >>>>>>>>>>>>>>>>> is one process, type >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> gdb ./blowup_batch_refine >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> then >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> when the debugger comes up (if it does not cut and paste all >>>>>>>>>>>>>>>>> output and send it) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> run -on_error_abort -snes_mf_operator and any other options >>>>>>>>>>>>>>>>> you normally use >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson >>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Running with that flag gives me this: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to >>>>>>>>>>>>>>>>>> ./blowup_batch_refine of pid 16111 on gs_air >>>>>>>>>>>>>>>>>> Unable to start debugger: No such file or directory >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -gideon >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith >>>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> This should not happen. Run with a debug version of PETSc >>>>>>>>>>>>>>>>>>> installed and the option -start_in_debugger noxterm Once >>>>>>>>>>>>>>>>>>> the debugger starts up type cont and when it crashes type >>>>>>>>>>>>>>>>>>> where or bt Send all output >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson >>>>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I’m getting an error with -snes_mf_operator, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02 >>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: >>>>>>>>>>>>>>>>>>>> ------------------------------------------------------------------------ >>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation >>>>>>>>>>>>>>>>>>>> Violation, probably memory access out of range >>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or >>>>>>>>>>>>>>>>>>>> -on_error_attach_debugger >>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or see >>>>>>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux >>>>>>>>>>>>>>>>>>>> and Apple Mac OS X to find memory corruption errors >>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, >>>>>>>>>>>>>>>>>>>> recompile, link, and run >>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash. >>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received >>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: See >>>>>>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for >>>>>>>>>>>>>>>>>>>> trouble shooting. >>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown >>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports >>>>>>>>>>>>>>>>>>>> named gs_air by gideon Mon Sep 7 21:08:19 2015 >>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local >>>>>>>>>>>>>>>>>>>> --prefix=/opt/local/lib/petsc --with-valgrind=0 >>>>>>>>>>>>>>>>>>>> --with-shared-libraries --with-debugging=0 >>>>>>>>>>>>>>>>>>>> --with-c2html-dir=/opt/local --with-x=0 >>>>>>>>>>>>>>>>>>>> --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate >>>>>>>>>>>>>>>>>>>> --with-hwloc-dir=/opt/local >>>>>>>>>>>>>>>>>>>> --with-suitesparse-dir=/opt/local >>>>>>>>>>>>>>>>>>>> --with-superlu-dir=/opt/local --with-metis-dir=/opt/local >>>>>>>>>>>>>>>>>>>> --with-parmetis-dir=/opt/local >>>>>>>>>>>>>>>>>>>> --with-scalapack-dir=/opt/local >>>>>>>>>>>>>>>>>>>> --with-mumps-dir=/opt/local >>>>>>>>>>>>>>>>>>>> --with-superlu_dist-dir=/opt/local >>>>>>>>>>>>>>>>>>>> CC=/opt/local/bin/mpicc-mpich-mp >>>>>>>>>>>>>>>>>>>> CXX=/opt/local/bin/mpicxx-mpich-mp >>>>>>>>>>>>>>>>>>>> FC=/opt/local/bin/mpif90-mpich-mp >>>>>>>>>>>>>>>>>>>> F77=/opt/local/bin/mpif90-mpich-mp >>>>>>>>>>>>>>>>>>>> F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os >>>>>>>>>>>>>>>>>>>> CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib >>>>>>>>>>>>>>>>>>>> -Wl,-headerpad_max_install_names" >>>>>>>>>>>>>>>>>>>> CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" >>>>>>>>>>>>>>>>>>>> CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os >>>>>>>>>>>>>>>>>>>> PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp >>>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in >>>>>>>>>>>>>>>>>>>> unknown file >>>>>>>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process >>>>>>>>>>>>>>>>>>>> 0 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> -gideon >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith >>>>>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct >>>>>>>>>>>>>>>>>>>>> "enough"), hence PETSc SNES is generating a poor descent >>>>>>>>>>>>>>>>>>>>> direction. You can try >>>>>>>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as >>>>>>>>>>>>>>>>>>>>> additional arguments. What happens? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson >>>>>>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> No problem Matt, I don’t think we had previously >>>>>>>>>>>>>>>>>>>>>> discussed that output. Here is a case where things fail. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09 >>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12 >>>>>>>>>>>>>>>>>>>>>> Nonlinear solve converged due to >>>>>>>>>>>>>>>>>>>>>> CONVERGED_SNORM_RELATIVE iterations 1 >>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03 >>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02 >>>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02 >>>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02 >>>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02 >>>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02 >>>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02 >>>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02 >>>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02 >>>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02 >>>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02 >>>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02 >>>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02 >>>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02 >>>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02 >>>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02 >>>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02 >>>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02 >>>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02 >>>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02 >>>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02 >>>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02 >>>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02 >>>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02 >>>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02 >>>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02 >>>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02 >>>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02 >>>>>>>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02 >>>>>>>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02 >>>>>>>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02 >>>>>>>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02 >>>>>>>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02 >>>>>>>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02 >>>>>>>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02 >>>>>>>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02 >>>>>>>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02 >>>>>>>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02 >>>>>>>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02 >>>>>>>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02 >>>>>>>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02 >>>>>>>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02 >>>>>>>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02 >>>>>>>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02 >>>>>>>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02 >>>>>>>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02 >>>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to >>>>>>>>>>>>>>>>>>>>>> DIVERGED_LINE_SEARCH iterations 45 >>>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03 >>>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03 >>>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03 >>>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03 >>>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03 >>>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03 >>>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03 >>>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03 >>>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03 >>>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03 >>>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03 >>>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03 >>>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03 >>>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03 >>>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03 >>>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03 >>>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03 >>>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03 >>>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03 >>>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03 >>>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03 >>>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03 >>>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03 >>>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03 >>>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03 >>>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03 >>>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03 >>>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03 >>>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to >>>>>>>>>>>>>>>>>>>>>> DIVERGED_LINE_SEARCH iterations 27 >>>>>>>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes >>>>>>>>>>>>>>>>>>>>>> type: newtonls >>>>>>>>>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000 >>>>>>>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, >>>>>>>>>>>>>>>>>>>>>> solution=1e-08 >>>>>>>>>>>>>>>>>>>>>> total number of linear solver iterations=28 >>>>>>>>>>>>>>>>>>>>>> total number of function evaluations=323 >>>>>>>>>>>>>>>>>>>>>> total number of grid sequence refinements=2 >>>>>>>>>>>>>>>>>>>>>> SNESLineSearch Object: 1 MPI processes >>>>>>>>>>>>>>>>>>>>>> type: bt >>>>>>>>>>>>>>>>>>>>>> interpolation: cubic >>>>>>>>>>>>>>>>>>>>>> alpha=1.000000e-04 >>>>>>>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12 >>>>>>>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, >>>>>>>>>>>>>>>>>>>>>> absolute=1.000000e-15, lambda=1.000000e-08 >>>>>>>>>>>>>>>>>>>>>> maximum iterations=40 >>>>>>>>>>>>>>>>>>>>>> KSP Object: 1 MPI processes >>>>>>>>>>>>>>>>>>>>>> type: gmres >>>>>>>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) >>>>>>>>>>>>>>>>>>>>>> Gram-Schmidt Orthogonalization with no iterative >>>>>>>>>>>>>>>>>>>>>> refinement >>>>>>>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30 >>>>>>>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero >>>>>>>>>>>>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50, >>>>>>>>>>>>>>>>>>>>>> divergence=10000 >>>>>>>>>>>>>>>>>>>>>> left preconditioning >>>>>>>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test >>>>>>>>>>>>>>>>>>>>>> PC Object: 1 MPI processes >>>>>>>>>>>>>>>>>>>>>> type: lu >>>>>>>>>>>>>>>>>>>>>> LU: out-of-place factorization >>>>>>>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14 >>>>>>>>>>>>>>>>>>>>>> matrix ordering: nd >>>>>>>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0 >>>>>>>>>>>>>>>>>>>>>> Factored matrix follows: >>>>>>>>>>>>>>>>>>>>>> Mat Object: 1 MPI processes >>>>>>>>>>>>>>>>>>>>>> type: seqaij >>>>>>>>>>>>>>>>>>>>>> rows=15991, cols=15991 >>>>>>>>>>>>>>>>>>>>>> package used to perform factorization: mumps >>>>>>>>>>>>>>>>>>>>>> total: nonzeros=255801, allocated nonzeros=255801 >>>>>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls >>>>>>>>>>>>>>>>>>>>>> =0 >>>>>>>>>>>>>>>>>>>>>> MUMPS run parameters: >>>>>>>>>>>>>>>>>>>>>> SYM (matrix type): 0 >>>>>>>>>>>>>>>>>>>>>> PAR (host participation): 1 >>>>>>>>>>>>>>>>>>>>>> ICNTL(1) (output for error): 6 >>>>>>>>>>>>>>>>>>>>>> ICNTL(2) (output of diagnostic msg): 0 >>>>>>>>>>>>>>>>>>>>>> ICNTL(3) (output for global info): 0 >>>>>>>>>>>>>>>>>>>>>> ICNTL(4) (level of printing): 0 >>>>>>>>>>>>>>>>>>>>>> ICNTL(5) (input mat struct): 0 >>>>>>>>>>>>>>>>>>>>>> ICNTL(6) (matrix prescaling): 7 >>>>>>>>>>>>>>>>>>>>>> ICNTL(7) (sequentia matrix ordering):6 >>>>>>>>>>>>>>>>>>>>>> ICNTL(8) (scalling strategy): 77 >>>>>>>>>>>>>>>>>>>>>> ICNTL(10) (max num of refinements): 0 >>>>>>>>>>>>>>>>>>>>>> ICNTL(11) (error analysis): 0 >>>>>>>>>>>>>>>>>>>>>> ICNTL(12) (efficiency control): >>>>>>>>>>>>>>>>>>>>>> 1 >>>>>>>>>>>>>>>>>>>>>> ICNTL(13) (efficiency control): >>>>>>>>>>>>>>>>>>>>>> 0 >>>>>>>>>>>>>>>>>>>>>> ICNTL(14) (percentage of estimated workspace >>>>>>>>>>>>>>>>>>>>>> increase): 20 >>>>>>>>>>>>>>>>>>>>>> ICNTL(18) (input mat struct): >>>>>>>>>>>>>>>>>>>>>> 0 >>>>>>>>>>>>>>>>>>>>>> ICNTL(19) (Shur complement info): >>>>>>>>>>>>>>>>>>>>>> 0 >>>>>>>>>>>>>>>>>>>>>> ICNTL(20) (rhs sparse pattern): >>>>>>>>>>>>>>>>>>>>>> 0 >>>>>>>>>>>>>>>>>>>>>> ICNTL(21) (somumpstion struct): >>>>>>>>>>>>>>>>>>>>>> 0 >>>>>>>>>>>>>>>>>>>>>> ICNTL(22) (in-core/out-of-core facility): >>>>>>>>>>>>>>>>>>>>>> 0 >>>>>>>>>>>>>>>>>>>>>> ICNTL(23) (max size of memory can be allocated >>>>>>>>>>>>>>>>>>>>>> locally):0 >>>>>>>>>>>>>>>>>>>>>> ICNTL(24) (detection of null pivot rows): >>>>>>>>>>>>>>>>>>>>>> 0 >>>>>>>>>>>>>>>>>>>>>> ICNTL(25) (computation of a null space basis): >>>>>>>>>>>>>>>>>>>>>> 0 >>>>>>>>>>>>>>>>>>>>>> ICNTL(26) (Schur options for rhs or solution): >>>>>>>>>>>>>>>>>>>>>> 0 >>>>>>>>>>>>>>>>>>>>>> ICNTL(27) (experimental parameter): >>>>>>>>>>>>>>>>>>>>>> -8 >>>>>>>>>>>>>>>>>>>>>> ICNTL(28) (use parallel or sequential ordering): >>>>>>>>>>>>>>>>>>>>>> 1 >>>>>>>>>>>>>>>>>>>>>> ICNTL(29) (parallel ordering): >>>>>>>>>>>>>>>>>>>>>> 0 >>>>>>>>>>>>>>>>>>>>>> ICNTL(30) (user-specified set of entries in >>>>>>>>>>>>>>>>>>>>>> inv(A)): 0 >>>>>>>>>>>>>>>>>>>>>> ICNTL(31) (factors is discarded in the solve >>>>>>>>>>>>>>>>>>>>>> phase): 0 >>>>>>>>>>>>>>>>>>>>>> ICNTL(33) (compute determinant): >>>>>>>>>>>>>>>>>>>>>> 0 >>>>>>>>>>>>>>>>>>>>>> CNTL(1) (relative pivoting threshold): 0.01 >>>>>>>>>>>>>>>>>>>>>> CNTL(2) (stopping criterion of refinement): >>>>>>>>>>>>>>>>>>>>>> 1.49012e-08 >>>>>>>>>>>>>>>>>>>>>> CNTL(3) (absomumpste pivoting threshold): 0 >>>>>>>>>>>>>>>>>>>>>> CNTL(4) (vamumpse of static pivoting): -1 >>>>>>>>>>>>>>>>>>>>>> CNTL(5) (fixation for null pivots): 0 >>>>>>>>>>>>>>>>>>>>>> RINFO(1) (local estimated flops for the elimination >>>>>>>>>>>>>>>>>>>>>> after analysis): >>>>>>>>>>>>>>>>>>>>>> [0] 1.95838e+06 >>>>>>>>>>>>>>>>>>>>>> RINFO(2) (local estimated flops for the assembly >>>>>>>>>>>>>>>>>>>>>> after factorization): >>>>>>>>>>>>>>>>>>>>>> [0] 143924 >>>>>>>>>>>>>>>>>>>>>> RINFO(3) (local estimated flops for the elimination >>>>>>>>>>>>>>>>>>>>>> after factorization): >>>>>>>>>>>>>>>>>>>>>> [0] 1.95943e+06 >>>>>>>>>>>>>>>>>>>>>> INFO(15) (estimated size of (in MB) MUMPS internal >>>>>>>>>>>>>>>>>>>>>> data for running numerical factorization): >>>>>>>>>>>>>>>>>>>>>> [0] 7 >>>>>>>>>>>>>>>>>>>>>> INFO(16) (size of (in MB) MUMPS internal data used >>>>>>>>>>>>>>>>>>>>>> during numerical factorization): >>>>>>>>>>>>>>>>>>>>>> [0] 7 >>>>>>>>>>>>>>>>>>>>>> INFO(23) (num of pivots eliminated on this >>>>>>>>>>>>>>>>>>>>>> processor after factorization): >>>>>>>>>>>>>>>>>>>>>> [0] 15991 >>>>>>>>>>>>>>>>>>>>>> RINFOG(1) (global estimated flops for the >>>>>>>>>>>>>>>>>>>>>> elimination after analysis): 1.95838e+06 >>>>>>>>>>>>>>>>>>>>>> RINFOG(2) (global estimated flops for the assembly >>>>>>>>>>>>>>>>>>>>>> after factorization): 143924 >>>>>>>>>>>>>>>>>>>>>> RINFOG(3) (global estimated flops for the >>>>>>>>>>>>>>>>>>>>>> elimination after factorization): 1.95943e+06 >>>>>>>>>>>>>>>>>>>>>> (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): >>>>>>>>>>>>>>>>>>>>>> (0,0)*(2^0) >>>>>>>>>>>>>>>>>>>>>> INFOG(3) (estimated real workspace for factors on >>>>>>>>>>>>>>>>>>>>>> all processors after analysis): 255801 >>>>>>>>>>>>>>>>>>>>>> INFOG(4) (estimated integer workspace for factors >>>>>>>>>>>>>>>>>>>>>> on all processors after analysis): 127874 >>>>>>>>>>>>>>>>>>>>>> INFOG(5) (estimated maximum front size in the >>>>>>>>>>>>>>>>>>>>>> complete tree): 11 >>>>>>>>>>>>>>>>>>>>>> INFOG(6) (number of nodes in the complete tree): >>>>>>>>>>>>>>>>>>>>>> 3996 >>>>>>>>>>>>>>>>>>>>>> INFOG(7) (ordering option effectively use after >>>>>>>>>>>>>>>>>>>>>> analysis): 6 >>>>>>>>>>>>>>>>>>>>>> INFOG(8) (structural symmetry in percent of the >>>>>>>>>>>>>>>>>>>>>> permuted matrix after analysis): 86 >>>>>>>>>>>>>>>>>>>>>> INFOG(9) (total real/complex workspace to store the >>>>>>>>>>>>>>>>>>>>>> matrix factors after factorization): 255865 >>>>>>>>>>>>>>>>>>>>>> INFOG(10) (total integer space store the matrix >>>>>>>>>>>>>>>>>>>>>> factors after factorization): 127890 >>>>>>>>>>>>>>>>>>>>>> INFOG(11) (order of largest frontal matrix after >>>>>>>>>>>>>>>>>>>>>> factorization): 11 >>>>>>>>>>>>>>>>>>>>>> INFOG(12) (number of off-diagonal pivots): 19 >>>>>>>>>>>>>>>>>>>>>> INFOG(13) (number of delayed pivots after >>>>>>>>>>>>>>>>>>>>>> factorization): 8 >>>>>>>>>>>>>>>>>>>>>> INFOG(14) (number of memory compress after >>>>>>>>>>>>>>>>>>>>>> factorization): 0 >>>>>>>>>>>>>>>>>>>>>> INFOG(15) (number of steps of iterative refinement >>>>>>>>>>>>>>>>>>>>>> after solution): 0 >>>>>>>>>>>>>>>>>>>>>> INFOG(16) (estimated size (in MB) of all MUMPS >>>>>>>>>>>>>>>>>>>>>> internal data for factorization after analysis: value on >>>>>>>>>>>>>>>>>>>>>> the most memory consuming processor): 7 >>>>>>>>>>>>>>>>>>>>>> INFOG(17) (estimated size of all MUMPS internal >>>>>>>>>>>>>>>>>>>>>> data for factorization after analysis: sum over all >>>>>>>>>>>>>>>>>>>>>> processors): 7 >>>>>>>>>>>>>>>>>>>>>> INFOG(18) (size of all MUMPS internal data >>>>>>>>>>>>>>>>>>>>>> allocated during factorization: value on the most memory >>>>>>>>>>>>>>>>>>>>>> consuming processor): 7 >>>>>>>>>>>>>>>>>>>>>> INFOG(19) (size of all MUMPS internal data >>>>>>>>>>>>>>>>>>>>>> allocated during factorization: sum over all >>>>>>>>>>>>>>>>>>>>>> processors): 7 >>>>>>>>>>>>>>>>>>>>>> INFOG(20) (estimated number of entries in the >>>>>>>>>>>>>>>>>>>>>> factors): 255801 >>>>>>>>>>>>>>>>>>>>>> INFOG(21) (size in MB of memory effectively used >>>>>>>>>>>>>>>>>>>>>> during factorization - value on the most memory >>>>>>>>>>>>>>>>>>>>>> consuming processor): 7 >>>>>>>>>>>>>>>>>>>>>> INFOG(22) (size in MB of memory effectively used >>>>>>>>>>>>>>>>>>>>>> during factorization - sum over all processors): 7 >>>>>>>>>>>>>>>>>>>>>> INFOG(23) (after analysis: value of ICNTL(6) >>>>>>>>>>>>>>>>>>>>>> effectively used): 0 >>>>>>>>>>>>>>>>>>>>>> INFOG(24) (after analysis: value of ICNTL(12) >>>>>>>>>>>>>>>>>>>>>> effectively used): 1 >>>>>>>>>>>>>>>>>>>>>> INFOG(25) (after factorization: number of pivots >>>>>>>>>>>>>>>>>>>>>> modified by static pivoting): 0 >>>>>>>>>>>>>>>>>>>>>> INFOG(28) (after factorization: number of null >>>>>>>>>>>>>>>>>>>>>> pivots encountered): 0 >>>>>>>>>>>>>>>>>>>>>> INFOG(29) (after factorization: effective number of >>>>>>>>>>>>>>>>>>>>>> entries in the factors (sum over all processors)): 255865 >>>>>>>>>>>>>>>>>>>>>> INFOG(30, 31) (after solution: size in Mbytes of >>>>>>>>>>>>>>>>>>>>>> memory used during solution phase): 5, 5 >>>>>>>>>>>>>>>>>>>>>> INFOG(32) (after analysis: type of analysis done): 1 >>>>>>>>>>>>>>>>>>>>>> INFOG(33) (value used for ICNTL(8)): 7 >>>>>>>>>>>>>>>>>>>>>> INFOG(34) (exponent of the determinant if >>>>>>>>>>>>>>>>>>>>>> determinant is requested): 0 >>>>>>>>>>>>>>>>>>>>>> linear system matrix = precond matrix: >>>>>>>>>>>>>>>>>>>>>> Mat Object: 1 MPI processes >>>>>>>>>>>>>>>>>>>>>> type: seqaij >>>>>>>>>>>>>>>>>>>>>> rows=15991, cols=15991 >>>>>>>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698 >>>>>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls >>>>>>>>>>>>>>>>>>>>>> =15991 >>>>>>>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5 >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> -gideon >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley >>>>>>>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson >>>>>>>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>> Barry, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid >>>>>>>>>>>>>>>>>>>>>>> sequencing within my code. I find that, in some cases, >>>>>>>>>>>>>>>>>>>>>>> even if it can solve successfully on the coarsest mesh, >>>>>>>>>>>>>>>>>>>>>>> the SNES fails, usually due to a line search failure, >>>>>>>>>>>>>>>>>>>>>>> when it tries to compute along the grid sequence. >>>>>>>>>>>>>>>>>>>>>>> Would you have any suggestions? >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me >>>>>>>>>>>>>>>>>>>>>>> -snes_view for the solver? I could not find it in the >>>>>>>>>>>>>>>>>>>>>>> email thread. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, >>>>>>>>>>>>>>>>>>>>>>> or precondition it with Richardson. It would be nice to >>>>>>>>>>>>>>>>>>>>>>> see -snes_monitor >>>>>>>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the >>>>>>>>>>>>>>>>>>>>>>> residual into fields and look at it again (if my custom >>>>>>>>>>>>>>>>>>>>>>> residual monitor >>>>>>>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which >>>>>>>>>>>>>>>>>>>>>>> part of the residual does not converge is key to >>>>>>>>>>>>>>>>>>>>>>> designing the NASM >>>>>>>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, >>>>>>>>>>>>>>>>>>>>>>> Xiao-Chuan Cai, present it. We need better monitoring >>>>>>>>>>>>>>>>>>>>>>> in PETSc. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Matt >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> -gideon >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith >>>>>>>>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson >>>>>>>>>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse >>>>>>>>>>>>>>>>>>>>>>>>> mesh, I can generally solve at all values. I do find >>>>>>>>>>>>>>>>>>>>>>>>> that I need to do some amount of continuation to >>>>>>>>>>>>>>>>>>>>>>>>> solve near the endpoint. The problem is that on the >>>>>>>>>>>>>>>>>>>>>>>>> coarse mesh, things are not fully resolved at all the >>>>>>>>>>>>>>>>>>>>>>>>> values along the continuation parameter, and I would >>>>>>>>>>>>>>>>>>>>>>>>> like to do refinement. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate >>>>>>>>>>>>>>>>>>>>>>>>> continuation solutions too. Currently, without >>>>>>>>>>>>>>>>>>>>>>>>> doing any grid sequence, I compute each, write it to >>>>>>>>>>>>>>>>>>>>>>>>> disk, and then go on to the next one. So I now need >>>>>>>>>>>>>>>>>>>>>>>>> to go back an refine them. I was thinking that >>>>>>>>>>>>>>>>>>>>>>>>> perhaps I could refine them on the fly, dump them to >>>>>>>>>>>>>>>>>>>>>>>>> disk, and use the coarse solution as the starting >>>>>>>>>>>>>>>>>>>>>>>>> guess at the next iteration, but that would seem to >>>>>>>>>>>>>>>>>>>>>>>>> require resetting the snes back to the coarse grid. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh >>>>>>>>>>>>>>>>>>>>>>>>> refinement in a post processing stage, where each >>>>>>>>>>>>>>>>>>>>>>>>> value of the continuation is parameter is loaded on >>>>>>>>>>>>>>>>>>>>>>>>> the coarse mesh, and refined. Perhaps that’s the >>>>>>>>>>>>>>>>>>>>>>>>> most practical thing to do. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a >>>>>>>>>>>>>>>>>>>>>>>> SNES that will do the continuation >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> loop over continuation parameter >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse); >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at >>>>>>>>>>>>>>>>>>>>>>>> this continuation point) { >>>>>>>>>>>>>>>>>>>>>>>> SNESCreate(comm,&snesrefine); >>>>>>>>>>>>>>>>>>>>>>>> SNESSetDM() >>>>>>>>>>>>>>>>>>>>>>>> etc >>>>>>>>>>>>>>>>>>>>>>>> SNESSetGridSequence(snesrefine,) >>>>>>>>>>>>>>>>>>>>>>>> SNESSolve(snesrefine,0,Ucoarse); >>>>>>>>>>>>>>>>>>>>>>>> SNESGetSolution(snesrefine,&Ufine); >>>>>>>>>>>>>>>>>>>>>>>> VecView(Ufine or do whatever you want to do with the >>>>>>>>>>>>>>>>>>>>>>>> Ufine at that continuation point >>>>>>>>>>>>>>>>>>>>>>>> SNESDestroy(snesrefine); >>>>>>>>>>>>>>>>>>>>>>>> end if >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> end loop over continuation parameter. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> -gideon >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith >>>>>>>>>>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> 3. This problem is actually part of a continuation >>>>>>>>>>>>>>>>>>>>>>>>>>> problem that roughly looks like this >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){ >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> solve with parameter p_i using solution from >>>>>>>>>>>>>>>>>>>>>>>>>>> p_{i-1}, >>>>>>>>>>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for >>>>>>>>>>>>>>>>>>>>>>>>>>> each value of parameter p_i on the coarse mesh, and >>>>>>>>>>>>>>>>>>>>>>>>>>> then do grid sequencing on that. But it appears >>>>>>>>>>>>>>>>>>>>>>>>>>> that after doing grid sequencing on the initial p_0 >>>>>>>>>>>>>>>>>>>>>>>>>>> = 0, the SNES is set to use the finer mesh. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good >>>>>>>>>>>>>>>>>>>>>>>>>> enough initial guess on the coarse level to even get >>>>>>>>>>>>>>>>>>>>>>>>>> convergence on the coarse level? First I would check >>>>>>>>>>>>>>>>>>>>>>>>>> if you even need the continuation (or can you not >>>>>>>>>>>>>>>>>>>>>>>>>> even solve the coarse problem without it). >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need >>>>>>>>>>>>>>>>>>>>>>>>>> to tweak how you do the grid sequencing. I think >>>>>>>>>>>>>>>>>>>>>>>>>> this will work: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your >>>>>>>>>>>>>>>>>>>>>>>>>> continuation parameter. This will all happen on the >>>>>>>>>>>>>>>>>>>>>>>>>> coarse mesh. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence() >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve >>>>>>>>>>>>>>>>>>>>>>>>>> on the coarse level and then interpolate to the next >>>>>>>>>>>>>>>>>>>>>>>>>> level etc. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>> What most experimenters take for granted before they >>>>>>>>>>>>>>>>>>>>>>> begin their experiments is infinitely more interesting >>>>>>>>>>>>>>>>>>>>>>> than any results to which their experiments lead. >>>>>>>>>>>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which >>>>>>>> their experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>> >>>>>> >>>>> >>>> >>> >> >
