Cool.
> On Dec 12, 2017, at 3:56 PM, Alexander Lindsay <[email protected]> > wrote: > > So I decided to look at the condition number of our matrix, running with > `-pc_type svd -pc_svd_monitor` and it was atrocious, roughly on the order of > 1e9. After doing some scaling we are down to a condition number of 1e3, and > both MF and FD operators now converge, regardless of the differencing types > chosen. I would say the problem was definitely on our end! > > On Tue, Dec 12, 2017 at 2:49 PM, Matthew Knepley <[email protected]> wrote: > On Tue, Dec 12, 2017 at 3:19 PM, Alexander Lindsay <[email protected]> > wrote: > I'm helping debug the finite strain models in the TensorMechanics module in > MOOSE, so unfortunately I don't have a nice small PetSc code I can hand you > guys :-( > > Hmm, interesting, if I run with `-snes_mf_operator -snes_fd -mat_mffd_type > ds`, I get DIVERGED_BREAKDOWN during the initial linear solve. > > So the MF operator always converges. The FD operator does not always > converge, and factorization also can fail (DIVERGED_BREAKDOWN) > so it seems that the FD operator is incorrect. Usually we have bugs with > coloring, but I do not think coloring is used by -snes_fd. What happens > if you get the coloring version by just deleting the FormJacobian pointer? > > Thanks, > > Matt > > If I run with `-snes_fd -mat_fd_type ds`, then the solve converges. > > So summary: > > - J = B = finite-differenced, differencing type = wp : Solve fails due to > DIVERGED_LINE_SEARCH > > - J = B = finite-differenced, differencing type = ds : Solve converges in 3 > non-linear iterations > 0 Nonlinear |R| = 2.259203e-02 > 0 Linear |R| = 2.259203e-02 > 1 Linear |R| = 6.084393e-11 > 1 Nonlinear |R| = 4.780691e-03 > 0 Linear |R| = 4.780691e-03 > 1 Linear |R| = 8.580132e-19 > 2 Nonlinear |R| = 4.806625e-09 > 0 Linear |R| = 4.806625e-09 > 1 Linear |R| = 1.650725e-24 > 3 Nonlinear |R| = 9.603678e-12 > > - J = matrix-free, B = finite-differenced, mat_mffd_type = mat_fd_type = wp: > Solve converges in 2 non-linear iterations > 0 Nonlinear |R| = 2.259203e-02 > 0 Linear |R| = 2.259203e-02 > 1 Linear |R| = 2.258733e-02 > 2 Linear |R| = 3.103342e-06 > 3 Linear |R| = 6.779865e-12 > 1 Nonlinear |R| = 7.497740e-06 > 0 Linear |R| = 7.497740e-06 > 1 Linear |R| = 8.265413e-12 > 2 Nonlinear |R| = 7.993729e-12 > > - J = matrix-free, B = finite-differenced, mat_mffd_type = ds, mat_fd_type = > wp: DIVERGED_BREAKDOWN in linear solve > > - J = matrix-free, B = finite-differenced, mat_mffd_type = wp, mat_fd_type = > ds: Solve converges in 2 non-linear iterations > 0 Nonlinear |R| = 2.259203e-02 > 0 Linear |R| = 2.259203e-02 > 1 Linear |R| = 4.635397e-03 > 2 Linear |R| = 5.413676e-11 > 1 Nonlinear |R| = 1.068626e-05 > 0 Linear |R| = 1.068626e-05 > 1 Linear |R| = 7.942385e-12 > 2 Nonlinear |R| = 5.444448e-11 > > - J = matrix-free, B = finite-differenced, mat_mffd_type = mat_fd_type = ds: > Solves converges in 3 non-linear iterations: > 0 Nonlinear |R| = 2.259203e-02 > 0 Linear |R| = 2.259203e-02 > 1 Linear |R| = 1.312921e-06 > 2 Linear |R| = 7.714018e-09 > 1 Nonlinear |R| = 4.780690e-03 > 0 Linear |R| = 4.780690e-03 > 1 Linear |R| = 7.773053e-09 > 2 Nonlinear |R| = 1.226836e-08 > 0 Linear |R| = 1.226836e-08 > 1 Linear |R| = 1.546288e-14 > 3 Nonlinear |R| = 1.295982e-10 > > > > > On Tue, Dec 12, 2017 at 12:33 PM, Smith, Barry F. <[email protected]> wrote: > > > > On Dec 12, 2017, at 11:26 AM, Alexander Lindsay <[email protected]> > > wrote: > > > > Ok, I'm going to go back on my original statement...the physics being run > > here is a sub-set of a much larger set of physics; for the current set the > > hand-coded Jacobian actually appears to be quite good. > > > > With hand-coded Jacobian, -pc_type lu, the convergence is perfect: > > > > 0 Nonlinear |R| = 2.259203e-02 > > 0 Linear |R| = 2.259203e-02 > > 1 Linear |R| = 1.129089e-10 > > 1 Nonlinear |R| = 6.295583e-11 > > > > So yea I guess at this point I'm just curious about the different behavior > > between `-snes_fd` and `-snes_fd -snes_mf_operator`. > > Now that you have provided the exact options you are using, yes it is very > unexpected behavior. Is there any chance you can send us the code that > reproduces this? > > The code that does the differencing in -snes_fd is similar to the code > that does the differencing for -snes_mf_operator so normally one expects > similar behavior but there are a couple of options you can try. Run with > -snes_mf_operator and -help | grep mat_mffd and this will show options to > control the differencing for the matrix free. For -snes_fd you have the > option -mat_fd_type wp or ds > > > > Does the hand-coded result change your opinion Matt that the rules for > > FormFunction/Jacobian might be being violated? > > > > I understand that a finite difference approximation of the true Jacobian is > > an approximation. However, in the absence of possible complications like > > Matt suggested where an on-the-fly calculation might stand a better chance > > of capturing the behavior, I would expect both `-snes_mf_operator -snes_fd` > > and `-snes_fd` to suffer from the same approximations, right? > > > > On Tue, Dec 12, 2017 at 9:43 AM, Matthew Knepley <[email protected]> wrote: > > On Tue, Dec 12, 2017 at 11:30 AM, Alexander Lindsay > > <[email protected]> wrote: > > I'm not using any hand-coded Jacobians. > > > > This looks to me like the rules for FormFunction/Jacobian() are being > > broken. If the residual function > > depends on some third variable, and it changes between calls independent of > > the solution U, then > > the stored Jacobian could look wrong, but one done every time on the fly > > might converge. > > > > Matt > > > > Case 1 options: -snes_fd -pc_type lu > > > > 0 Nonlinear |R| = 2.259203e-02 > > 0 Linear |R| = 2.259203e-02 > > 1 Linear |R| = 7.821248e-11 > > 1 Nonlinear |R| = 2.258733e-02 > > 0 Linear |R| = 2.258733e-02 > > 1 Linear |R| = 5.277296e-11 > > 2 Nonlinear |R| = 2.258733e-02 > > 0 Linear |R| = 2.258733e-02 > > 1 Linear |R| = 5.993971e-11 > > Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 2 > > > > Case 2 options: -snes_fd -snes_mf_operator -pc_type lu > > > > 0 Nonlinear |R| = 2.259203e-02 > > 0 Linear |R| = 2.259203e-02 > > 1 Linear |R| = 2.258733e-02 > > 2 Linear |R| = 3.103342e-06 > > 3 Linear |R| = 6.779865e-12 > > 1 Nonlinear |R| = 7.497740e-06 > > 0 Linear |R| = 7.497740e-06 > > 1 Linear |R| = 8.265413e-12 > > 2 Nonlinear |R| = 7.993729e-12 > > Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 2 > > > > > > On Tue, Dec 12, 2017 at 9:12 AM, zakaryah . <[email protected]> wrote: > > When you say "Jacobians are bad" and "debugging the Jacobians", do you mean > > that the hand-coded Jacobian is wrong? In that case, why would you be > > surprised that the finite difference Jacobians, which are "correct" to > > approximation error, perform better? Otherwise, what does "Jacobians are > > bad" mean - ill-conditioned? Singular? Not symmetric? Not positive > > definite? > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which their > > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ >
