You may benefit from a literature search on your model AND preconditioners to see what others have used. But I would try PETSc/MUMPS on the biggest size you want and see how it goes (better it runs for a little longer and you don't waste months trying to find a good preconditioner).
> On Mar 28, 2024, at 2:20 PM, Zou, Ling <l...@anl.gov> wrote: > > Thank you, Barry. > Yes, I have tried different preconditioners, but in a naïve way, i.e., > looping through possible options using `-pc_type <option>` command line. > But no, not in a meaningful way because the lack of understanding of the > connection between physics (the problem we are dealing with) to math (the > correct combination of those preconditioners). > > -Ling > > From: Barry Smith <bsm...@petsc.dev <mailto:bsm...@petsc.dev>> > Date: Thursday, March 28, 2024 at 1:09 PM > To: Zou, Ling <l...@anl.gov <mailto:l...@anl.gov>> > Cc: Zhang, Hong <hzh...@mcs.anl.gov <mailto:hzh...@mcs.anl.gov>>, > petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov> > <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> > Subject: Re: [petsc-users] Does ILU(15) still make sense or should just use > LU? > > 1 million is possible for direct solvers using PETSc with the MUMPS direct > solver when you cannot get a preconditioner to work well for your problems. > ILU are not very robust preconditioners and I would not rely on them. Have > you investigated > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > 1 million is possible for direct solvers using PETSc with the MUMPS direct > solver when you cannot get a preconditioner to work well for your problems. > > ILU are not very robust preconditioners and I would not rely on them. > Have you investigated other preconditioners in PETSc, PCGAMG, PCASM, > PCFIELDSPLIT or some combination of these preconditioners work for many > problems, though certainly not all. > > > > On Mar 28, 2024, at 1:14 PM, Zou, Ling <l...@anl.gov <mailto:l...@anl.gov>> > wrote: > > Thank you, Barry. > Yeah, this is unfortunate given that the problem we are handling is quite > heterogeneous (in both mesh and physics). > I expect that our problem sizes will be mostly smaller than 1 million DOF, > should LU still be a practical solution? Can it scale well if we choose to > run the problem in a parallel way? > > PS1: -ksp_norm_type unpreconditioned did not work as the true residual did > not go down, even with 300 linear iterations. > PS2: what do you think if it will be beneficial to have more detailed > discussions (e.g., a presentation?) on the problem we are solving to seek > more advice? > > -Ling > > From: Barry Smith <bsm...@petsc.dev <mailto:bsm...@petsc.dev>> > Date: Thursday, March 28, 2024 at 11:14 AM > To: Zou, Ling <l...@anl.gov <mailto:l...@anl.gov>> > Cc: Zhang, Hong <hzh...@mcs.anl.gov <mailto:hzh...@mcs.anl.gov>>, > petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov> > <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> > Subject: Re: [petsc-users] Does ILU(15) still make sense or should just use > LU? > > This is a bad situation, the solver is not really converging. This can happen > with ILU() sometimes, it so badly scales things that the preconditioned > residual decreases a lot but the true residual is not really getting smaller. > Since your matrices > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > This is a bad situation, the solver is not really converging. This can > happen with ILU() sometimes, it so badly scales things that the > preconditioned residual decreases a lot but the true residual is not really > getting smaller. Since your matrices are small best to stick to LU. > > You can use -ksp_norm_type unpreconditioned to force the convergence test > to use the true residual for a convergence test and the solver will discover > that it is not converging. > > Barry > > > > On Mar 28, 2024, at 11:43 AM, Zou, Ling via petsc-users > <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> wrote: > > Hong, thanks! That makes perfect sense. > A follow up question about ILU. > > The following is the performance of ILU(5). Note that each KPS solving > reports converged but as the output shows, the preconditioned residual does > while true residual does not. Is there any way this performance could be > improved? > Background: the preconditioning matrix is finite difference generated, and > should be exact. > > -Ling > > Time Step 21, time = -491.75, dt = 1 > NL Step = 0, fnorm = 6.98749E+01 > 0 KSP preconditioned resid norm 1.684131526824e+04 true resid norm > 6.987489798042e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.970568556551e+02 true resid norm > 6.459553545222e+01 ||r(i)||/||b|| 9.244455064582e-01 > 2 KSP preconditioned resid norm 3.349113985192e+02 true resid norm > 7.250836872274e+01 ||r(i)||/||b|| 1.037688366186e+00 > 3 KSP preconditioned resid norm 3.290585904777e+01 true resid norm > 1.186282435163e+02 ||r(i)||/||b|| 1.697723316169e+00 > 4 KSP preconditioned resid norm 8.530606201233e+00 true resid norm > 4.088729421459e+01 ||r(i)||/||b|| 5.851499665310e-01 > Linear solve converged due to CONVERGED_RTOL iterations 4 > NL Step = 1, fnorm = 4.08788E+01 > 0 KSP preconditioned resid norm 1.851047973094e+03 true resid norm > 4.087882723223e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.696809614513e+01 true resid norm > 2.720016413105e+01 ||r(i)||/||b|| 6.653851387793e-01 > 2 KSP preconditioned resid norm 5.751891392534e+00 true resid norm > 3.326338240872e+01 ||r(i)||/||b|| 8.137068663873e-01 > 3 KSP preconditioned resid norm 8.540729397958e-01 true resid norm > 8.672410748720e+00 ||r(i)||/||b|| 2.121492062249e-01 > Linear solve converged due to CONVERGED_RTOL iterations 3 > NL Step = 2, fnorm = 8.67124E+00 > 0 KSP preconditioned resid norm 5.511333966852e+00 true resid norm > 8.671237519593e+00 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.174962622023e+00 true resid norm > 8.731034658309e+00 ||r(i)||/||b|| 1.006896032842e+00 > 2 KSP preconditioned resid norm 1.104604471016e+00 true resid norm > 1.018397505468e+01 ||r(i)||/||b|| 1.174454630227e+00 > 3 KSP preconditioned resid norm 4.257063674222e-01 true resid norm > 4.023093124996e+00 ||r(i)||/||b|| 4.639583584126e-01 > 4 KSP preconditioned resid norm 1.023038868263e-01 true resid norm > 2.365298462869e+00 ||r(i)||/||b|| 2.727751901068e-01 > 5 KSP preconditioned resid norm 4.073772638935e-02 true resid norm > 2.302623112025e+00 ||r(i)||/||b|| 2.655472309255e-01 > 6 KSP preconditioned resid norm 1.510323179379e-02 true resid norm > 2.300216593521e+00 ||r(i)||/||b|| 2.652697020839e-01 > 7 KSP preconditioned resid norm 1.337324816903e-02 true resid norm > 2.300057733345e+00 ||r(i)||/||b|| 2.652513817259e-01 > 8 KSP preconditioned resid norm 1.247384902656e-02 true resid norm > 2.300456226062e+00 ||r(i)||/||b|| 2.652973374174e-01 > 9 KSP preconditioned resid norm 1.247038855375e-02 true resid norm > 2.300532560993e+00 ||r(i)||/||b|| 2.653061406512e-01 > 10 KSP preconditioned resid norm 1.244611343317e-02 true resid norm > 2.299441241514e+00 ||r(i)||/||b|| 2.651802855496e-01 > 11 KSP preconditioned resid norm 1.227243209527e-02 true resid norm > 2.273668115236e+00 ||r(i)||/||b|| 2.622080308720e-01 > 12 KSP preconditioned resid norm 1.172621459354e-02 true resid norm > 2.113927895437e+00 ||r(i)||/||b|| 2.437861828442e-01 > 13 KSP preconditioned resid norm 2.880752338189e-03 true resid norm > 1.076190247720e-01 ||r(i)||/||b|| 1.241103412620e-02 > Linear solve converged due to CONVERGED_RTOL iterations 13 > NL Step = 3, fnorm = 1.59729E-01 > 0 KSP preconditioned resid norm 1.676948440854e+03 true resid norm > 1.597288981238e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.266131510513e+00 true resid norm > 1.819663943811e+00 ||r(i)||/||b|| 1.139220244542e+01 > 2 KSP preconditioned resid norm 2.239911493901e+00 true resid norm > 1.923976907755e+00 ||r(i)||/||b|| 1.204526501062e+01 > 3 KSP preconditioned resid norm 1.446859034276e-01 true resid norm > 8.692945031946e-01 ||r(i)||/||b|| 5.442312026225e+00 > Linear solve converged due to CONVERGED_RTOL iterations 3 > NL Step = 4, fnorm = 1.59564E-01 > 0 KSP preconditioned resid norm 1.509663716414e+03 true resid norm > 1.595641817504e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.995956587709e+00 true resid norm > 1.712323298361e+00 ||r(i)||/||b|| 1.073125108390e+01 > 2 KSP preconditioned resid norm 1.994336275847e+00 true resid norm > 1.741263472491e+00 ||r(i)||/||b|| 1.091262119975e+01 > 3 KSP preconditioned resid norm 1.268035008497e-01 true resid norm > 8.197057317360e-01 ||r(i)||/||b|| 5.137153731769e+00 > Linear solve converged due to CONVERGED_RTOL iterations 3 > Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 4 > Solve Did NOT Converge! > > > > From: Zhang, Hong <hzh...@mcs.anl.gov <mailto:hzh...@mcs.anl.gov>> > Date: Wednesday, March 27, 2024 at 4:59 PM > To: petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov> > <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>>, Zou, Ling > <l...@anl.gov <mailto:l...@anl.gov>> > Subject: Re: Does ILU(15) still make sense or should just use LU? > > Ling, > ILU(level) is used for saving storage space with more computations. Normally, > we use level=1 or 2. It does not make sense to use level 15. If you have > sufficient space, LU would be the best. > Hong > > From: petsc-users <petsc-users-boun...@mcs.anl.gov > <mailto:petsc-users-boun...@mcs.anl.gov>> on behalf of Zou, Ling via > petsc-users <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> > Sent: Wednesday, March 27, 2024 4:24 PM > To: petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov> > <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> > Subject: [petsc-users] Does ILU(15) still make sense or should just use LU? > > Hi, I’d like to avoid using LU, but in some cases to use ILU and still > converge, I have to go to ILU(15), i.e., `-pc_factor_levels 15`. Does it > still make sense, or should I give it up and switch to LU? > > > > For this particular case, ~2k DoF, and both ILU(15) and LU perform similarly > in terms of wall time. > > > > -Ling >
