Re: [petsc-users] Using nonzero -pc_hypre_boomeramg_restriction_type in field split

[Alexander Lindsay]

OpenMP is definitely linked in and appears in the stacktrace but I haven’t asked for any threads (to my knowledge). On Apr 13, 2023, at 7:03 PM, Mark Adams <mfad...@lbl.gov> wrote:  Are you using OpenMP? ("OMP"). If so try without it. On Thu, Apr 13, 2023 at 5:07 PM Alexander Lindsay <alexlindsay...@gmail.com> wrote: Here's the result.     0 KSP unpreconditioned resid norm 1.033076851740e+03 true resid norm 1.033076851740e+03 ||r(i)||/||b|| 1.000000000000e+00       Residual norms for fieldsplit_u_ solve.       0 KSP Residual norm           -nan       Residual norms for fieldsplit_p_ solve.       0 KSP Residual norm           -nan       Residual norms for fieldsplit_u_ solve.       0 KSP Residual norm           -nan       1 KSP Residual norm           -nan       Residual norms for fieldsplit_u_ solve.       0 KSP Residual norm           -nan   Linear solve did not converge due to DIVERGED_PC_FAILED iterations 0                  PC failed due to SUBPC_ERROR I probably should have read the FAQ on `-fp_trap` before sending my first email.  Working with this stack trace  (gdb) bt #0  0x00007fffe83a4286 in hypre_ParMatmul._omp_fn.1 () at par_csr_matop.c:1124 #1  0x00007ffff4982a16 in GOMP_parallel () from /lib/x86_64-linux-gnu/libgomp.so.1 #2  0x00007fffe83abfd1 in hypre_ParMatmul (A=<optimized out>, B=B@entry=0x55555da2ffa0) at par_csr_matop.c:967 #3  0x00007fffe82f09bf in hypre_BoomerAMGSetup (amg_vdata=<optimized out>, A=<optimized out>, f=<optimized out>,     u=<optimized out>) at par_amg_setup.c:2790 #4  0x00007fffe82d54f0 in HYPRE_BoomerAMGSetup (solver=<optimized out>, A=<optimized out>, b=<optimized out>,     x=<optimized out>) at HYPRE_parcsr_amg.c:47 #5  0x00007fffe940d33c in PCSetUp_HYPRE (pc=<optimized out>)     at /home/lindad/projects/moose/petsc/src/ksp/pc/impls/hypre/hypre.c:418 #6  0x00007fffe9413d87 in PCSetUp (pc=0x55555d5ef390)     at /home/lindad/projects/moose/petsc/src/ksp/pc/interface/precon.c:1017 #7  0x00007fffe94f856b in KSPSetUp (ksp=ksp@entry=0x55555d5eecb0)     at /home/lindad/projects/moose/petsc/src/ksp/ksp/interface/itfunc.c:408 #8  0x00007fffe94fa6f4 in KSPSolve_Private (ksp=ksp@entry=0x55555d5eecb0, b=0x55555d619730, x=<optimized out>)     at /home/lindad/projects/moose/petsc/src/ksp/ksp/interface/itfunc.c:852 #9  0x00007fffe94fd8b1 in KSPSolve (ksp=ksp@entry=0x55555d5eecb0, b=<optimized out>, x=<optimized out>)     at /home/lindad/projects/moose/petsc/src/ksp/ksp/interface/itfunc.c:1086 #10 0x00007fffe93d84a1 in PCApply_FieldSplit_Schur (pc=0x555555bef790, x=0x555556d5a510, y=0x555556d59e30)     at /home/lindad/projects/moose/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1185 #11 0x00007fffe9414484 in PCApply (pc=pc@entry=0x555555bef790, x=x@entry=0x555556d5a510, y=y@entry=0x555556d59e30)     at /home/lindad/projects/moose/petsc/src/ksp/pc/interface/precon.c:445 #12 0x00007fffe9415ad7 in PCApplyBAorAB (pc=0x555555bef790, side=PC_RIGHT, x=0x555556d5a510,     y=y@entry=0x555556e922a0, work=0x555556d59e30)     at /home/lindad/projects/moose/petsc/src/ksp/pc/interface/precon.c:727 #13 0x00007fffe9451fcd in KSP_PCApplyBAorAB (w=<optimized out>, y=0x555556e922a0, x=<optimized out>,     ksp=0x555556068fc0) at /home/lindad/projects/moose/petsc/include/petsc/private/kspimpl.h:421 #14 KSPGMRESCycle (itcount=itcount@entry=0x7fffffffcca0, ksp=ksp@entry=0x555556068fc0)     at /home/lindad/projects/moose/petsc/src/ksp/ksp/impls/gmres/gmres.c:162 #15 0x00007fffe94536f9 in KSPSolve_GMRES (ksp=0x555556068fc0)     at /home/lindad/projects/moose/petsc/src/ksp/ksp/impls/gmres/gmres.c:247 #16 0x00007fffe94fb1c4 in KSPSolve_Private (ksp=0x555556068fc0, b=b@entry=0x55555568e510, x=<optimized out>,     x@entry=0x55555607cce0) at /home/lindad/projects/moose/petsc/src/ksp/ksp/interface/itfunc.c:914 #17 0x00007fffe94fd8b1 in KSPSolve (ksp=<optimized out>, b=b@entry=0x55555568e510, x=x@entry=0x55555607cce0)     at /home/lindad/projects/moose/petsc/src/ksp/ksp/interface/itfunc.c:1086 #18 0x00007fffe9582850 in SNESSolve_NEWTONLS (snes=0x555556065610)     at /home/lindad/projects/moose/petsc/src/snes/impls/ls/ls.c:225 #19 0x00007fffe959c7ee in SNESSolve (snes=0x555556065610, b=0x0, x=<optimized out>)     at /home/lindad/projects/moose/petsc/src/snes/interface/snes.c:4809 On Thu, Apr 13, 2023 at 1:54 PM Barry Smith <bsm...@petsc.dev> wrote:   It would be useful to see the convergences inside the linear solve so perhaps start with  -ksp_monitor_true_residual  -fieldsplit_u_ksp_type richardson    (this is to allow the monitor below to work) -fieldsplit_u_ksp_max_its 1  -fieldsplit_u_ksp_monitor Perhaps others, Matt/Jed/Pierre/Stefano likely know better off the cuff than me. We should have a convenience option like -pc_fieldsplit_schur_monitor similar to the -pc_fieldsplit_gkb_monitor On Apr 13, 2023, at 4:33 PM, Alexander Lindsay <alexlindsay...@gmail.com> wrote: Hi, I'm trying to solve steady Navier-Stokes for different Reynolds numbers. My options table -dm_moose_fieldsplit_names u,p -dm_moose_nfieldsplits 2 -fieldsplit_p_dm_moose_vars pressure -fieldsplit_p_ksp_type preonly -fieldsplit_p_pc_type jacobi -fieldsplit_u_dm_moose_vars vel_x,vel_y -fieldsplit_u_ksp_type preonly -fieldsplit_u_pc_hypre_type boomeramg -fieldsplit_u_pc_type hypre -pc_fieldsplit_schur_fact_type full -pc_fieldsplit_schur_precondition selfp -pc_fieldsplit_type schur -pc_type fieldsplit works wonderfully for a low Reynolds number of 2.2. The solver performance crushes LU as I scale up the problem. However, not surprisingly this options table struggles when I bump the Reynolds number to 220. I've read that use of AIR (approximate ideal restriction) can improve performance for advection dominated problems. I've tried setting -pc_hypre_boomeramg_restriction_type 1 for a simple diffusion problem and the option works fine. However, when applying it to my field-split preconditioned Navier-Stokes system, I get immediate non-convergence:  0 Nonlinear |R| = 1.033077e+03       0 Linear |R| = 1.033077e+03   Linear solve did not converge due to DIVERGED_NANORINF iterations 0 Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 Does anyone have an idea as to why this might be happening? If not, I'd take a suggestion on where to set a breakpoint to start my own investigation. Alternatively, I welcome other preconditioning suggestions for an advection dominated problem. Alex