On Wed, Nov 8, 2017 at 8:02 PM, Mark Lohry <[email protected]> wrote: > Do you mean the element Jacobian matrix has 35 "vertices"? You have 5x5 >> dense blocks (or at least you don't care about resolving any sparsity or >> you want to use block preconditioners. So your element Jacobians are >> [35x5]^2 dense(ish) matrices. This is not particularly useful for the >> solvers. > > > Yes, the local element jacobians are (35x5)^2 ( (N+1)(N+2)(N+3)/6 vertices > per Nth order element for 3D). They'd be 100% dense. So the real blocks of > the full system jacobian are of that size, and not 5x5. Block ILU is pretty > popular for these discretizations for obvious reasons. >
OK, but don't set that blocks size. > > This is on the full NS problem I assume. You need special solvers for >> this. Scalable solvers for NS is not trivial. > > > Yes, compressible 3D NS. I'm aware scaling isn't trivial, but this isn't a > scaling problem, I'm obviously computing a wrong preconditioner since it's > actively hurting the linear solve. > By scaling I mean not a direct solver. > > Yes, you should use Gauss-Seidel for non-symmetric systems. Or indefinite >> systems > > > Thanks! > sor > > You want to use the Schur complement PCs for NS. We have some support for >> PC where you give us a mass matrix. > > > I'll look into it, but I'm not aware of Schur complement being used for > compressible/coupled NS, only for incompressible/segregated. > Really? OK, you've looked into it more than me. This is DG where the mass matrix is a block diagonal constant you typically > just invert once at the beginning; is there a PC where this can be > exploited? > Don't know but being able to explicitly invert this might let you compute an exact Schur complement matrix and then use algebraic solvers (like LU and AMG). > > > On Wed, Nov 8, 2017 at 7:04 PM, Mark Adams <[email protected]> wrote: > >> >> >> On Tue, Nov 7, 2017 at 3:00 PM, Mark Lohry <[email protected]> wrote: >> >>> I've now got gamg running on matrix-free newton-krylov with the jacobian >>> provided by coloring finite differences (thanks again for the help). 3D >>> Poisson with 4th order DG or higher (35^2 blocks), gamg with default >>> settings is giving textbook convergence, which is great. Of course coupled >>> compressible navier-stokes is harder, and convergence is bad-to-nonexistent. >>> >>> >>> 1) Doc says "Equations must be ordered in “vertex-major” ordering"; in >>> my discretization, each "node" has 5 coupled degrees of freedom (density, 3 >>> x momentum, energy). I'm ordering my unknowns: >>> >>> rho_i, rhou_i, rhov_i, rhow_i, Et_i, rho_i+1, rhou_i+1, ... e.g. >>> row-major matrix order if you wrote the unknowns [{rho}, {rhou}, ... ]. >>> >>> and then setting block size to 5. Is that correct? >>> >> >> yes >> >> >>> I've also tried using the actual sparsity of the matrix >>> >> >> Sorry, but using for what? >> >> >>> which has larger dense blocks (e.g. [35x5]^2), but neither seemed to >>> help. >>> >> >> Do you mean the element Jacobian matrix has 35 "vertices"? You have 5x5 >> dense blocks (or at least you don't care about resolving any sparsity or >> you want to use block preconditioners. So your element Jacobians are >> [35x5]^2 dense(ish) matrices. This is not particularly useful for the >> solvers. >> >> >>> >>> >>> 2) With default settings, and with -pc_gamg_square_graph, >>> pc_gamg_sym_graph, agg_nsmooths 0 mentioned in the manual, the eigenvalue >>> estimates explode on the coarsest level, which I don't see with poisson: >>> >>> Down solver (pre-smoother) on level 1 ------------------------------- >>> KSP Object: (mg_levels_1_) 32 MPI processes >>> type: chebyshev >>> eigenvalue estimates used: min = 0.18994, max = 2.08935 >>> eigenvalues estimate via gmres min 0.00933256, max 1.8994 >>> Down solver (pre-smoother) on level 2 ------------------------------- >>> KSP Object: (mg_levels_2_) 32 MPI processes >>> type: chebyshev >>> eigenvalue estimates used: min = 0.165969, max = 1.82566 >>> eigenvalues estimate via gmres min 0.0290728, max 1.65969 >>> Down solver (pre-smoother) on level 3 ------------------------------- >>> KSP Object: (mg_levels_3_) 32 MPI processes >>> type: chebyshev >>> eigenvalue estimates used: min = 0.146479, max = 1.61126 >>> eigenvalues estimate via gmres min 0.204673, max 1.46479 >>> Down solver (pre-smoother) on level 4 ------------------------------- >>> KSP Object: (mg_levels_4_) 32 MPI processes >>> type: chebyshev >>> eigenvalue estimates used: min = 6.81977e+09, max = 7.50175e+10 >>> eigenvalues estimate via gmres min -2.76436e+12, max 6.81977e+10 >>> >>> What's happening here? (Full -ksp_view below) >>> >> >> This is on the full NS problem I assume. You need special solvers for >> this. Scalable solvers for NS is not trivial. >> >> >>> >>> 3) I'm not very familiar with chebyshev smoothers, but they're only for >>> SPD systems (?). >>> >> >> Yes, you should use Gauss-Seidel for non-symmetric systems. Or indefinite >> systems >> >> >>> Is this an inappropriate smoother for this problem? >>> >>> 4) With gmres, the preconditioned residual is ~10 orders larger than the >>> true residual; and the preconditioned residual drops while the true >>> residual rises. I'm assuming this means something very wrong? >>> >> >> Yes, your preconditioner is no good. >> >> >>> >>> 5) -pc_type hyper -pc_hypre_type boomeramg also works perfectly for the >>> poisson case, but hits NaN on the first cycle for NS. >>> >>> >>> >> You want to use the Schur complement PCs for NS. We have some support for >> PC where you give us a mass matrix. These are motivated by assuming there >> is a "commutation" (that is not true). These were developed by, among >> others, Andy Wathan, if you want to do a literature search. You will >> probably have to read the literature to understand the options and issues >> for your problem. Look at the PETSc manual and you should find a >> description of PETSc support for these PCs and some discussion of them and >> references. This should get you started. >> >> Mark >> >> >> >> >> > >
