Hello,
We are attempting to use the PETSc KSP solver framework in a fluid dynamics
simulation we developed. The solution is part of a pressure projection and
solves a Poisson problem. We use a cell-centered layout with a regular grid in
3d. We started with ex34.c from the KSP tutorials since it has the correct
calls for the 3d DMDA, uses a cell-centered layout, and states that it works
with multi-grid. We modified the operator construction function to match the
coefficients and Dirichlet boundary conditions used in our problem (we’d also
like to support Neumann but left those out for now to keep things simple). As a
result of the modified boundary conditions, our version does not perform a null
space removal on the right hand side or operator as the original did. We also
modified the right hand side to contain a sinusoidal pattern for testing. Other
than these changes, our code is the same as the original ex34.c
With the default KSP options and using CG with the default pre-conditioner and
without a pre-conditioner, we see good convergence. However, we’d like to
accelerate the time to solution further and target larger problem sizes (>=
1024^3) if possible. Given these objectives, multi-grid as a pre-conditioner
interests us. To understand the improvement that multi-grid provides, we ran
ex45 from the KSP tutorials. ex34 with CG and no pre-conditioner appears to
converge in a single iteration and we wanted to compare against a problem that
has similar convergence patterns to our problem. Here’s the tests we ran with
ex45:
mpirun -n 16 ./ex45 -da_grid_x 129 -da_grid_y 129 -da_grid_z 129
time in KSPSolve(): 7.0178e+00
solver iterations: 157
KSP final norm of residual: 3.16874e-05
mpirun -n 16 ./ex45 -da_grid_x 129 -da_grid_y 129 -da_grid_z 129 -ksp_type cg
-pc_type none
time in KSPSolve(): 4.1072e+00
solver iterations: 213
KSP final norm of residual: 0.000138866
mpirun -n 16 ./ex45 -da_grid_x 129 -da_grid_y 129 -da_grid_z 129 -ksp_type cg
time in KSPSolve(): 3.3962e+00
solver iterations: 88
KSP final norm of residual: 6.46242e-05
mpirun -n 16 ./ex45 -da_grid_x 129 -da_grid_y 129 -da_grid_z 129 -pc_type mg
-pc_mg_levels 5 -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1
-mg_levels_pc_type bjacobi
time in KSPSolve(): 1.3201e+00
solver iterations: 4
KSP final norm of residual: 8.13339e-05
mpirun -n 16 ./ex45 -da_grid_x 129 -da_grid_y 129 -da_grid_z 129 -pc_type mg
-pc_mg_levels 5 -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1
-mg_levels_pc_type bjacobi -ksp_type cg
time in KSPSolve(): 1.3008e+00
solver iterations: 4
KSP final norm of residual: 2.21474e-05
We found the multi-grid pre-conditioner options in the KSP tutorials makefile.
These results make sense; both the default GMRES and CG solvers converge and CG
without a pre-conditioner takes more iterations. The multi-grid pre-conditioned
runs are pretty dramatically accelerated and require only a handful of
iterations.
We ran our code (modified ex34.c as described above) with the same parameters:
mpirun -n 16 ./solver_test -da_grid_x 128 -da_grid_y 128 -da_grid_z 128
time in KSPSolve(): 5.3729e+00
solver iterations: 123
KSP final norm of residual: 0.00595066
mpirun -n 16 ./solver_test -da_grid_x 128 -da_grid_y 128 -da_grid_z 128
-ksp_type cg -pc_type none
time in KSPSolve(): 3.6154e+00
solver iterations: 188
KSP final norm of residual: 0.00505943
mpirun -n 16 ./solver_test -da_grid_x 128 -da_grid_y 128 -da_grid_z 128
-ksp_type cg
time in KSPSolve(): 3.5661e+00
solver iterations: 98
KSP final norm of residual: 0.00967462
mpirun -n 16 ./solver_test -da_grid_x 128 -da_grid_y 128 -da_grid_z 128
-pc_type mg -pc_mg_levels 5 -mg_levels_ksp_type richardson
-mg_levels_ksp_max_it 1 -mg_levels_pc_type bjacobi
time in KSPSolve(): 4.5606e+00
solver iterations: 44
KSP final norm of residual: 949.553
mpirun -n 16 ./solver_test -da_grid_x 128 -da_grid_y 128 -da_grid_z 128
-pc_type mg -pc_mg_levels 5 -mg_levels_ksp_type richardson
-mg_levels_ksp_max_it 1 -mg_levels_pc_type bjacobi -ksp_type cg
time in KSPSolve(): 1.5481e+01
solver iterations: 198
KSP final norm of residual: 0.916558
We performed all tests with petsc-3.7.6.
The trends with CG and GMRES seem consistent with the results from ex45.
However, with multi-grid, something doesn’t seem right. Convergence seems poor
and the solves run for many more iterations than ex45 with multi-grid as a
pre-conditioner. I extensively validated the code that builds the matrix and
also confirmed that the solution produced by CG, when evaluated with the system
of equations elsewhere in our simulation, produces the same residual as
indicated by PETSc. Given that we only made minimal modifications to the
original example code, it seems likely that the operators constructed for the
multi-grid levels are ok.
We also tried a variety of other suggested parameters for the multi-grid
pre-conditioner as suggested in related mailing list posts but we didn’t
observe any significant improvements over the results above.
Is there anything we can do to check the validity of the coefficient matrices
built for the different multi-grid levels? Does it look like there could be
problems there? Or any other suggestions to achieve better results with
multi-grid? I have the -log_view, -ksp_view, and convergence monitor output
from the above tests and can post any of it if it would assist.
Thanks
