On Dec 29, 2011, at 12:45 PM, Dave Nystrom wrote: > Jed Brown writes: >> On Thu, Dec 29, 2011 at 11:01, Dave Nystrom <dnystrom1 at comcast.net> wrote: >> >>> I have recently added the capability to have a separate preconditioning >>> matrix in the petsc interface for the code I am working with. I have two >>> types of preconditioning matrices that I have questions about. One is >>> tridiagonal and the other is 7 diagonals. In both cases, the the diagonals >>> are all lexically adjacent. Or phrased differently, the tridiagonal matrix >>> has a bandwidth of 3 and the 7 diagonal matrix has a bandwidth of 7 and so >>> they are compact or dense band systems. >>> >>> I was wondering what petsc ilu will do for preconditioning matrices like >>> these. Will it produce an exact lu factorization or a nearly exact >>> factorization? >> >> Yes >> >>> I'm interested in the answer to this question because I am thinking I >>> might be able to run this preconditioner on the gpu using the txpetscgpu >>> package. >> >> Likely pointless because this solve is probably not a big part of run-time. >> The bigger concern is the convergence rate that you lose by using this >> approximation. Matt and I mentioned this the last time you brought it up, >> but I recommend getting familiar with the literature. Then, get the math >> straight before looking for ways to transform into problems that you can >> run fast on a GPU or whatever. If you just optimize kernels, you're likely >> to optimize something that takes a small part of run-time and isn't really >> helping you anyway. > > So the application code is one that I am not the primary author on and it is > a 2d resistive MHD with extensions code for magnetized fusion problems. > Magnetized fusion problems tend to be highly anisotropic with qualitatively > different physics parallel and perpendicular to the magnetic field. The code > has 3 approaches to solving the linear systems. One is a suite of native cg > solvers that use a cholesky solver for the inner band of the matrix as the > preconditioner. As noted, this preconditioner ignores the coupling to the > second coordinate. However, this preconditioner produces the lowest > iteration count of the various preconditioner approaches that I have tried. >
This sounds like it is and SPD matrix. I would think GAMG would be OK. If you can set -pc_gamg_verbose -ksp_monitor' and send me the output I can see of anything jumps out. > The second solver approach is use of the agmg package which works well on all > of the linear solves except for the Hall matrix. For the Hall matrix, agmg > fails to produce a solution. > Are you using CG and Cholesky for the Hall matrix? If so it is probably a div(alpha(x)grad(u)). Mark > The third approach is an interface to petsc that I have added in the last few > months. So far, I have been able to get really good results with petsc on > all of the linear systems except for the Hall matrix. In general, the > fastest solution on these other linear systems has been petsc using jacobi > preconditioning running on the gpu. However, I think I may be able to do > even better on these other systems using petsc with the separate > preconditioning matrix and using ilu plus txpetscgpu. > > But, having invested the time to interface to petsc, I am now trying to > explore the vast set of options available to me to see what works best for my > problems from the petsc toolbox. But solving the Hall matrix, especially for > larger mesh sizes, still remains problematic. The lead author of this code > is investigating the Hall matrix formulation to see if there are better ways > to either formulate it or solve it and I am exploring what petsc has to offer > for solving it. The matrix is very ill conditioned. The formulation is > complicated enough that perhaps there is just a bug somewhere. But these > sort of questions are being investigated by the main author of the code. > > Anyway, I have so far gotten good performance gains using the gpu with petsc > but am totally open to and am trying to explore the whole petsc toolbox. >
