What's up with this? Begin forwarded message:
> From: "Stephen C. Jardin" <sjardin at pppl.gov> > Date: January 27, 2011 6:49:27 PM CST > To: <bsmith at mcs.anl.gov> > Cc: "Lois Curfman McInnes" <curfman at mcs.anl.gov>, <egng at lbl.gov>, > <kd2112 at columbia.edu> > Subject: Request for new PETSc capability > > Dear Barry, > > The M3D-C1 project is one of the major code projects in CEMM. It is a fully > implicit formulation of the 3D MHD equations using high-order 3D finite > elements with continuous derivatives in all directions. In a typical > problem, the 3D domain consists of approximately 100 2D planes, spread out > equally around a torus. The grid we use is unstructured within each 2D plane > (where the coupling of elements is very strong), but is structured and > regular across the planes (where the coupling is much weaker and is confined > to nearest neighbors. > > Our plan has always been to solve the large sparse matrix equation we get > using GMRES with a block Jacobi preconditioner obtained by using SuperLU_dist > within each 2D plane. We have implemented this using PETSc and find that it > leads to a very efficient iterative solve that converges in just a few > iterations for the time step and other parameters that we normally use. > However, the present version of PETSc/3.1 only allows a single processor per > plane(block) when using the block Jacobi preconditioner. This severely > limits the maximum problem size that we can run, as we can use only 100 > processors for a problem with 100 2D planes. > > Several years ago, when we were planning this project, we spoke with Hong > Zhang about this solver strategy and she told us that if there was a demand > for it, the present limitation restricting the block Jacobi preconditioner to > a single processor could be lifted. We are now at point in our project > where we need to request this. We have demonstrated good convergence of the > iterative solver, but need to be able to run with 10-100 processors per plane > (block) in order use 1000-10000 processors total to obtain the kind of > resolution we need for our applications. > > Would it be possible for your group to generalize the block Jacobi > preconditioner option so that the blocks could be distributed over multiple > processors? If so, could you give us a timeline for this to appear in a > PETSc release? > > Thank you, and Best Regards, > > Steve Jardin (for the CEMM team) > > > Cc Lois McInnes > Esmond Ng > David Keyes
