I agree with Matt, if you have generated a parallel matrix and have to then load balance the matrix by migrating matrix entries you have already lost the game. The goal should always be to partition whatever the underlying infrastructure (be it a mesh, a network, etc) before generating the matrix so that the matrix generated will be well-balanced. We use DM as the object that manages the interaction between the "underlying infrastructure" and the algebra: we provide them for structured grids, DMDA, DMStag, for unstructured grids DMPLEX, for quad/oct tree DMPFOREST, networks DMNETWORK, and for composites of these DMCOMPOSITE. You are, of course, also free to manage your "underlying infrastructure" yourself and manage the connections to the linear algebra.
That said, as Matt notes, you can use the MatPartitioning object and the MatPermute method to repartition a matrix that is already built to get the effect you appear to be looking for but that is not a recommended approach. Barry You mention specifically a nonlinear solver, for matrix based Newton methods; this requires evaluation of the nonlinear function, evaluation of the Jacobian entries and solution of the linear system. Would you evaluate the function and the matrix and then "move around" the vector and Jacobian for load balancing of the linear solve and then move back the Newton update to the original partitioning and repeat for each Newton step? That seems to be what you are asking for? What we are simply advocating is determine the needed "moving around" entirely outside of the Newton method, so that everything, the function evaluation, the Jacobian entry evaluation and the linear solve all take place on the "moved around" ordering and one is not moving back and forth between two different partitioning inside the nonlinear solve (or ODE integrator etc). The moving around inside the Newton is expensive and not needed if the partitioning is handled outside of the Newton. > On Oct 11, 2019, at 7:15 AM, Matthew Knepley via petsc-users > <petsc-users@mcs.anl.gov> wrote: > > On Fri, Oct 11, 2019 at 4:09 AM von Ramm, Alexander via petsc-users > <petsc-users@mcs.anl.gov> wrote: > Hi together, > I'm currently evaluating to use PETSc as framework for our next software > project. > My question concerns dynamic repartitioning: I have already found in the > Documentation (Chapt. 3.5) that PETSC does not currently support dynamic > repartitioning, load balancing by migrating matrix entries between processes, > etc. I know that Zoltan e.g., which PETSC can be configured with offers this > functionality. Is there an tutorial demonstrating how the automatic > load-balancing features of Zoltan can be used in combination with > linear/non-linear solvers offered by PETSC? Or should one, for problems > requiring dynamic load-balancing strategies move to Trilinos all together? > > I am not sure Zoltan offers anything beyond what we have now. Here is a > sketch of dynamic load balancing for a given matrix: > > 1) Develop a partition that is good for you using MatPartitioning > > This is the problem specific part since you have to use weights to tell > the partitioner about your computation. No package can do this automatically. > > 2) Move the matrix using MatPermute > > However, very few problems actually work this way. We have the same workflow > for meshes > > DMPlexDistribute() > > and based upon the partition you get out you can permute Vec/Mat and also > arbitrary data > > DMPlexDistributeData() > > Thanks, > > Matt > > Thank you for any input, > Best Regards, > Alex von Ramm > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/
