On Dec 23, 2011, at 2:15 AM, Jed Brown wrote: > Related: Mark's parallel Gauss-Seidel involves splitting the off-diagonal > update apart into two or more pieces that can operate independently. >
Humm, my G-S in not in PETSc and it is perfectly scalable. It does have more complex communication patterns but they are O(1) in latency and bandwidth. I'm not sure I understand your description above. Mark > On Fri, Dec 23, 2011 at 01:01, Jed Brown <jedbrown at mcs.anl.gov> wrote: > Can we enumerate the matrix operations that are known to be non-scalable > currently? > > 1. MatPermute() does ISAllGather() on the row indices and requires the user > to ISAllGather() the column indices before calling (I consider this to be an > interface bug similar to the MatGetSubMatrix() one we fixed before > petsc-3.1). The problem is that we have to determine the new row and column > or each locally visible row and column. This would be equivalent to having a > scalable ISInvertPermutation(), which is not difficult to implement, and a > way to retrieve data from off-process (new column indices for the "B" part). > The latter can be done with a "VecScatter for integers", or trivially with > PetscBG (name subject to change). > > 2. MatGetSubMatrix(), also called by MatPermute(). For this, it is sufficient > to determine where each currently owned entry should go (or be skipped). We > could do this by making a parallel int-vector for the entire column space, > setting it all to -1, then pushing the "new" column index (location in the > IS) over the IS. This gives a parallel vector that is -1 for all column > indices we don't want and the non-negative new column index for those that we > do want. Now we retrieve from the parallel vector everything we need for the > local column space. > > 3. MatIncreaseOverlap_MPISBAIJ(), I haven't looked carefully. > > I'm writing MatConvert_Nest_AIJ() which needs a similar feature. Mark wants > parallel MIS and MOOSE wants parallel coloring, both of which involve moving > integer data over the scatter context. > > What other implementations are currently non-scalable? Do we need other > primitives, or would two-way integer operations over the MPIXAIJ scatter and > via an IS be sufficient? > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20111223/2d136573/attachment.html>
