It is possible there could be other methods added to IS that would be used by VecScatterCreate() and MatGet...Matrix() that "gather" needed information without everyone getting everything. What they are I don't know but by looking at the parallel VecScatterCreates and MatGetSubMatrices() you might get some ideas.
Barry On Nov 18, 2010, at 4:41 PM, Jed Brown wrote: > On Thu, Nov 18, 2010 at 23:12, Dmitry Karpeev <karpeev at mcs.anl.gov> wrote: > I'm not disagreeing, but I also don't see a general way of figuring > out the off-rank > columns for a submatrix that is destined to live on a subcomm. > > VecScatter figures out who is sending to who without allgathering everyone's > indices. I realize it's a slightly different problem and I don't have a > complete solution right now. And iscol often has some structure (even if > just block). > > Another point is that since we anticipate a submatrix is extracted on a > subcomm, > the size of the submatrix and the subcomm might not grow in the weak > scaling case. > > That depends whether the subdomains are different materials or just > solver-friendly subdomains (e.g. to reduce iterations in ASM and make overlap > less expensive in 3D). Consider getting submatrices for Fluid and > > The names are definitely an stop-gap fix: for consistency we'd have to change > ISGetIndices --> ISGetLocalIndices, ISGetTotalIndices --> > ISGetIndices, VecGetArray --> VecGetLocalArray, etc. > Maybe this will be done one day :-) > > I don't know, I like to endorse the domain decomposition philosophy and make > the simple things (and in particular, storage) local by default, with > parallel semantics where appropriate. In particular, you should have to > spell out a word with non-scalable connotations in order to call a blatantly > non-scalable function. > > Jed
