On Thu, Nov 18, 2010 at 22:50, Dmitry Karpeev <karpeev at mcs.anl.gov> wrote:
> In general, I don't see how ISAllGather can be any better than > ISGetTotalIndices: > the operations underneath are essentially the same. > ISGetTotalIndices can take advantage of any special structure just as > ISAllGather could: > by specializing the implementation to those subtypes. This can save > some communication > (e.g., ISAllGather_Stride might need to gather the initial offsets, > strides and the total sizes > only). The minute the actual indices are requested from the > resulting IS, it will have to allocate > the same amount of storage that ISGetTotalIndices requires. > Indeed, but there are a lot of operations you can do on an IS without getting the indices. Also, gathering strides is O(# procs) instead of O(global problem size), so there is a big communication difference. I'm not saying the difference will be immediately realized for the important use cases, but one leaves a straightforward way to optimize, the other offers no such thing. > ISAllGather followed by ISDifference is NOT what I need from > ISGetNonlocalIndices. > That would eliminate the indices in the range of the local portion of > the IS, which might > also wipe out some of the remote indices just gathered: multiple ranks > might contain the same > global index. Furthermore, even if all of the local ranges within the > IS are disjoint, > ISDifference is a relatively complicated operation, while I can easily > excise the local > part after the MPI_Allgatherv occurred, because, as a byproduct, I > obtain the offset > of the local part in the gathered indices. Otherwise I'd have to > reconstruct it using a scan. > Okay, but what is the meaning of a parallel IS where a given global index shows up in multiple procs' owned parts? In particular when do you need the "nonlocal indices" of such an IS? I guess the answer is that it would be needed to produce a "blown up" global matrix where the local pieces were overlapping. I know I mentioned this as a dirty hack for building overlapping matrices on subcomms, but the thing you are actually looking for is a matrix on a subcomm, where there are no duplicate indices in the IS of the subcomm. If you are doing that by making MatGetSubMatrices() handle (isrow,iscol) on subcomms, then I think you would get the right thing with ISDifference. Note that ISDifference can be implemented very efficiently if the arrays are sorted (I feel like I wrote code for that, or something very similar, not too long ago.) Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20101118/5128bf98/attachment.html>
