On Nov 18, 2010, at 4:14 PM, Jed Brown wrote: > On Thu, Nov 18, 2010 at 23:04, Barry Smith <bsmith at mcs.anl.gov> wrote: > Jed, this is what started this entire process that you are unhappy with :-). > > I know, but I was bothered by an IS interface that destroys structure > (necessarily flattens everything to scalar indices).
I understand your concern. > > > Currently it only produces serial matrices, in which case both isrow and > > iscol should be serial. But to extract on subcomms, the user would pass in > > (isrow,iscol) living on subcomms. This would actually make MatGetSubMatrix > > a redundant API because the user could pass a single global (isrow,iscol) > > to MatGetSubMatrices(). Along these lines, it is easy to implement the > > trivially cheap ISGetLocalIS() so that, e.g. MatGetSubMatrix_MPIAIJ_Private > > doesn't need to do anything weird. > > We are getting closer. But you still haven't resolved my big questions. > > (1) Where do we cache ISGetAllIndices(IS,IS*) (replacement for > ISAllGather()) and ISGetNonlocalIndices(IS,IS*)? I want to cache them in the > original IS, is that ok with you? > > Yes, perhaps with some way to free that memory later (we've already gotten > questions about why calling MatGetSubMatrix triples the memory usage of the > original matrix, and they have to make another copy, and then destroy the > matrix they used to get the submatrix in order to reclaim that memory). But > that can be a long-term API issue. > > If IS is immutable, the I definitely agree with caching the gathered part on > the IS. But MatGetSubMatrix_MPIAIJ should also cache the original IS so that > it doesn't produce garbage if the user tries to MAT_REUSE_MATRIX with a > different IS. I have no problem with the new matrix also caching the ORIGINAL IS for a quick check. That's the particular implementations business. > > (2) What is the meaning of the IS that is returned? It really bothers me that > it is a sequential beast. What is it suppose to mean? To me it is information > about a parallel object therefor belongs in that parallel object. Am I just > wrong to be bothered by it? > > It doesn't bother me that it is sequential. After all, you are requesting a > sequential matrix defined by extracting the part defined by sequential > (isrow,iscol) from the parallel matrix. In this particular case. But will this be true for all uses of ISAllGather() (or any new name given it) in the future? Barry > Other procs are requesting different parts and will get back their own > sequential parts. > > Jed
