Thanks Jed for your quick and helpful answer!
On 23/06/2010 12:31, Jed Brown wrote: > On Wed, 23 Jun 2010 11:45:56 +0200, francois pacull<fpacull at fluorem.com> > wrote: > >> Dear PETSc team, >> >> I have a question regarding the MatGetSubMatrix routine (release >> 3.0.0-p7). I am using it to change the partitioning of a MATMPIAIJ matrix: >> > How many processes are you using to provide the numbers below? > I used only 2 processes in that test. > The MPIAIJ implementation does a non-scalable operation ISAllGather and > caches the result so that all future calls (with MAT_REUSE_MATRIX) will > be fast. If you won't be doing the operation again, you could destroy > this cache (it is PetscObjectCompose'd with the submatrix under the name > "ISAllGather"), but it won't change the peak usage since that index set > is needed when extracting the submatrix. > > It is possible to write a scalable MatGetSubMatrix, but it would be > significantly more complicated than the current implementation and > hasn't been done. Is this issue coming up with your use of > PCFieldSplit? No, I was just trying to understand the results I got from a call to PetscMemoryGetMaximumUsage. I am dealing with a very large matrix and trying to save memory... > In that case, I would suggest living with the memory use > until you find a split that you like, then create a MatShell that holds > the blocks separately and implement MatGetSubMatrix (by matching against > your distributed index sets). Dmitry is working on a new matrix type in > petsc-dev that may be able to do this for you, but it's not ready yet. > > Jed > > francois.
