On Wed, 23 Jun 2010 11:45:56 +0200, francois pacull <fpacull at fluorem.com> wrote: > Dear PETSc team, > > I have a question regarding the MatGetSubMatrix routine (release > 3.0.0-p7). I am using it to change the partitioning of a MATMPIAIJ matrix:
How many processes are you using to provide the numbers below? The MPIAIJ implementation does a non-scalable operation ISAllGather and caches the result so that all future calls (with MAT_REUSE_MATRIX) will be fast. If you won't be doing the operation again, you could destroy this cache (it is PetscObjectCompose'd with the submatrix under the name "ISAllGather"), but it won't change the peak usage since that index set is needed when extracting the submatrix. It is possible to write a scalable MatGetSubMatrix, but it would be significantly more complicated than the current implementation and hasn't been done. Is this issue coming up with your use of PCFieldSplit? In that case, I would suggest living with the memory use until you find a split that you like, then create a MatShell that holds the blocks separately and implement MatGetSubMatrix (by matching against your distributed index sets). Dmitry is working on a new matrix type in petsc-dev that may be able to do this for you, but it's not ready yet. Jed
