MATMPIDENSE does not implement any cyclic distribution. In parallel, a dense matrix is split by rows. Each process owns localrows*globalcols entries. Local sizes are to be intended as the size of the right and left vectors used in matvec operations, and are not strictly related with storage considerations. https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSize.html
Il Mer 13 Gen 2021, 15:40 Roland Richter <[email protected]> ha scritto: > Hei, > > I am currently struggling a bit with my understanding of local and global > rows/columns in distributed dense matrices in PETSc. > > The attached short program is creating a local matrix in every thread with > [matrix_size_rows, matrix_size_cols] as global size. Afterwards, I flatten > the matrix into a vector, and split this vector using the boost-function > split() when running using several MPI threads. In addition I create a > dense MPI-distributed matrix based on PETSc, using the same input sizes, > and set all values to zero (after I'm not interested in the content itself, > just the shape). Finally, I retrieve the global and local values for the > number of rows and columns using MatGetLocalSize, and the number of the > first and last row in each matrix using MatGetOwnershipRange() and print > them. According to the documentation I would expect that the PETSc-based > matrix is split up > > Now, for an input size of [8, 8] and a single thread I get > > > > > > > *Rank 0 has a total size of 8, 8 Rank 0 has an initial size of 64 Rank 0 > has a local matrix size of 8, 8 Rank 0 has a global matrix size of 8, 8 > Rank 0 has a matrix spanning from row 0 to 8 Rank 0 has a vector size of 64* > > which is what I expect. For two threads, though, I get > *Rank 0 has a total size of 8, 8* > *Rank 0 has an initial size of 64* > *Rank 0 has a local matrix size of 4, 4* > *Rank 0 has a global matrix size of 8, 8* > *Rank 0 has a matrix spanning from row 0 to 4* > *Rank 0 has a vector size of 32* > *Rank 1 has a total size of 8, 8* > *Rank 1 has an initial size of 64* > *Rank 1 has a local matrix size of 4, 4* > *Rank 1 has a global matrix size of 8, 8* > *Rank 1 has a matrix spanning from row 4 to 8* > * Rank 1 has a vector size of 32* > > Here, most entries make sense, except the size of the local matrices. Why > do I get a size of [4, 4], and not a size of [4, 8]? Each row should be > contiguous in the local process, and therefore each row should contain all > columns, not only a part of it. > > Is there a misunderstanding about how to use MatGetLocalSize(), or > something else? > > Thanks! > > Regards, > > Roland Richter > > >
