Bikash, I implemented MatTransposeMatMult_MPIDense_MPIDense() in the branch hzhang/mattransmatmult_dense. Once it passes our nightly tests, I'll merge it to petsc master branch.
See petsc/src/mat/examples/tests/ex104.c, in which I added a simple example doing B = P^T * A * P, where P is an MPIDENSE matrix and A is an MPIAIJ matrix. Let us know if you see any bug or performance issues. Hong On Fri, Oct 16, 2015 at 10:25 AM, Jed Brown <[email protected]> wrote: > Hong <[email protected]> writes: > > > Jed: > >> > >> > >> > I plan to implement MatTransposeMatMult_MPIDense_MPIDense via > >> > > >> > 1. add MatTransposeMatMult_elemental_elemental() > >> > 2. C_dense = P_dense^T * B_dense > >> > via MatConvert_dense_elemental() and MatConvert_elemental_dense() > >> > >> The above involves a ton of data movement and MPIDense is a logical > >> distribution for matrices with a modest number of columns. I think I > >> would just do the local GEMM and then MPI_Reduce_scatter it. > >> > > This would work. > > > > However, I recall that you did a smart ordering which allows > > MatConvert_mpidense_elemental() uses same physical matrix storage for > petsc > > and elemental, > > Same storage for vectors. This is your code, but I think you'll find > that it moves the matrix entries around. (Note that Elemental [MC,MR] > is a 2D distribution while MPIDense is 1D.) Also, I think it would be > better if this particular operation did not depend on Elemental. > > You could write a conversion to Elemental [VC,*], which would then match > the MPIDense distribution and thus not need to move any matrix entries. >
