Thanks a ton, Hong. You guys are amazing! I'll report any bug or performance issue if I find any. On Oct 21, 2015 11:43 AM, "Hong" <[email protected]> wrote:
> Bikash, > I implemented MatTransposeMatMult_MPIDense_MPIDense() > in the branch hzhang/mattransmatmult_dense. > Once it passes our nightly tests, I'll merge it to petsc master branch. > > See petsc/src/mat/examples/tests/ex104.c, in which I added a simple > example doing > B = P^T * A * P, where P is an MPIDENSE matrix and A is an MPIAIJ matrix. > > Let us know if you see any bug or performance issues. > > Hong > > On Fri, Oct 16, 2015 at 10:25 AM, Jed Brown <[email protected]> wrote: > >> Hong <[email protected]> writes: >> >> > Jed: >> >> >> >> >> >> > I plan to implement MatTransposeMatMult_MPIDense_MPIDense via >> >> > >> >> > 1. add MatTransposeMatMult_elemental_elemental() >> >> > 2. C_dense = P_dense^T * B_dense >> >> > via MatConvert_dense_elemental() and MatConvert_elemental_dense() >> >> >> >> The above involves a ton of data movement and MPIDense is a logical >> >> distribution for matrices with a modest number of columns. I think I >> >> would just do the local GEMM and then MPI_Reduce_scatter it. >> >> >> > This would work. >> > >> > However, I recall that you did a smart ordering which allows >> > MatConvert_mpidense_elemental() uses same physical matrix storage for >> petsc >> > and elemental, >> >> Same storage for vectors. This is your code, but I think you'll find >> that it moves the matrix entries around. (Note that Elemental [MC,MR] >> is a 2D distribution while MPIDense is 1D.) Also, I think it would be >> better if this particular operation did not depend on Elemental. >> >> You could write a conversion to Elemental [VC,*], which would then match >> the MPIDense distribution and thus not need to move any matrix entries. >> > >
