Matthew Knepley wrote: > On Tue, Nov 10, 2009 at 4:51 AM, Jed Brown <jed at 59a2.org > <mailto:jed at 59a2.org>> wrote: > > jarunan at ascomp.ch <mailto:jarunan at ascomp.ch> wrote: > > Total number of cells is 744872, divided into 40 blocks. In one > > processor, MatCreateMPIAIJWithArrays() takes 0.097 sec but 280 sec > with > > 4 processors. Usually, this routine has no problem with small test > case. > > It works the same for one or more than one processors. > > This sounds like incorrect preallocation. Is your PETSc built with > debugging? Debug does some extra integrity checks that don't add > significantly to the time (although other Debug checks do), but it would > be useful to know that they pass. In particular, it checks that your > rows are sorted. If they are not sorted then PETSc's preallocation > would be wrong. (I actually don't think this requirement enables > significantly faster implementation, so I'm tempted to change it to work > correctly with unsorted rows.) > > > I do not think its preallocation per se, since 1 proc is fast. I think > that your partition of rows fed to the MatCreate() call does not match > what you provide to MatSetValues() and thus you do a lot of > communication in MatAssemblyEnd(). There are 2 ways to debug this:
Matt, he says MatSetValues() is fast, but MatCreateMPIAIJWithArrays() is slow. Look how preallocation is done (mpiaij.c:3263). This would do the correct thing in serial, but be under-allocate the diagonal part when the rows are not sorted. It looks like a silly "optimization" to me. Jed -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 261 bytes Desc: OpenPGP digital signature URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20091110/c2355211/attachment.pgp>
