On Tue, Nov 10, 2009 at 4:51 AM, Jed Brown <jed at 59a2.org> wrote: > jarunan at ascomp.ch wrote: > > Total number of cells is 744872, divided into 40 blocks. In one > > processor, MatCreateMPIAIJWithArrays() takes 0.097 sec but 280 sec with > > 4 processors. Usually, this routine has no problem with small test case. > > It works the same for one or more than one processors. > > This sounds like incorrect preallocation. Is your PETSc built with > debugging? Debug does some extra integrity checks that don't add > significantly to the time (although other Debug checks do), but it would > be useful to know that they pass. In particular, it checks that your > rows are sorted. If they are not sorted then PETSc's preallocation > would be wrong. (I actually don't think this requirement enables > significantly faster implementation, so I'm tempted to change it to work > correctly with unsorted rows.) >
I do not think its preallocation per se, since 1 proc is fast. I think that your partition of rows fed to the MatCreate() call does not match what you provide to MatSetValues() and thus you do a lot of communication in MatAssemblyEnd(). There are 2 ways to debug this: 1) -log_summary to see where the time is spent 2) MatSetOption(A, *MAT_NEW_NONZERO_LOCATION_ERR)* Matt You can also run with -info |grep malloc, there should be no mallocs in > MatSetValues(). > > > in the first iteration. > > Mat Ap > > > > call MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, istorf_no_ovcell, > & > > istorf_no_ovcell, PETSC_DETERMINE, PETSC_DETERMINE, rowind, > columnind, & > > A, Ap, ierr) > > > > call MatAssemblyBegin(Ap,MAT_FINAL_ASSEMBLY,ierr) > > call MatAssemblyEnd(Ap,MAT_FINAL_ASSEMBLY,ierr) > > This assembly is superfluous (but harmless). > > > Does the communication of MatCreateMPIAIJWithArrays() in parallel > > computation cost a lot? What could be the cause that > > MatCreateMPIAIJWithArrays() so slow in the first iteration? > > There is no significant communication, it has to be preallocation. > > Jed > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20091110/35a51c31/attachment-0001.htm>
