i call ierr = MatStashSetInitialSize(A[*seqsolve],stash_size, stash_size);CHKERRQ(ierr); with 100 000 000 for the stash size to make sure that's not the bottleneck
the assemble time remains unchanged however. nstash in MatAssemblyBegin_MPIAIJ (CPU=0) = 109485 reallocs in MatAssemblyBegin_MPIAIJ = 0 cheers Thomas On Saturday 02 February 2008 23:19, Barry Smith wrote: > The matstash has a concept of preallocation also. During the first > setvalues > it is allocating more and more memory for the stash. In the second > setvalues > the stash is large enough so does not require any addition allocation. > > You can use the option -matstash_initial_size <size> to allocate > enough space > initially so that the first setvalues is also fast. It does not look > like there is a way > coded to get the <size> that you should use. It should be set to the > maximum nonzeros > any process has that belongs to other processes. The stash handling > code is > in src/mat/utils/matstash.c, perhaps you can figure out how to > printout with PetscInfo() > the sizes needed? > > > Barry > > On Feb 2, 2008, at 12:30 PM, Thomas Geenen wrote: > > On Saturday 02 February 2008 18:33, Hong Zhang wrote: > >> On Sat, 2 Feb 2008, Thomas Geenen wrote: > >>> Dear Petsc users, > >>> > >>> I would like to understand what is slowing down the assembly phase > >>> of my > >>> matrix. I create a matrix with MatCreateMPIAIJ i make a rough > >>> guess of > >>> the number of off diagonal entries and then use a conservative > >>> value to > >>> make sure I do not need extra mallocs. (the number of diagonal > >>> entries is > >>> exact) > >>> next i call MatSetValues and MatAssemblyBegin, MatAssemblyEnd. > >>> The first time i call MatSetValues and MatAssemblyBegin, > >>> MatAssemblyEnd it takes about 170 seconds > >>> the second time 0.3 seconds. > >>> I run it on 6 cpu's and I do fill quit a number of row-entries on > >>> the > >>> "wrong" cpu. However thats also the case the second run. I checked > >>> that there are no additional mallocs > >>> MatGetInfo info.mallocs=0 both after MatSetValues and after > >>> MatAssemblyBegin, MatAssemblyEnd. > >> > >> Run your code with the option '-log_summary' and check which function > >> call dominates the execution time. > > > > the time is spend in MatStashScatterGetMesg_Private > > > >>> I run it on 6 cpu's and I do fill quit a number of row-entries on > >>> the > >>> "wrong" cpu. > >> > >> Likely, the communication that sending the entries to the > >> corrected cpu consume the time. Can you fill the entries in the > >> correct cpu? > > > > the second time the entries are filled on the wrong CPU as well. > > i am curious about the difference in time between run 1 and 2. > > > >> Hong > >> > >>> cheers > >>> Thomas
