What happened was that I used MatDiagonalSet and then used MatSetValues and MatAssemblyEnd, but the log shows that MatAssemblyEnd was called twice; MatDiagonal must force an implicit MatAssemblyBegin/End. By avoiding the call to MatDiagonalSet, everything worked fine (in particular, that shows that my preallocations were correct, as was).
> -----Original Message----- > From: owner-petsc-users at mcs.anl.gov > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Matthew Knepley > Sent: Sunday, October 28, 2007 4:29 PM > To: petsc-users at mcs.anl.gov > Subject: Re: Slow assembly > > > Perhaps this is a miscalculation for parallel allocation, > meaning you allocate the correct number of values, but do > not divide them correctly between the diagonal and offdiagonal parts. > > Matt > > On 10/28/07, John R. Wicks <jwicks at cs.brown.edu> wrote: > > I ran on a smaller example with the -info switch set. I > preallocated > > all my matrices (there are two of them, one sequential and one > > distributed) so that there should be no allocs needed, but > I notice it > > reports: something like: [3] MatAssemblyEnd_SeqAIJ(): > Matrix size: 11 > > X 11; storage space: 10 unneeded,11 used [3] > MatAssemblyEnd_SeqAIJ(): > > Number of mallocs during MatSetValues() is 0 [3] > > MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] > > MatAssemblyEnd_SeqAIJ(): Matrix size: 10 X 10; storage space: 13 > > unneeded,10 used [1] MatAssemblyEnd_SeqAIJ(): Number of > mallocs during > > MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum > nonzeros in > > any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 11 X 11; > > storage space: 22 unneeded,11 used > > [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during > MatSetValues() is 0 > > [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 10 X 10; storage space: 9 > > unneeded,10 used > > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during > MatSetValues() is 0 > > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 9 X 9; storage space: 13 > > unneeded,9 used > > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during > MatSetValues() is 0 > > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 11 X 11; storage space: 19 > > unneeded,11 used > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during > MatSetValues() is 0 > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > > [6] MatAssemblyEnd_SeqAIJ(): Matrix size: 10 X 10; storage space: 17 > > unneeded,10 used > > [6] MatAssemblyEnd_SeqAIJ(): Number of mallocs during > MatSetValues() is 0 > > [6] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > > [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 11 X 11; storage > space: 115 > > unneeded,31 used > > [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during > MatSetValues() is 9 > > [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 5 > > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 10 X 10; storage space: 67 > > unneeded,18 used > > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during > MatSetValues() is 5 > > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3 > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 11 X 11; storage space: 88 > > unneeded,28 used > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during > MatSetValues() is 7 > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 5 > > > > I can't figure why it should allocate anything, b/c I've > precomputed > > the number of entries in each row. > > > > > -----Original Message----- > > > From: owner-petsc-users at mcs.anl.gov > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry Smith > > > Sent: Friday, October 26, 2007 7:54 PM > > > To: petsc-users at mcs.anl.gov > > > Subject: RE: Slow assembly > > > > > > > > > > > > The sorting does not matter. > > > > > > Under normal conditions the MatAssembly should take a > fraction of > > > a second. The only cause that we know that slows it down to the > > > extreme you have is that it is sending a huge amount of > data across > > > processes (the -info option Satish suggested will tell us > if that is > > > true). > > > > > > Are you only call MatAssemblyBegin/End() once? You > should, don't > > > call it multiple times. > > > > > > The sorting is not important (in fact it takes advantage of it > > > automatically and does not need to be set). > > > > > > Barry > > > > > > > > > On Fri, 26 Oct 2007, John R. Wicks wrote: > > > > > > > I have confirmed that I am calling MatSetValues() for local > > > rows only > > > > and am only setting each value exactly once. > > > > > > > > Because of how the matrix was partitioned for another non-Petsc > > > > program, each partition is partitioned (by columns) > into 32 blocks > > > > (corresponding to the row partitions). I enter the > data for each > > > > block one row at a time, i.e., for any one SetValues call, > > > the entries > > > > are sorted by increasing column index. Does that mean > I can use > > > > MatrixSetOption(A,MAT_COLUMNS_SORTED). Should that help? > > > > > > > > P.S.: I tried it, and it still seems to be taking quite a long > > > > time. > > > > > > > > > -----Original Message----- > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of > Satish Balay > > > > > Sent: Friday, October 26, 2007 3:04 PM > > > > > To: petsc-users at mcs.anl.gov > > > > > Subject: Re: Slow assembly > > > > > > > > > > > > > > > On Fri, 26 Oct 2007, John R. Wicks wrote: > > > > > > > > > > > I am working on computing PageRank for a web scale graph > > > > > which uses a > > > > > > square matrix which is 1.2x10^8 dimensional with about 10^9 > > > > > entries. > > > > > > I have partitioned the matrix for 32 processors myself > > > into my own > > > > > > ascii format, and I know the memory allocation, so I: > > > > > > > > > > > > 1) create the matrix with "A = MatCreateMPIAIJ(*n, *n, *N, > > > > > *N, 0, nnz, > > > > > > 0, onnz)", > > > > > > 2) load the entries by repeatedly calling > > > > > > "MatSetValues(A,1,&row,links,cols,vals,INSERT_VALUES)", and > > > > > > > > > > > > 3) call MatAssemblyBegin/End. > > > > > > > > > > > > Steps 1 and 2 complete in a couple minutes, but step 3 is > > > > > > taking several hours. What is going on? Is there a way to > > > > > > speed > > > > > up matrix > > > > > > assembly? > > > > > > > > > > Are you makeing sure that you call > MatGetOwnershipRange() - and > > > > > calling MatSetValues() for mostly local rows only? > > > > > > > > > > Also can you confirm that multiple processes [for eg: > proc-0 and > > > > > proc-1 etc..] are not setting the same value [i.e > both of them > > > > > calling MatSetValues(row=0,col=0)] > > > > > > > > > > Satish > > > > > > > > > > > > > > > > > > > > > > > -- > What most experimenters take for granted before they begin > their experiments is infinitely more interesting than any > results to which their experiments lead. > -- Norbert Wiener >
