I have confirmed that I am calling MatSetValues() for local rows only and am only setting each value exactly once.
Because of how the matrix was partitioned for another non-Petsc program, each partition is partitioned (by columns) into 32 blocks (corresponding to the row partitions). I enter the data for each block one row at a time, i.e., for any one SetValues call, the entries are sorted by increasing column index. Does that mean I can use MatrixSetOption(A,MAT_COLUMNS_SORTED). Should that help? P.S.: I tried it, and it still seems to be taking quite a long time. > -----Original Message----- > From: owner-petsc-users at mcs.anl.gov > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Satish Balay > Sent: Friday, October 26, 2007 3:04 PM > To: petsc-users at mcs.anl.gov > Subject: Re: Slow assembly > > > On Fri, 26 Oct 2007, John R. Wicks wrote: > > > I am working on computing PageRank for a web scale graph > which uses a > > square matrix which is 1.2x10^8 dimensional with about 10^9 > entries. > > I have partitioned the matrix for 32 processors myself into my own > > ascii format, and I know the memory allocation, so I: > > > > 1) create the matrix with "A = MatCreateMPIAIJ(*n, *n, *N, > *N, 0, nnz, > > 0, onnz)", > > 2) load the entries by repeatedly calling > > "MatSetValues(A,1,&row,links,cols,vals,INSERT_VALUES)", and > > > > 3) call MatAssemblyBegin/End. > > > > Steps 1 and 2 complete in a couple minutes, but step 3 is taking > > several hours. What is going on? Is there a way to speed > up matrix > > assembly? > > Are you makeing sure that you call MatGetOwnershipRange() - > and calling MatSetValues() for mostly local rows only? > > Also can you confirm that multiple processes [for eg: proc-0 > and proc-1 etc..] are not setting the same value [i.e both > of them calling MatSetValues(row=0,col=0)] > > Satish >
