If matassembly takes a long time - it usually means that lots of values are set on the wrong processor [they get communicated to the correct processor during assembly]. But several hours is a too long for this operation [so I don't understand whats hapenning here]
Can you run the code with the option '-info' - and then send us this log at petsc-maint at mcs.anl.gov Satish On Fri, 26 Oct 2007, John R. Wicks wrote: > I have confirmed that I am calling MatSetValues() for local rows only and am > only setting each value exactly once. > > Because of how the matrix was partitioned for another non-Petsc program, > each partition is partitioned (by columns) into 32 blocks (corresponding to > the row partitions). I enter the data for each block one row at a time, > i.e., for any one SetValues call, the entries are sorted by increasing > column index. Does that mean I can use > MatrixSetOption(A,MAT_COLUMNS_SORTED). Should that help? > > P.S.: I tried it, and it still seems to be taking quite a long time. > > > -----Original Message----- > > From: owner-petsc-users at mcs.anl.gov > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Satish Balay > > Sent: Friday, October 26, 2007 3:04 PM > > To: petsc-users at mcs.anl.gov > > Subject: Re: Slow assembly > > > > > > On Fri, 26 Oct 2007, John R. Wicks wrote: > > > > > I am working on computing PageRank for a web scale graph > > which uses a > > > square matrix which is 1.2x10^8 dimensional with about 10^9 > > entries. > > > I have partitioned the matrix for 32 processors myself into my own > > > ascii format, and I know the memory allocation, so I: > > > > > > 1) create the matrix with "A = MatCreateMPIAIJ(*n, *n, *N, > > *N, 0, nnz, > > > 0, onnz)", > > > 2) load the entries by repeatedly calling > > > "MatSetValues(A,1,&row,links,cols,vals,INSERT_VALUES)", and > > > > > > 3) call MatAssemblyBegin/End. > > > > > > Steps 1 and 2 complete in a couple minutes, but step 3 is taking > > > several hours. What is going on? Is there a way to speed > > up matrix > > > assembly? > > > > Are you makeing sure that you call MatGetOwnershipRange() - > > and calling MatSetValues() for mostly local rows only? > > > > Also can you confirm that multiple processes [for eg: proc-0 > > and proc-1 etc..] are not setting the same value [i.e both > > of them calling MatSetValues(row=0,col=0)] > > > > Satish > > > >
