I'm attempting to implement PETSc for the parallel solution of a large sparse
matrix in a computer program I work with. The program is written in Fortran. I
currently have been able to get the program working using PETSc and giving
correct results, but the issue is its performance. It is much slower than
solving the matrix in serial. I seem to have narrowed the major PETSc time sink
down to actually constructing the matrix (getting the matrix terms into the
PETSc matrix). Specifically, it is hanging for a very long time to do this step:
do values_counter = 1, total_values
call
MatSetValues(A,1,coeff_row(values_counter),1,coeff_col(values_counter),coeff(values_counter),INSERT_VALUES,petsc_err)
end do
Building the right-hand-side vector for this matrix is much faster:
call VecSetValues(y,nrows,index_of_values,rhs_values,INSERT_VALUES,petsc_err)
The difference, as I see it, is that I'm sending the matrix coefficients to the
PETSc matrix one element at a time whereas with the RHS vector, I'm sending all
the values in one shot. There are 7 times more elements in the matrix than the
vector, so I get that it will take longer. So I did some timing studies using
MPI_Wtime and putting the matrix values into the PETSc matrix is taking 4,800
times longer than putting the values into the RHS vector.
Then there is the actual assembly of the matrix that you have to do using
MatAssemblyBegin and MatAssemblyEnd. That takes even longer than assigning
values to the matrix. More than twice as long. Actually solving the matrix and
then distributing the solution back to the domains in the program takes about
as much time as the RHS vector assignment step, so that is nearly 5,000 times
faster than putting data into PETSc. Surely there must be something wrong that
I'm doing with the assignment of data into PETSc.
So my question is, is there a more efficient way to build the matrix? I already
have my coefficients stored in AIJ form in the program. I understand that PETSc
uses AIJ. Can't I somehow just send the three AIJ vectors to PETSc in one shot?
Thanks,
Bob
