Hi Matt, For option 1), that would be using MPI and not any special functions in PETSc? I ask since I've never done parallel programming before. I tried using OpenMP to parallelize that for loop but it resulted in some errors and I didn't investigate further, but I'm assuming it's because each process wasn't communicating properly with the others during MatSetValues?
Sincerely, Justin On Sun, Apr 27, 2014 at 5:57 AM, Matthew Knepley <[email protected]> wrote: > On Sun, Apr 27, 2014 at 4:14 AM, Justin Dong <[email protected]> wrote: > >> Hi all, >> >> I'm currently coding a finite element problem in PETSc. I'm computing all >> of the matrices by myself and would prefer to do it that way. I want to >> parallelize the assembly of the global matrices. This is a simplified >> version of the assembly routine (pseudocode): >> >> for (k = 0; k < nelements; ++k) >> { >> get_index(k,ie,je); /* ie and je are arrays that indicate where to >> place A_local */ >> compute_local_matrix(A_local,...); >> >> MatSetValues(A_global, nlocal, ie, nlocal, je, A_local, ADD_VALUES); >> } >> >> This is for DG finite elements and I know the matrix easily be assembled >> in parallel. Even on my laptop, this would allow me to significantly speed >> up my code. The only problem is, I'm not sure how to do this in PETSc. I'd >> assume I need to use MatCreateMPIAIJ instead of MatCreateSeqAIJ, but wasn't >> able to find any using tutorials on how I might do this. >> > > 1) If you just split this loop across processes, it would work > immediately. However, that can be non-optimal in terms > of communication. > > 2) PETSc matrices are distributed by contiguous blocks of rows (see manual > section on matrices), so you would like > those row blocks to correspond roughly to your element blocks. > > 3) You will also have to preallocate, but that should be the same as you > do now for the sequential case, except you > check whether a column is inside the diagonal block. > > Thanks, > > Matt > > >> Sincerely, >> Justin >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener >
