Trying to solve many "one-dimensional" problems each in parallel on different subset of ranks will be massive pain to do specifically. I recommend just forming a single matrix for all these systems and solving it with KSPSolve and block Jacobi preconditioning or even a parallel direct solver such as with -pc_type lu -pc_factor_mat_solver_type mumps
Barry Yes, this single system, in a certain ordering is block diagonal (each block being tridiagonal) so contains "independent" subsystems; what I suggest above essentially takes advantage of this structure to be reasonably efficient, yet trivial to code. > On Aug 20, 2021, at 9:14 AM, Matthew Knepley <[email protected]> wrote: > > On Fri, Aug 20, 2021 at 7:53 AM Joauma Marichal <[email protected] > <mailto:[email protected]>> wrote: > Dear Sir or Madam, > > I am looking for advice regarding some of PETSc functionnalities. I am > currently using PETSc to solve the Navier-Stokes equations on a 3D mesh > decomposed over several processors. However, until now, the processors are > distributed along the x and z directions but not along the y one. Indeed, at > some point in the algorithm, I must solve a tridiagonal system that depends > only on y. Until now, I have therefore performed something like this: > for(int k = cornp->zs, k<cornp->zs+cornp->zm; ++k){ > for(int i = cornp->xs, i<cornp->xs+cornp->xm; ++i){ > Create and solve a tridiagonal system for all the y coordinates > (which are on the same process) > } > However, I would like to decompose my mesh in the y direction (as this should > improve the code efficiency). > I managed to do so by creating a system based on the 3D DM of all my case (so > 1 system of size x*y*z). Unfortunately, this does not seem to be very > efficient. > Do you have some advice on how to cut in the y direction while still being > able to solve x*z systems of size y? Should I create 1D DMs? > > 1) Are you using a 3D DMDA? > > 2) Is the coupling much different in the x and z than in the y direction? > > Thanks, > > Matt > > Thanks a lot for your help. > > Best regards, > > Joauma Marichal > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
