Barry, See attached below. Hong
---------- Forwarded message ---------- From: Hong Zhang <[email protected]> Date: Wed, Jan 12, 2011 at 9:57 PM Subject: Re: [PPPL-SCOREC application project] (fwd) To: fabien delalondre <delalf at scorec.rpi.edu> Cc: Satish Balay <balay at mcs.anl.gov>, Barry Smith <bsmith at mcs.anl.gov>, Fan Zhang <zhangf4 at rpi.edu>, Mark Shephard <shephard at scorec.rpi.edu>, Hong Zhang <HZhang at mcs.anl.gov> fabien: For eqn(0.4), do all diagonal blocks B have same size? If so, you can implement bjacobi+superlu_dist with mpiexec -n <np> ./your_prog -pc_type bjacobi -pc_bjacobi_blocks <nBblocks> -sub_pc_type lu -sub_pc_factor_mat_solver_package superlu_dist When np > nBblocks, subcommunicators are formed by petsc to implement parallel LU within each block, otherwise, sequential LU are applied for each block. Hong >> At application level, the outer KSP is for solving global equation, >> e.g., your Eqn0.4, while block jacobian preconditioner, i.e., >> diag(B's) from Eqn0.4, is applied. MatGetMultiProcBlock() is used >> internally by >> petsc to get preconditioner diag(B's). My understanding is that you >> want apply superlu_dist >> on each of block B (2D solution), and use it as a preconditioner for >> global 3D Eqn0.4. > > That would be our first attempt although we are not yet sure what > solver/precondition methods would best suit the problem we are trying to > solve. > >> >> For this, you need set your own subcommtype >> (see "type == PETSC_SUBCOMM_GENERAL" in ex37.c). >> I can provide an example that apply user's subcommtype to a global KSP >> solver, >> if it is helpful. > > I think I understand but would appreciate if you could provide such an > example. > If I understand correctly the?MatGetMultiProcBlock?generates a bjacobi > submatrix from a larger matrix (that is usually the complete matrix spanning > all processors). That submatrix spans the processors included in the > provided subCommunicator. > Thank you. > Fabien > >> >> Hong >> >> > >> > Fabien >> > >> > On Mon, Jan 10, 2011 at 10:51 PM, Hong Zhang <hzhang at mcs.anl.gov> wrote: >> >> >> >> Dear fabien: >> >> >> >> Yes, petsc-dev now supports block jacobian preconditioner with >> >> each block distributed into multiple processors as I told you in my >> >> previous email (cannot believe my >> >> email was dated 2009!) >> >> >> >> For example, you can run petsc-dev/src/ksp/ksp/examples/tutorials/ex2 >> >> with >> >> mpiexec -n 4 ./ex2 -pc_type bjacobi -pc_bjacobi_blocks 2 -sub_pc_type >> >> lu -sub_pc_factor_mat_solver_package superlu_dist -ksp_view >> >> i.e., with np=4, using block jacobi preconditoner with 2 blocks, each >> >> block running LU using superlu_dist with np=2 processors. >> >> >> >> For your application, I guess you may want use your own matrix and >> >> processor partitioning in stead of petsc default. >> >> petsc-dev/src/ksp/ksp/examples/tests/ex37.c might be relevant. >> >> Let us know what you need from here. >> >> >> >> I'll let other petsc maintainers to answer your question regarding >> >> matrix/mesh partitioning. >> >> >> >> Hong >> >> >> >> > >> >> > As mentioned in a pretty old email, the SCOREC team is helping Steve >> >> > Jardin's team at Princeton Plasma Physics Lab in developing M3DC1 >> >> > code >> >> > to >> >> > support simulation on plasma. As for now, we can build a partitioned >> >> > 3D >> >> > mesh (torus) made of joint 2D planes that are evenly distributed >> >> > around >> >> > the toroidal direction. ?Each 2D plane can be distributed over one or >> >> > several processors. >> >> > >> >> > The 3D matrix we end up with has a periodic tridiagonal block pattern >> >> > where each block represents either a single 2D plane (see pages 2-3 >> >> > of >> >> > the >> >> > attached document from Steve Jardin for the matrix pattern). As for >> >> > now, >> >> > PPPL is testing different combinations to solve (gmres) and >> >> > precondition >> >> > (superlu, ilu, ...) such a the system using both available Petsc and >> >> > Superlu_dist distributions. >> >> > >> >> > Since you mentioned in a previous communication that you were >> >> > starting >> >> > to >> >> > develop methods to support solving such a type of system with several >> >> > processors by plane (several processors by matrix block), we were >> >> > wondering if that functionality is already available in the latest >> >> > petsc >> >> > distribution. In the long run, the scorec team would be interested in >> >> > interacting more closely with the petsc team on a couple of topics >> >> > that >> >> > would potentially include: >> >> > * Increasing PPPL application resolution performance by selecting a >> >> > better >> >> > combination of solver/preconditioner methods (the development of a >> >> > preconditioner might be something we might want to look at). We would >> >> > appreciate your feedback and expertise to work toward the best >> >> > solution >> >> > in >> >> > this regards. >> >> > * Working on understanding how mesh adaptation could be more tightly >> >> > linked to the solver/preconditioner to reduce the required number of >> >> > operations to be carried out. >> >> > >> >> > On a side note, I was wondering if you guys could give me some >> >> > information >> >> > on matrix partitioning. I would like to get the Matrix partitioned in >> >> > the >> >> > same way I already have my mesh partitioned (not a petsc mesh). I was >> >> > wondering if there already exists some API functions that would >> >> > support >> >> > such an operation (assuming I can provide the needed information from >> >> > my >> >> > mesh partitioning). >> >> > >> >> > Best regards, >> >> > >> >> > Fabien Delalondre >> >> > >> >> >> Fabien, >> >> >> >> >> >>> Thank you for your reply. We are currently working on the design of >> >> >>> an >> >> >>> implementation that will support PPPL activities in parallel for >> >> >>> multiple >> >> >>> planes. Obviously, we will need to support multiple parts by plane >> >> >>> (running a block on multiple processors) in the near future >> >> >>> (probably >> >> >>> some >> >> >>> few months from now). >> >> >>> We will then first begin with the one part (one block) by processor >> >> >>> that >> >> >>> will be considered as a specific case of the more general approach >> >> >>> that >> >> >>> will support multiple parts by processors (running a block on >> >> >>> multiple >> >> >>> processors). Having this running within a few months (2-3) will >> >> >>> consequently be advantageous. Do you think this would be a >> >> >>> reasonable >> >> >>> time >> >> >>> frame for you ? >> >> >> >> >> >> Let us know after you get the 1st phase of your work done. >> >> >> If we still have not accomplished our task then, we'll put it >> >> >> at the top of our development and get it done in few days. >> >> >> >> >> >> Sorry for late reply, >> >> >> >> >> >> Hong >> >> >>> >> >> >>>> >> >> >>>> Fabien, >> >> >>>> >> >> >>>> Your matrix has pattern of periodic tridiagonal blocks, >> >> >>>> each block represents a single 2D plane previously >> >> >>>> computed using superlu_dist in parallel. >> >> >>>> >> >> >>>> Our understanding is that you want use >> >> >>>> the diagonal block matrice as a preconditoner. >> >> >>>> We support bjacobi preconditioner, this means, >> >> >>>> if you use a sigle process for each block, >> >> >>>> then current petsc can easily invok pc=jacobi >> >> >>>> and use superlu within the block. >> >> >>>> As block size becomes too large for a single processor, >> >> >>>> then you want run each block on several processes, >> >> >>>> for which we are still developing. >> >> >>>> We can accelerate this development if you've reached >> >> >>>> this stage of computation. >> >> >>>> Let us know your need. >> >> >>>> >> >> >>>> Sorry for late reply. We are moving to a new building ... >> >> >>>> >> >> >>>> Hong >> >> >>>> >> >> >>>>> ---------- Forwarded message ---------- >> >> >>>>> Date: Wed, 9 Sep 2009 23:36:48 -0400 (EDT) >> >> >>>>> From: delalf at scorec.rpi.edu >> >> >>>>> To: hzhang at mcs.anl.gov >> >> >>>>> Cc: shephard at scorec.rpi.edu >> >> >>>>> Subject: [PPPL-SCOREC application project] >> >> >>>>> >> >> >>>>> Dear Hong, >> >> >>>>> >> >> >>>>> My name is Fabien Delalondre. I am currently working at SCOREC >> >> >>>>> with >> >> >>>>> Mark >> >> >>>>> Shephard on the development of parallel adaptive multiscale >> >> >>>>> simulation. >> >> >>>>> >> >> >>>>> I am contacting you regarding one of the projects we are working >> >> >>>>> on >> >> >>>>> that >> >> >>>>> involves the Princeton Plasma Physics lab research team. Mark let >> >> >>>>> me >> >> >>>>> know >> >> >>>>> that you already discussed some aspects of that project. I >> >> >>>>> attached >> >> >>>>> some >> >> >>>>> documents to remind you the problematic we will soon be facing: >> >> >>>>> - PPPL_Jardin-Solvers.pdf: Description of MHD equations the PPPL >> >> >>>>> research >> >> >>>>> team is solving (2D) >> >> >>>>> - 3DExtension.pdf: Description of the 3D extension of MHD >> >> >>>>> equations. >> >> >>>>> - PPPL ??? SCOREC.pdf: Brief description of the matrix pattern >> >> >>>>> that >> >> >>>>> will >> >> >>>>> be >> >> >>>>> obtained using the 3D extension of MHD equations (banded matrix >> >> >>>>> with >> >> >>>>> some >> >> >>>>> extra diagonal coupling terms). >> >> >>>>> >> >> >>>>> Mark let me know that the petsc team might already have developed >> >> >>>>> something for supporting the resolution of algebraic system that >> >> >>>>> includes >> >> >>>>> such matrix pattern. Could you please give me some information >> >> >>>>> regarding >> >> >>>>> this. >> >> >>>>> >> >> >>>>> Regards, >> >> >>>>> >> >> >>>>> Fabien Delalondre<3DExtension.pdf><PPPL ? ? ??? >> >> >>>>> SCOREC.pdf><PPPL_Jardin-Solvers.pdf> >> >> >>> >> >> >>> >> >> >>> >> >> > >> > >> > >> > >> > -- >> > Fabien Delalondre, PhD. >> > Senior Research Associate, Scientific Computation Research Center >> > (SCOREC). >> > Rensselaer Polytechnic Institute (RPI), Troy, NY. >> > Email:?delalf at scorec.rpi.edu, Phone: (518)-276-8045 >> > > > > > -- > Fabien Delalondre, PhD. > Senior Research Associate, Scientific Computation Research Center (SCOREC). > Rensselaer Polytechnic Institute (RPI), Troy, NY. > Email:?delalf at scorec.rpi.edu, Phone: (518)-276-8045 >
