Thanks for your comments, Matt. I have a fluid-structural application with a really large fluid discretization and a really small structural discretization. Due to the relative difference in size, I have defined the structural system on only a single node, and the fluid system on (say) N nodes.
So far, I have hand-coded a Schur-Complement for a frequency-domain analysis that is able to handle the difference in comms. I am attempting to migrate to the nested matrix constructs for some future work, and was looking at the possibility of reusing the same distribution of comms. Additionally, I am looking to add additional disciplines and was considering the possibility of defining the systems on different comms. I wasn’t sure if I was creating more problems with this approach than what I was trying to solve. Would you recommend that all objects exist on a global_comm so that there is no confusion about these operations? Thanks, Manav > On Jul 25, 2016, at 3:21 PM, Matthew Knepley <[email protected]> wrote: > > On Mon, Jul 25, 2016 at 1:13 PM, Manav Bhatia <[email protected] > <mailto:[email protected]>> wrote: > Hi, > > I have a multi physics application with discipline1 defined on comm1 and > discipline2 on comm2. > > My intent is to use the nested matrix for the KSP solver where each > diagonal block is provided by the disciplines, and the off-diagonal blocks > are defined as shell-matrices with matrix vector products. > > I am a bit unclear about how to deal with the case of different set of > processors on comm1 and comm2. I have the following questions and would > appreciate some guidance: > > — Would it make sense to define a comm_global as a union of comm1 and comm2 > for the MatCreateNest? > > — The diagonal blocks are available on comm1 and comm2 only. Should > MatAssemblyBegin/End for these diagonal blocks be called on comm1 and comm2 > separately? > > — What comm should be used for the off-diagonal shell matrices? > > — Likewise, when calling VecGetSubVector and VecRestoreSubVector to get > sub-vectors corresponding to discipline1 (or 2), what comm should these > function calls be made? > > I would first ask if you have a convincing reason for doing this, because it > sounds like the genesis of a million programming errors. > > All the linear algebra objects would have to be in a global comm that > contained any subcomms you want to use. I don't > think it would make sense to define submatrices on subcomms. You can have > your assembly code run on a subcomm certainly, > but again this is a tricky business and I find it hard to understand the gain. > > Matt > > Thanks, > Manav > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener
