On Mon, Jul 25, 2016 at 1:34 PM, Manav Bhatia <[email protected]> wrote:
> Thanks for your comments, Matt. > > I have a fluid-structural application with a really large fluid > discretization and a really small structural discretization. Due to the > relative difference in size, I have defined the structural system on only a > single node, and the fluid system on (say) N nodes. > > So far, I have hand-coded a Schur-Complement for a frequency-domain > analysis that is able to handle the difference in comms. > > I am attempting to migrate to the nested matrix constructs for some future > work, and was looking at the possibility of reusing the same distribution > of comms. Additionally, I am looking to add additional disciplines and was > considering the possibility of defining the systems on different comms. > > I wasn’t sure if I was creating more problems with this approach than what > I was trying to solve. > > Would you recommend that all objects exist on a global_comm so that there > is no confusion about these operations? > Yes. I think the confusion here is between the problem you are trying to solve, and the tool for doing it. Disparate size of subsystems seems to me to be a _load balancing_ problem. Here you can use data layout to alleviate this. On the global comm, you can put all the fluid unknowns on ranks 0..N-2, and the structural unknowns on N-1. You can have more general splits than that. IF for some reason in the structural assembly you used a large number of collective operations (like say did artificial timestepping to get to some steady state property), then it might make sense to pull out a subcomm of only the occupied ranks, but only above 1000 procs, and only on a non-BlueGene machine. This is also easily measure before you do this work. Matt > Thanks, > Manav > > > > On Jul 25, 2016, at 3:21 PM, Matthew Knepley <[email protected]> wrote: > > On Mon, Jul 25, 2016 at 1:13 PM, Manav Bhatia <[email protected]> > wrote: > >> Hi, >> >> I have a multi physics application with discipline1 defined on comm1 >> and discipline2 on comm2. >> >> My intent is to use the nested matrix for the KSP solver where each >> diagonal block is provided by the disciplines, and the off-diagonal blocks >> are defined as shell-matrices with matrix vector products. >> >> I am a bit unclear about how to deal with the case of different set >> of processors on comm1 and comm2. I have the following questions and would >> appreciate some guidance: >> >> — Would it make sense to define a comm_global as a union of comm1 and >> comm2 for the MatCreateNest? >> >> — The diagonal blocks are available on comm1 and comm2 only. Should >> MatAssemblyBegin/End for these diagonal blocks be called on comm1 and comm2 >> separately? >> >> — What comm should be used for the off-diagonal shell matrices? >> >> — Likewise, when calling VecGetSubVector and VecRestoreSubVector to get >> sub-vectors corresponding to discipline1 (or 2), what comm should these >> function calls be made? >> > > I would first ask if you have a convincing reason for doing this, because > it sounds like the genesis of a million programming errors. > > All the linear algebra objects would have to be in a global comm that > contained any subcomms you want to use. I don't > think it would make sense to define submatrices on subcomms. You can have > your assembly code run on a subcomm certainly, > but again this is a tricky business and I find it hard to understand the > gain. > > Matt > > >> Thanks, >> Manav >> > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
