On Tue, Jun 2, 2020 at 4:49 AM Lawrence Mitchell <[email protected]> wrote:
> > On 2 Jun 2020, at 09:35, Matthew Knepley <[email protected]> wrote: > > > > On Tue, Jun 2, 2020 at 4:25 AM Lawrence Mitchell <[email protected]> > wrote: > > Hi Dave, > > > > > On 2 Jun 2020, at 05:43, Dave May <[email protected]> wrote: > > > > > > > > > > > > On Tue 2. Jun 2020 at 03:30, Matthew Knepley <[email protected]> > wrote: > > > On Mon, Jun 1, 2020 at 7:03 PM Danyang Su <[email protected]> > wrote: > > > Thanks Jed for the quick response. Yes I am asking about the > repartitioning of coarse grids in geometric multigrid for unstructured > mesh. I am happy with AMG. Thanks for letting me know. > > > > > > All the pieces are there, we just have not had users asking for this, > and it will take some work to put together. > > > > > > Matt - I created a branch for you and Lawrence last year which added > full support for PLEX within Telescope. This implementation was not a fully > automated algmoeration strategy - it utilized the partition associated with > the DM returned from DMGetCoarseDM. Hence the job of building the > distributed coarse hierarchy was let to the user. > > > > > > I’m pretty sure that code got merged into master as the branch also > contained several bug mixes for Telescope. Or am I mistaken? > > > > I think you're right. I didn't manage to get the redistribution of the > DMPlex object done last summer (it's bubbling up again). > > > > As I see it, for redistributed geometric multigrid on plexes, the > missing piece is a function: > > > > DMPlexRedistributeOntoComm(DM old, MPI_Comm comm, DM *new) > > > > I went down a rabbit hole of trying to do this, since I actually think > this replaced the current interface to DMPlexDistribute, which is > > > > DMPlexDistribute(DM old, PetscInt overlap, PetscSF *pointDistSF, DM *new) > > > > Where the new DM comes out on the same communicator as the old DM, just > with a different partition. > > > > This has lots of follow-on consequences, for example, one can't easily > load on P processes and then compute on Q. > > > > Unfortunately, collectiveness over MPI_Comm(old) is baked into the > redistribution routines everywhere, and I didn't manage to finish things. > > > > Yes, I remember thinking this out. I believe the conclusion was that > redistribution should happen on the large comm, which > > some fraction of processes getting no cells. Then at the end we call one > new function which copies that DM onto the smaller comm. > > I think this kind of works, but I worry that it makes the following > usecase really painful. > > 1. Parallel load on P processes > 2. Redistribute onto Q > P processes > ... Compute > > 3. (Occasionally), checkpoint by mapping onto P processes and dumping > > Perhaps I don't understand your proposal properly, but I think you're > saying that the _input_ DM in redistribute defines the processes that > participate in the redistribution (and the _output_ DM just controls the > size). > > This would then require me in my example to load on Q processes (not P) or > else do some futzing around. > > Or do I have the wrong end of the stick? > > I think I would prefer an interface where I go from > > commOld -> commNew > > with the number of partitions determined by size(commNew) > > Then the restriction you have on the two communicators is one of: > > i. commOld \subseteq commNew > > ii. commOld \supseteq commNew > > I guess if you wanted to be really fancy you could make an > intercommunicator of commOld \cap commNew, but now you have to carry > another communicator around with you for data transfer etc... > > Perhaps internally you do this by making a DM on whichever the smaller of > the two communicators is that is empty on a bunch of ranks and then copying > in (or copying out) appropriately. > > I think a first step here is to break the assumption everywhere in the > redistribution interface that the size of commOld determines the number of > partitions you want, and instead move to using size(commNew). Then for now > we just pass commNew=commOld. > I almost agree. I still think we do not change Distribute(), since it is really convenient, but we do check sizes on input as you say. We either, 1) Copy the DM to a larger comm with empty slots on input or 2) Copy the DM to a smaller comm eliminating empty slots on output depending on P_in < P_out, or the reverse. THanks, Matt > Lawrence > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
