Ok thanks for the clarification. In theory, if before the Reduction back to the original distribution, if I call DMGlobaltoLocal then even with MPI_REPLACE all the leafs corresponding to the original root should have the same value so I won't have an ambiguity, correct?
On Thu, Jan 19, 2023 at 9:28 PM Matthew Knepley <[email protected]> wrote: > On Thu, Jan 19, 2023 at 9:13 PM Nicholas Arnold-Medabalimi < > [email protected]> wrote: > >> Hi Matt >> >> Yep, that makes sense and is consistent. >> >> My question is a little more specific. So let's say I take an >> initial mesh and distribute it and get the distribution SF with an overlap >> of one. Consider a cell that is a root on process 0 and a leaf on process 1 >> after the distribution. >> >> Will the distribution pointSF have an entry for the cell that is a leaf >> in the ghost cell sense? >> >> I guess, in short does the distribution SF only have entries for the >> movement of points that are roots in the ghost SF? >> > > I do not understand the question. Suppose that a certain cell, say 0, in > the original distribution goes to two different processes, say 0 and 1, and > will happen when you distribute with overlap. Then the migration SF has two > leaf entries for that cell, one from process 0 and one from process 1. They > both point to root cell 0 on process 0. > > >> Sorry if this is a little unclear. >> >> Maybe my usage will be a bit clearer. I am generating a distributionSF >> (type 2 in your desc) then using that to generate a dof distribution(type >> 3) using the section information. I then pass the information from the >> initial distribution to new distribution with PetscSFBcast with >> MPI_REPLACE. That scatters the vector to the new distribution. I then do >> "stuff" and now want to redistribute back. So I pass the same dof >> distributionSF but call PetscSFReduce with MPI_REPLACE. My concern is I am >> only setting the root cell values on each partition. So if the ghost cells >> are part of the distribution SF there will be multiple cells reducing to >> the original distribution cell? >> > > Yes, definitely. > > Thanks, > > Matt > > >> Thanks >> Nicholas >> >> >> On Thu, Jan 19, 2023 at 8:28 PM Matthew Knepley <[email protected]> >> wrote: >> >>> On Thu, Jan 19, 2023 at 11:58 AM Nicholas Arnold-Medabalimi < >>> [email protected]> wrote: >>> >>>> Hi Petsc Users >>>> >>>> I'm working with a distribution start forest generated by >>>> DMPlexDistribute and PetscSFBcast and Reduce to move data between the >>>> initial distribution and the distribution generated by DMPlex Distribute. >>>> >>>> I'm trying to debug some values that aren't being copied properly and >>>> wanted to verify I understand how a redistribution SF works compared with a >>>> SF that describes overlapped points. >>>> >>>> [0] 0 <- (0,7) point 0 on the distributed plex is point 7 on >>>> process 0 on the initial distribution >>>> [0] 1 <- (0,8) point 1 on the distributed plex is point 8 on >>>> process 0 on the initial distribution >>>> [0] 2 <- (0,9) >>>> [0] 3 <- (0,10) >>>> [0] 4 <- (0,11) >>>> >>>> [1] 0 <- (1,0) point 0 on the distributed plex is point 0 on >>>> process 1 on the initial distribution >>>> [1] 1 <- (1,1) >>>> [1] 2 <- (1,2) >>>> [1] 3 <- (0,0) point 3 on the distributed plex is point 0 on >>>> process 0 on the initial distribution >>>> [1] 4 <- (0,1) >>>> [1] 5 <- (0,2) >>>> >>>> my confusion I think is how does the distributionSF inform of what >>>> cells will be leafs on the distribution? >>>> >>> >>> I should eventually write something to clarify this. I am using SF in >>> (at least) two different ways. >>> >>> First, there is a familiar SF that we use for dealing with "ghost" >>> points. These are replicated points where one process >>> is said to "own" the point and another process is said to hold a >>> "ghost". The ghost points are leaves in the SF which >>> point back to the root point owned by another process. We call this the >>> pointSF for a DM. >>> >>> Second, we have a migration SF. Here the root points give the original >>> point distribution. The leaf points give the new >>> point distribution. Thus a PetscSFBcast() pushes points from the >>> original to new distribution, which is what we mean >>> by a migration. >>> >>> Third, instead of point values, we might want to communicate fields over >>> those points. For this we make new SFes, >>> where the numbering does not refer to points, but rather to dofs. >>> >>> Does this make sense? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Sincerely >>>> Nicholas >>>> >>>> -- >>>> Nicholas Arnold-Medabalimi >>>> >>>> Ph.D. Candidate >>>> Computational Aeroscience Lab >>>> University of Michigan >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <http://www.cse.buffalo.edu/~knepley/> >>> >> >> >> -- >> Nicholas Arnold-Medabalimi >> >> Ph.D. Candidate >> Computational Aeroscience Lab >> University of Michigan >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> > -- Nicholas Arnold-Medabalimi Ph.D. Candidate Computational Aeroscience Lab University of Michigan
