Jed, Matt, Junchao, Lawrence, I need some expert advice with this https://gitlab.com/petsc/petsc/-/merge_requests/4542 <https://gitlab.com/petsc/petsc/-/merge_requests/4542>. I was able to translate the cython to PETSc C for moving the ghost points to the end in the local section.
But I don't know * the best way to generate the ghosts list needed by VecCreateGhost()? or should I not use VecCreateGhost() * but somehow instead use the existing DMGetSectionSF() to be able to PetscSF between the owned part and the ghosted part of a hybrid vector (that is, don't use VecCreateGhost() but create an equivalent somehow using the sf that already exists?) Thanks Barry > On Oct 28, 2021, at 6:37 PM, Jed Brown <j...@jedbrown.org> wrote: > > Yeah, I think that's about true. Should the interface for this be to set a > flag before creating fields (and perhaps that flag can become true by default > at some point)? > > Barry Smith <bsm...@petsc.dev> writes: > >> Seems like it should be possible to simply "rewrite" the cython code into >> PETSc C. >> >> Barry >> >> >>> On Oct 28, 2021, at 12:36 PM, Matthew Knepley <knep...@gmail.com> wrote: >>> >>> I agree it is. >>> >>> Matt >>> >>> On Thu, Oct 28, 2021 at 11:53 AM Barry Smith <bsm...@petsc.dev >>> <mailto:bsm...@petsc.dev>> wrote: >>> >>> Thanks, sounds like something DMPLEX should be automatically optionally >>> doing directly. >>> >>>> On Oct 28, 2021, at 10:58 AM, Lawrence Mitchell <we...@gmx.li >>>> <mailto:we...@gmx.li>> wrote: >>>> >>>> Hi Barry et al., >>>> >>>>> On 28 Oct 2021, at 15:37, Barry Smith <bsm...@petsc.dev >>>>> <mailto:bsm...@petsc.dev>> wrote: >>>>> >>>>> >>>>> >>>>>> On Oct 28, 2021, at 10:31 AM, Matthew Knepley <knep...@gmail.com >>>>>> <mailto:knep...@gmail.com>> wrote: >>>>>> >>>>>> On Thu, Oct 28, 2021 at 9:37 AM Barry Smith <bsm...@petsc.dev >>>>>> <mailto:bsm...@petsc.dev>> wrote: >>>>>> >>>>>> Matt, >>>>>> >>>>>> How difficult would it be to rework DMPLEX to allow the use of >>>>>> VecGhost? We have performance problems with GPUs with simple DMNETWORK >>>>>> models because the code spends more time uselessly copying the local >>>>>> part of the vector to another vector in global to local and local to >>>>>> global; more than 1/2 the time of the total simulation. >>>>>> >>>>>> Firedrake already does this because they "vec ghost" their vectors by >>>>>> default. Here is what you need: >>>>>> >>>>>> When you create the PetscSection, by default it orders the unknowns >>>>>> according to the default point numbering. This >>>>>> is what causes the ghost unknowns to be mixed in with the local >>>>>> unknowns. However, PetscSection allows you to set >>>>>> a point permutation >>>>>> >>>>>> >>>>>> https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermutation.html >>>>>> >>>>>> <https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermutation.html> >>>>>> >>>>>> This determines the order of dogs by iterating through points in this >>>>>> permutation, and you can put all shared points at the end. >>>>> >>>>> How do I know what are shared points to put at the end? Couldn't DMPLEX >>>>> do this automatically with an option? Where is the Firedrake code that >>>>> does this with DMPLEX so I can see it? >>>> >>>> The DM's "point sf" indicates shared points. To label them, do something >>>> like: >>>> >>>> DMCreateLabel(dm, "ghost_points"); >>>> DMGetPointSF(dm, &pointsf); >>>> PetscSFGetGraph(pointsf, NULL, &nleaves, &ilocal, NULL); >>>> for (PetscInt p = 0; p < nleaves; p++) { >>>> DMSetLabelValue(dm, "ghost_points", p, 1); >>>> } >>>> >>>> Then you do something like (this is more pseudo-codey): >>>> >>>> PetscInt *permutation; >>>> DMPlexGetChart(dm, &pStart, &pEnd); >>>> PetscMalloc1(pEnd - pStart, &permutation); >>>> PetscInt offsets[2] = {0, pEnd - pStart - nleaves}; >>>> DMGetLabel(dm, &ghosts); >>>> DMLabelCreateIndex(ghosts, pStart, pEnd); >>>> for (PetscInt p = pStart, p < pEnd; p++) { >>>> DMLabelHasPoint(ghosts, p, &has); >>>> if (has) { >>>> // this point is ghost point >>>> permutation[offsets[1]++] = p; >>>> } else { >>>> permutation[offsets[0]++] = p; >>>> } >>>> } >>>> DMLabelDestroyIndex(ghosts, pStart, pEnd); >>>> ISCreateGeneral(..., permutation, &isperm); >>>> >>>> // Now whenever your do PetscSectionCreate, do >>>> PetscSectionSetPermutation(..., isperm); >>>> >>>> And now ghost point dofs will appear after local ones. >>>> >>>> Note that this is probably more complicated for multi-field setups, >>>> depending whether you are point major or field major. >>>> >>>> You can see what we actually do (if you like reading Cython) here: >>>> https://github.com/firedrakeproject/firedrake/blob/master/firedrake/cython/dmcommon.pyx#L1734 >>>> >>>> <https://github.com/firedrakeproject/firedrake/blob/master/firedrake/cython/dmcommon.pyx#L1734> >>>> >>>> It's more complicated because here we are doing some additional things: >>>> >>>> 1. We compute an RCM-order traversal for cells with DMPlexGetOrdering; >>>> 2. Rather than ordering all plex points in the permutation, we walk the >>>> cells in the RCM order and then greedily number points in the transitive >>>> closure (so that in a section cell and vertex dofs from the same cell will >>>> be "close" to each other in the final Vec). >>>> >>>> Thanks, >>>> >>>> Lawrence >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
