We had this discussion last November. Now that DMPlex has further matured
and is likely to play a big role in some major applications I'd like to revisit
the issue and see if PetscSection can be refactored to be easier to understand
and more useful and more "compatible" with how DMPlex is developing.
PetscSection seems to have two distinct roles:
Role 1 (I'll call this object XX) Provides a way of indexing into an array
of "items" where different items can be of different sizes and there can be
spaces in the memory between items.
---------
struct _n_PetscUniformSection atlasLayout; /* Layout for the atlas */
PetscInt *atlasDof; /* Describes layout of storage,
point --> # of values */
PetscInt *atlasOff; /* Describes layout of storage,
point --> offset into storage */
PetscInt maxDof; /* Maximum dof on any point */
PetscBool setup;
So if we have a regular C array pointer to type int, double etc we do addr =
array + p == &array[p] to get the address of the pth item (where p may only be
valid between pstart and pend.) With PetscSection instead one would do
XXGetOffset(xx,p,&offset); addr = (type*) (array + offset); while
XXGetDOF(xx,p,dof); gives you the "size" of the memory at location addr.
For historical reasons the offsets are in terms of int* instead of char*;
(this should be fixed)
This is a useful thing but in itself not a killer app. What makes it a
killer app are
DMPlexDistributeField(DM dm, PetscSF pointSF, XX originalSection, Vec
originalVec, XX newSection, Vec newVec)
DMPlexDistributeData(DM dm, PetscSF pointSF, XX originalSection, MPI_Datatype
datatype, void *originalData, XX newSection, void **newData)
Note that these do not actually even use the DM (the code should be fixed
by moving it out of DMPlex into the "lower level" code.) Note also these really
create newSection internally (the code should be fixed by making these take
PetscSection *newSection).
What this means is that if I have a set of "points" distributed across MPI
processes and for each point a "bunch of data" (in an "array" offsetted by a
XX) and I redistribute the points across MPI processes including adding ghost
points then I can redistribute all the "bunch of data" (including that needed
for ghost points) trivially and conveniently access it using the newSection.
Role 2 (I'll call this object CC) Provides a container that tells you which
parts of "bunches of data" in an array or Vec are associated with different
"physical simulation fields" and are possibly constrained (for example by
boundary conditions). Uses XX to indicate each field etc.
--------
PetscSection bc; /* Describes constraints, point
--> # local dofs which are constrained */
PetscInt *bcIndices; /* Local indices for constrained
dofs */
PetscInt numFields; /* The number of fields making up
the degrees of freedom */
const char **fieldNames; /* The field names */
PetscInt *numFieldComponents; /* The number of components
in each field */
PetscSection *field; /* A section describing the
layout and constraints for each field */
Suggestion:
We split PetscSection into XX and CC reorganize and clean them up.
Relationship of XX to IS.
An IS defines a subset of entries in a Vec or possibly in an array of
type int. Generally individual entries in an IS do not have particular meaning
relative to other entries. That is one does not iterate over entries in an IS,
one just uses the IS to get a new Vec and then does whatever on that Vec.
An XX is something you often iterate over, for example
for (p = pStart; p < pEnd; ++p) {
PetscInt dof, off, s;
ierr = PetscSectionGetDof(mesh->supportSection, p, &dof);CHKERRQ(ierr);
ierr = PetscSectionGetOffset(mesh->supportSection, p, &off);CHKERRQ(ierr);
do something
that is you get a single entry of XX and do something with it (though you can
also get a bunch of entries and do something on the whole bunch).
Question:
Is XX a powerful and useful thing? Do we have the correct API for it? Can
it be used for other purposes?
Can it be made more powerful: for example the "chunk" that it gives you is
always a contiqous set of bytes in the "array". Would be be better off with a
YY object that allowed them to be non-contiquous? Or should YY be a higher
level object implemented with XX?
Looking for a vigorous discussion and then code that I can better understand.
Barry
On Nov 8, 2012, at 10:18 PM, Jed Brown <[email protected]> wrote:
> This is inevitable, but may still not resolve anything. I'm sure Matt will
> correct whatever below is incorrect. The main question is whether to promote
> PetscSection to a PetscObject or to hide/replace it with something else.
>
> So PetscSection is an unloved (by most) beastie that is sort of intermediate
> between an IS and a DM. Its core functionality is to provide indirect
> indexing with variable block sizes. It consists of two parts, the first of
> which is less than a PetscLayout.
>
> struct _n_PetscUniformSection {
> MPI_Comm comm;
> PetscInt pStart, pEnd; /* The chart: all points are contained in [pStart,
> pEnd) */
> PetscInt numDof; /* Describes layout of storage, point --> (constant
> # of values, (p - pStart)*constant # of values) */
> };
>
> numDof is always 1 and PetscUniformSection is never referenced from a *.c
> file so this is really just a range of valid "point" values [pStart...pEnd).
>
> struct _n_PetscSection {
> struct _n_PetscUniformSection atlasLayout; /* Layout for the atlas */
> PetscInt *atlasDof; /* Describes layout of storage,
> point --> # of values */
> PetscInt *atlasOff; /* Describes layout of storage,
> point --> offset into storage */
>
> ^^ These two fields are the essential part. Each index in [pStart,pEnd) is
> mapped to a block of size atlasDof[p-pStart] located at atlasOff[p-pStart].
> Matt composes these indirect maps for storing meshes and other indexing
> tasks. This "unsorted offset mapping" is indeed used frequently in PETSc,
> though outside of Matt's code, it's just raw arrays.
>
> User's primarily encounter PetscSection when accessing unstructured meshes or
> any time the number of degrees of freedom or locations of each "point"
> (topological entity in the mesh) is not uniform. DMs do a similar thing
> (provide more logical access to a Vec), but DMDAVecGetArray() is of course
> only natural for a structured grid.
>
> At the user interface level, provided the user doesn't need low-level access
> to topology, I think there are three possibilities.
>
> 1. To compute a residual at a cell, user does
>
> VecGetArray(F,&f); // normal access to the Vec
> DMComplexGetHeightStratum(dm,0,&cstart,&cend); // "height 0" is like
> codimension 0 with respect to the maximum topological dimension. Matt and I
> have have a philosophical debate over whether users prefer to think in
> "strata" (that can skip dimensions, so "height 1" could be a face, an edge,
> or a vertex depending on what is stored in the mesh) or in "topological
> dimensions" (where codim=1 is a "face" for a 3D mesh and an edge for a 2D
> mesh).
>
> DMGetDefaultSection(dm,§ion);
> for (c=cstart; c<cend; c++) {
> PetscInt off;
> PetscSectionGetOffset(section,c,&off);
> f[off+0] = residual of first equation at this cell;
> f[off+1] = residual of second equation;
> // brushing under the rug how we access neighbor values
> }
>
> The above is basically the current model. Matt has DMComplexGet/SetClosure(),
> but that interface carries a sleeper reference to the Vec's array and he says
> he plans to remove it.
>
> 2. Add DMComplex-specific access routines so the user does not need to see
> the PetscSection. Presumably this would be something like
> DMComplexGetPointOffset(dm,c,&offset); // offset into owned part of global
> vector?
> DMComplexGetPointOffsetLocal(dm,c,&loffset); // offset into local vector
>
> 3. Access cursors (another object).
> DMComplexGetCursors(dm,Vec U,3,&uface,&ucellL,&ucellR);
> // similar for F
> DMComplexGetHeightStratum(dm,1,&fstart,&fend);
> for (f=fstart; f<fend; f++) {
> const PetscInt *c;
> const PetscScalar *uL,*uR,*uF;
> DMComplexGetSupport(dm,f,&c); // left and right cells
> DMCursorGetValues(uface,f,&uF); // only used for staggered/mixed
> discretization
> DMCursorGetValues(ucellL,c[0],&uL);
> DMCursorGetValues(ucellR,c[1],&uR);
> // Solve Riemann problem with "face" values uF[] (if using a staggered
> discretization), uL[], and uR[].
> DMCursorRestoreValues(uface,f,&uF); // ...
>
> The "advantage" of the cursors here is that DMComplex doesn't need to hold
> the context manager itself (perhaps for multiple threads), the code dealing
> with the number of concurrent accesses is outside the inner loop, and the
> implementations of these accessors are trivial (offset). I think it's still
> not compelling for the sketched discretization above, but consider also FEM
> methods where we need to access all degrees of freedom on the closure of an
> element. The following packs a buffer with the closure (because it's not
> contiguous in memory).
>
> DMCursorGetClosure(cell,c[0],&uc);
>
>
> 4. Alternative suggestions?
>
>
> I'm so sick of the name DMComplex. Let's just rename to DMPlex. It's shorter
> and doesn't have a double-meaning. Or pretty much any other name but
> "Complex".
>
> For completeness, the last bit of _n_PetscSection:
>
> PetscInt maxDof; /* Maximum dof on any point */
> PetscSection bc; /* Describes constraints, point
> --> # local dofs which are constrained */
> PetscInt *bcIndices; /* Local indices for
> constrained dofs */
>
> These BC fields are for doing masked updates, such as VecSetValuesSection()
> which ignores constrained dofs.
>
> PetscInt refcnt; /* Vecs obtained with
> VecDuplicate() and from MatGetVecs() reuse map of input object */
> PetscBool setup;
>
> PetscInt numFields; /* The number of fields making
> up the degrees of freedom */
> const char **fieldNames; /* The field names */
> PetscInt *numFieldComponents; /* The number of
> components in each field */
> PetscSection *field; /* A section describing the
> layout and constraints for each field */
> };
>
