@+
--
Nicolas
On 31/03/2021 17:51, Matthew Knepley wrote:
On Sat, Mar 27, 2021 at 9:27 AM Nicolas Barral
<[email protected]
<mailto:[email protected]>> wrote:
Hi all,
First, I'm not sure I understand what the overlap parameter in
DMPlexDistributeOverlap does. I tried the following: generate a small
mesh on 1 rank with DMPlexCreateBoxMesh, then distribute it with
DMPlexDistribute. At this point I have two nice partitions, with shared
vertices and no overlapping cells. Then I call DMPlexDistributeOverlap
with the overlap parameter set to 0 or 1, and get the same resulting
plex in both cases. Why is that ?
The overlap parameter says how many cell adjacencies to go out. You
should not get the same
mesh out. We have lots of examples that use this. If you send your small
example, I can probably
tell you what is happening.
Ok so I do have a small example on that and the DMClone thing I set up
to understand! I attach it to the email.
For the overlap, you can change the overlap constant at the top of the
file. With OVERLAP=0 or 1, the distributed overlapping mesh (shown using
-over_dm_view, it's DMover) are the same, and different from the mesh
before distributing the overlap (shown using -distrib_dm_view). For
larger overlap values they're different.
The process is:
1/ create a DM dm on 1 rank
2/ clone dm into dm2
3/ distribute dm
4/ clone dm into dm3
5/ distribute dm overlap
I print all the DMs after each step. dm has a distributed overlap, dm2
is not distributed, dm3 is distributed but without overlap. Since
distribute and distributeOverlap create new DMs, I don't seem have a
problem with the shallow copies.
Second, I'm wondering what would be a good way to handle two overlaps
and associated local vectors. In my adaptation code, the remeshing
library requires a non-overlapping mesh, while the refinement criterion
computation is based on hessian computations, which require a layer of
overlap. What I can do is clone the dm before distributing the overlap,
then manage two independent plex objects with their own local sections
etc. and copy/trim local vectors manually. Is there a more automatic
way
to do this ?
DMClone() is a shallow copy, so that will not work. You would maintain
two different Plexes, overlapping
and non-overlapping, with their own sections and vecs. Are you sure you
need to keep around the non-overlapping one?
Maybe if I understood what operations you want to work, I could say
something more definitive.
I need to be able to pass the non-overlapping mesh to the remesher. I
can either maintain 2 plexes, or trim the overlapping plex when I create
the arrays I give to the remesher. I'm not sure which is the best/worst ?
Thanks
--
Nicolas
Thanks,
Matt
Thanks
--
Nicolas
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which
their experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
static char help[] = "Tests plex distribution and overlaps.\n";
#include <petsc/private/dmpleximpl.h>
#define OVERLAP 1
int main (int argc, char * argv[]) {
DM dm, dm2, dm3, distributedMesh = NULL, odm;
PetscInt dim = 2, faces[2] = {5, 5};
Vec localVec, globalVec;
PetscInt vStart, vEnd, eStart, eEnd, fStart, fEnd, cStart, cEnd;
PetscInt size, overlap;
PetscBool useCone, useClosure;
MPI_Comm comm;
PetscMPIInt numProcs, rank;
PetscErrorCode ierr;
ierr = PetscInitialize(&argc, &argv, NULL, help);if (ierr) return ierr;
comm = PETSC_COMM_WORLD;
MPI_Comm_size(comm, &numProcs);
MPI_Comm_rank(comm, &rank);
DMPlexCreateBoxMesh(comm, dim, PETSC_TRUE, faces, NULL, NULL, NULL, PETSC_TRUE, &dm);
DMSetFromOptions(dm);
PetscObjectSetName((PetscObject) dm, "DMinitial");
DMViewFromOptions(dm, NULL, "-initial_dm_view");
DMClone(dm, &dm2);CHKERRQ(ierr);
PetscObjectSetName((PetscObject) dm2, "DM2initial");
DMViewFromOptions(dm2, NULL, "-initial_dm_view");
ierr = DMPlexDistribute(dm, 0, NULL, &distributedMesh);
if (distributedMesh) {
DMDestroy(&dm);
dm = distributedMesh;
}
ierr = DMPlexDistribute(dm2, 0, NULL, &distributedMesh);
if (distributedMesh) {
DMDestroy(&dm2);
dm2 = distributedMesh;
}
PetscObjectSetName((PetscObject) dm, "DMdistributed");
DMViewFromOptions(dm, NULL, "-distrib_dm_view");
PetscObjectSetName((PetscObject) dm2, "DM2distributed");
DMViewFromOptions(dm2, NULL, "-distrib_dm_view");
DMClone(dm, &dm3);CHKERRQ(ierr);
odm = dm;
DMPlexDistributeOverlap(odm, OVERLAP, NULL, &dm);
if (!dm) {printf("Big problem\n"); dm = odm;}
else {DMDestroy(&odm);}
PetscObjectSetName((PetscObject) dm, "DMover");
DMViewFromOptions(dm, NULL, "-over_dm_view");
PetscObjectSetName((PetscObject) dm2, "DM2over");
DMViewFromOptions(dm2, NULL, "-over_dm_view");
PetscObjectSetName((PetscObject) dm3, "DM3over");
DMViewFromOptions(dm3, NULL, "-over_dm_view");
fflush(stdout);
MPI_Barrier(comm);
DMPlexGetOverlap(dm, &overlap);
printf("DEBUG(%d) dm overlap: %d\n", rank, overlap);
DMPlexGetOverlap(dm2, &overlap);
printf("DEBUG(%d) dm2 overlap: %d\n", rank, overlap);
DMPlexGetOverlap(dm3, &overlap);
printf("DEBUG(%d) dm3 overlap: %d\n", rank, overlap);
DMDestroy(&dm);
DMDestroy(&dm2);
DMDestroy(&dm3);
PetscFinalize();
return 0;
}
