On Thu, 25 Jan 2018, David Knezevic wrote:
Thanks for that info. I'm only using ReplicatedMesh for now. Regarding this:
IIRC: on ReplicatedMesh we try to avoid communication as much as
possible, so you have to add user constraints identically on
every processor. There might be ways to get around that (add local
constraints only, then re-add after any repartitioning?) but
I definitely wouldn't suggest trying to depend on them.
hmm, the reason I asked about this was because I had a case that hit
a "cyclic loop" assertion in parallel when I added the constraints
identically on every processor,
and then when I changed it so that I only added the constraint on
the processor that owned the constrained dof, it worked fine. I
assumed that I must have been violating an assumption by adding the
constraints identically on all processors, but your comment would
indicate that isn't the case...
Looking back at dof_map_constraints.C, it looks like my comment was
wrong - I did parallelize the constraint calculations, even on
ReplicatedMesh. We're looping only over local elements when we
compute hanging node and Dirichlet constraints. That means we *must*
communicate (e.g. to handle cases where a local constraint only gets
computed on a ghost element) later.
I'd like to understand this better. Can you point me to the relevant
code in the library? I've been looking through
allgather_recursive_constraints and similar, and I figured they send
and receive constraint rows as needed, which is why I assumed we
didn't need to add constraint rows identically on all processors.
The first section of allgather_recursive_constraints() is what's
relevant here - it does the initial push of constraints that we've
calculated to their owners.
And based on that code, it looks like the rule is "we don't need to
add constraint rows on all processors, but if we *do* then we need to
add them identically".
It might be worth adding some error checking there to make sure that
happens. Right now if a processor has a constrained row, and gets
pushed a constraint for that row, it assumes the remote constraint is
equivalent to the local and it ignores it. We should probably
libmesh_assert equivalency (within TOLERANCE*TOLERANCE ought to be
safe; these are nondimensional numbers) instead.
---
Roy
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Libmesh-users mailing list
Libmesh-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libmesh-users
