Anders Logg wrote:
>>> I think it looks good.
>>>
>>> As far as I understand, you build a global numbering of all mesh
>>> entities (which may be different from the local numbering on each
>>> processor), and then the (global parallel) local-to-global mapping
>>> follows from tabulate_dofs just as usual.
>>>
>>> So, the difference is that you build a global numbering of the mesh
>>> entities, and we wanted to build a global numbering of the dofs. The
>>> only advantage I can see with our approach is that it may use less
>>> memory, since we don't need to store an extra numbering scheme for all
>>> mesh entities but this not a big deal.
>>>
>>> A few questions:
>>>
>>> 1. Is the above interpretation correct?
>>>
>>>
>> Yes.
>>
>> Another disadvantage with the global numbering scheme is the mesh
>> connectivity calculations (mesh.init in MeshRenumber).
>>
>
> Why is this a problem? As far as I understand, there are always two
> different numberings of mesh entities, one local (same as we have
> now) and one global. The local can be computed as usual and then the
> global can be reconstructed from the local + the overlap.
>
> (overlap = how the local pieces fit together)
>
>
>
Iterating over the local + overlap requires some mesh connectivity,
which are costly to generate.
>> No, the mesh is always distributed.
>>
>
> How does this work? Is the IO also parallel? I can't see any parallel
> stuff in XMLMesh.cpp.
>
>
The mesh file is parsed in parallel, PXMLMesh.cpp. It depends on a
geometric partitioner for an initial distribution of the coordinates,
thus not SCOTCH friendly.
>>> 4. Is the extra storage dynamic? If only vertices are needed (for P1
>>> elements), then we only need to store extra vertex numbers.
>>>
>>>
>> No, it always stores a global number for each mesh entity.
>>
>
> ok, this needs to be fixed but it shouldn't be hard. The DofMap knows
> which entities are needed. This might be an argument in favor of
> computing a parallel dof map from local dof maps rather than computing
> a parallel mesh numbering from local mesh numberings.
>
>
>>> 5. MeshRenumber seems specific to triangles and tets. Can it be done
>>> without reference to specific entities with special cases put in
>>> CellType?
>>>
>>>
>> Maybe, the problem is to construct a key that could be used to uniquely
>> identify the entity.
>>
>
> Isn't (dim, index) enough? Or is this because you only have vertex
> information in your overlap?
>
>
Yes. With more information in the overlap, (dim, index) should be enough.
>> But, since each entity could be seen as a set of vertices it shouldn't
>> be any problem.
>>
>>> 6. Does it work for assembly over interior facets (like in DG
>>> methods)?
>>>
>>>
>> I'm not sure, haven't tried any DG type problems.
>>
>
> I expect it to be problematic. For interior facet integrals, we need
> to iterate over all interior facets in the mesh and for each facet
> access its two adjacent cells. One of these may be on another
> processor.
>
>
But shouldn't a set of ghosted cells for the interior boundary solve
this problem?
>>> 7. Is it possible to make it work with SCOTCH (in addition to
>>> ParMetis)?
>>>
>>>
>> Yes, with some modification to MPIMeshCommunicator.
>>
>
> ok.
>
>
>>> Then some suggestions:
>>>
>>> 1. I'd like to move the implementation of DofMap::build() to
>>> DofMapBuilder (to simplify DofMap.cpp).
>>>
>>> 2. Function names should be fooBar(), not foo_bar().
>>>
>>>
>> I recently found doc/misc/policy :)
>>
>>> 3. There needs to be some #ifdef HAS_PARMETIS so it may be built
>>> without Parmetis.
>>>
>>> 4. I'd like to add a new class GlobalNumbering (to replace
>>> MeshDistributedData) that holds the global numbering scheme.
>>>
>>> The Mesh class can have a pointer to a GlobalNumbering object which is
>>> 0 by default so no extra data (or at least not more than 4 bytes) is
>>> stored when not running in parallel.
>>>
>>> Then we can add a function MeshEntity::number() which returns the same
>>> as index() if GlobalNumbering is 0. Otherwise, it returns what is
>>> stored in GlobalNumbering.
>>>
>>> GlobalNumbering can have an array of MeshFunctions, one for each
>>> topological dimension, that maps the local entity indices to their
>>> global numbers.
>>>
>>> Thus, a MeshEntity will have two functions index() and number().
>>> These will return the same value in sequential and possibly different
>>> values in parallel.
>>>
>>> Let's await some more comments and then get started. It would be nice
>>> to get it in small patches to give us an opportunity to comment/edit.
>>> Adding GlobalNumbering and MeshEntity::number() would be a good start.
>>>
>>>
>> Sounds good, However I think the {T,S,F,V,O} design mentioned by Johan
>> is more efficient.
>>
>
> These are different issues. One issue is how to represent the overlap,
> however we choose to do that, either {T,S,F,V,O} or just {T, S, F}.
> The other issue is how we store either the global mesh numbering or
> the global dof map. We need to discuss both.
>
> I'd like to continue this discussion for some time before we jump on
> to either option.
>
> 1. Do we want to compute a global mesh numbering and then let the
> global dof map follow as usual, or do we want to have just a local
> mesh numbering and then compute a global dof map from the local dof
> maps + the overlap?
>
> In either case, the resulting global dof map can be renumbered to
> reduce bandwidth/communication.
>
> 2. How do we want to represent the overlap? Is it enough to store it
> as {T, S, F} or do we need more? Do we need a class Overlap that
> stores the overlap?
>
> --
> Anders
>
I think the {V,O} information is needed in order to redistribute
(reconstruct with the MeshEditor) a mesh during computation (due to load
balancing). A Global numbering is (probably) also needed for
refinement/coarsening.
Niclas
_______________________________________________
DOLFIN-dev mailing list
[email protected]
http://www.fenics.org/mailman/listinfo/dolfin-dev
