Anders Logg wrote:
> On Wed, Aug 20, 2008 at 09:39:30AM +0200, Niclas Jansson wrote:
>> Anders Logg wrote:
>>> On Mon, Aug 18, 2008 at 11:05:30AM +0200, Niclas Jansson wrote:
>>>
>>>> Anders Logg wrote:
>>>>
>>>>>>> I think it looks good.
>>>>>>>
>>>>>>> As far as I understand, you build a global numbering of all mesh
>>>>>>> entities (which may be different from the local numbering on each
>>>>>>> processor), and then the (global parallel) local-to-global mapping
>>>>>>> follows from tabulate_dofs just as usual.
>>>>>>>
>>>>>>> So, the difference is that you build a global numbering of the mesh
>>>>>>> entities, and we wanted to build a global numbering of the dofs. The
>>>>>>> only advantage I can see with our approach is that it may use less
>>>>>>> memory, since we don't need to store an extra numbering scheme for all
>>>>>>> mesh entities but this not a big deal.
>>>>>>>
>>>>>>> A few questions:
>>>>>>>
>>>>>>> 1. Is the above interpretation correct?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Yes.
>>>>>>
>>>>>> Another disadvantage with the global numbering scheme is the mesh
>>>>>> connectivity calculations (mesh.init in MeshRenumber).
>>>>>>
>>>>>>
>>>>> Why is this a problem? As far as I understand, there are always two
>>>>> different numberings of mesh entities, one local (same as we have
>>>>> now) and one global. The local can be computed as usual and then the
>>>>> global can be reconstructed from the local + the overlap.
>>>>>
>>>>> (overlap = how the local pieces fit together)
>>>>>
>>>>>
>>>>>
>>>>>
>>>> Iterating over the local + overlap requires some mesh connectivity,
>>>> which are costly to generate.
>>>>
>>> What's your point? Are you arguing against a global numbering scheme
>>> for the mesh entities? I thought this is what you have implemented.
>>>
>>>
>> I'm not sure if the global numbering scheme is the best approach. It
>> worked well for simple dof_maps / elements, with a low renumbering time.
>> But for a more general implementation, renumbering starts to take too
>> much time.
>
> ok. So do you suggest we implement the other strategy instead, building a
> global dof map from local dof maps?
Yes, it's probably more efficient. The only problem with algorithm 5 (in
my opinion) is the communication pattern in stage 0 and Stage 2.
Parallel efficiency in stage 0 would probably be low due to the pipeline
styled offset calculation, it should be easy to fix with MPI_(Ex)Scan.
Stage 2 seems to involve a lot of communication, with small messages.
I think it would be more efficient if the stage were reorganized such
that all messages could be exchanged "at once", in a couple of larger
messages.
Niclas
>
>>>>>>> 6. Does it work for assembly over interior facets (like in DG
>>>>>>> methods)?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> I'm not sure, haven't tried any DG type problems.
>>>>>>
>>>>>>
>>>>> I expect it to be problematic. For interior facet integrals, we need
>>>>> to iterate over all interior facets in the mesh and for each facet
>>>>> access its two adjacent cells. One of these may be on another
>>>>> processor.
>>>>>
>>>>>
>>>>>
>>>> But shouldn't a set of ghosted cells for the interior boundary solve
>>>> this problem?
>>>>
>>> Do you mean an overlap consisting of cells, not facets?
>>>
>>>
>> Yes, cells. Otherwise, how should a processor access a cell on a
>> adjacent processor?
>
> ok, good.
>
>>>>>>> Then some suggestions:
>>>>>>>
>>>>>>> 1. I'd like to move the implementation of DofMap::build() to
>>>>>>> DofMapBuilder (to simplify DofMap.cpp).
>>>>>>>
>>>>>>> 2. Function names should be fooBar(), not foo_bar().
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> I recently found doc/misc/policy :)
>>>>>>
>>>>>>
>>>>>>> 3. There needs to be some #ifdef HAS_PARMETIS so it may be built
>>>>>>> without Parmetis.
>>>>>>>
>>>>>>> 4. I'd like to add a new class GlobalNumbering (to replace
>>>>>>> MeshDistributedData) that holds the global numbering scheme.
>>>>>>>
>>>>>>> The Mesh class can have a pointer to a GlobalNumbering object which is
>>>>>>> 0 by default so no extra data (or at least not more than 4 bytes) is
>>>>>>> stored when not running in parallel.
>>>>>>>
>>>>>>> Then we can add a function MeshEntity::number() which returns the same
>>>>>>> as index() if GlobalNumbering is 0. Otherwise, it returns what is
>>>>>>> stored in GlobalNumbering.
>>>>>>>
>>>>>>> GlobalNumbering can have an array of MeshFunctions, one for each
>>>>>>> topological dimension, that maps the local entity indices to their
>>>>>>> global numbers.
>>>>>>>
>>>>>>> Thus, a MeshEntity will have two functions index() and number().
>>>>>>> These will return the same value in sequential and possibly different
>>>>>>> values in parallel.
>>>>>>>
>>>>>>> Let's await some more comments and then get started. It would be nice
>>>>>>> to get it in small patches to give us an opportunity to comment/edit.
>>>>>>> Adding GlobalNumbering and MeshEntity::number() would be a good start.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Sounds good, However I think the {T,S,F,V,O} design mentioned by Johan
>>>>>> is more efficient.
>>>>>>
>>>>>>
>>>>> These are different issues. One issue is how to represent the overlap,
>>>>> however we choose to do that, either {T,S,F,V,O} or just {T, S, F}.
>>>>> The other issue is how we store either the global mesh numbering or
>>>>> the global dof map. We need to discuss both.
>>>>>
>>>>> I'd like to continue this discussion for some time before we jump on
>>>>> to either option.
>>>>>
>>>>> 1. Do we want to compute a global mesh numbering and then let the
>>>>> global dof map follow as usual, or do we want to have just a local
>>>>> mesh numbering and then compute a global dof map from the local dof
>>>>> maps + the overlap?
>>>>>
>>>>> In either case, the resulting global dof map can be renumbered to
>>>>> reduce bandwidth/communication.
>>>>>
>>>>> 2. How do we want to represent the overlap? Is it enough to store it
>>>>> as {T, S, F} or do we need more? Do we need a class Overlap that
>>>>> stores the overlap?
>>>>>
>>>>>
>>>> I think the {V,O} information is needed in order to redistribute
>>>> (reconstruct with the MeshEditor) a mesh during computation (due to load
>>>> balancing). A Global numbering is (probably) also needed for
>>>> refinement/coarsening.
>>>>
>>> Again, aren't these separate issues? One issue is how we compute the
>>> global dof map (which is needed for assembly). The other is how we
>>> store information about the overlap which we need both for computing a
>>> global numbering or global dof map, and for load balancing,
>>> refinement, coarsening etc.
>>>
>>> Other opinions?
>>>
>>> Do we want to (i) build a global numbering of mesh entities or (ii)
>>> have local numbering of mesh entities and build a global dof map?
>>>
>> True, they are different issues, but I think the choice of dof map
>> method should consider how expensive the overlap information are to
>> compute, and if it could be done with an already distributed mesh.
>
> I'm not very particular about how we represent/store the overlap, but
> let's solve (1) first.
>
> --
> Anders
>
_______________________________________________
DOLFIN-dev mailing list
[email protected]
http://www.fenics.org/mailman/listinfo/dolfin-dev
