Dear Roy, dear Cody, thank you very much for these thorough elaborations. Judging by these I think we are doing the setup correctly. However, it seems our code gets stuck somewhere in the get_neighboring_elements() method in debug mode. It halts and does not seem to advance even with very coarse meshes (9x9). Maybe I can try this manually. I added some more comments/questions below.
> On 12 Jan 2017, at 00:06, Roy Stogner <royst...@ices.utexas.edu> wrote: > > > On Tue, 10 Jan 2017, Barna Becsek wrote: > >> What we DON’T HAVE: >> - A clear understanding of how ghost elements in LibMesh work. > > If you need complicated ghosting (multiple layers of ghost elements, > ghost elements defined by contact problems, ghosted DoFs on distant > elements, etc) then the best writeup of the problem is probably the > one in include/base/ghosting_functor.h > > If you have a simple problem, then "ghost elements" are just > "neighbors of local elements which are not themselves local”. We only need simple ghosting. So if I got it right: One process’s ghost element is its neighbor’s local element. > >> - Knowledge about the difference of ghost and semilocal elements. > > "semilocal" is just shorthand for "either local or ghost" > >> - A clear outline of operations to perform in order to set up such a >> parallel mesh with the availability of the above mentioned functions >> for numbering. > > The operations you listed looked correct at first glance, but if > you're throwing assertion failures then there must be some subtle bug. This only concerns debug mode. Assertions only get tripped when we leave away the get_neighboring_elements(). If we add it, then the code seems to get stuck in there. I could not yet figure out why this happens. > >> A note on this: The above mentioned functions create elements >> across the entire FD grid also such element that span across >> processor boundaries. > > I do not understand this sentence. What I meant by this is actually what you explain in your 1D example. Meaning I cut the mesh such that certain nodes of certain elements are located on different processes than the elements themselves or their remaining nodes. > >> If I read them in like that will Libmesh be able to find out >> about which of them are ghost elements just by knowing that >> their vertices are not all on the same processor (e.g. after >> calling prepare_for_use). > > Some ghost elements *do* have vertices all on the same processor! > > For instance, in 1D, you might have elements and nodes: > > 0 1 2 3 4 > *---*---*---*---* > 0 1 2 3 > > Suppose that elements 0 and 1 belong to processor 0; libMesh will > typically also assign all their nodes, 0, 1, and 2, to processor 0. > The rest of each set belong to processor 1. > > Then, on processor 1, element 1 is a ghost element (because it belongs > to processor 0, but it neighbors element 2 on processor 1 and so must > be retained on processor 1), but both vertices of element 1 belong to > the same processor, processor 0. > > Suppose instead that elements 0 and 2 belong to processor 0, but 1 and > 3 belong to processor 1. Processor 0 might end up with nodes 0, 1, 2, > and 3. In this case, element 1 is a ghost element on processor 0, > even though every one of its nodes is a local node! > >> If this is not the case how exactly would I go about telling >> this kind of information to Libmesh? > > You tell libMesh that an element is local or not local by setting its > processor_id(). > > libMesh will automatically collect ghost elements (in the simple case > of one layer of side neighbors!) if you call the > gather_neighboring_elements() method; however this method is very > general and requires tricky all-to-all communications of data > regarding all elements at processor boundaries; If you can add ghost > elements manually then doing so may be much faster at large scales. > > If you add ghost elements manually then there is a trick that wasn't > in your list of operations above: you need to set their neighbor links > to remote_elem on all sides where their neighbor is a remote (i.e. > neither local nor ghosted) element. gather_neighboring_elements() > does that for you, and everything in prepare_for_use() simply assumes > that remote_elem links are already correct. > --- > Roy ------------------------------------------------------------------------------ Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms. With one year of Intel Parallel Studio XE. Training and support from Colfax. Order your platform today. http://sdm.link/xeonphi _______________________________________________ Libmesh-users mailing list Libmesh-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/libmesh-users
