Hi Manav, Extra communication can be significant. It's been awhile (~2 years) since I've run a large problem at scale on DistributedMesh but at that time it was about 30-40% slower than the equivalent problem on ReplicatedMesh. Thinking back, those numbers seem really bad and I wouldn't expect it to take that much extra time but I'm fairly certain that's where we were at. I'd like to take another crack at timing a large run again soon to see if I was just unlucky or if there were other effects in play. I'd be really curious to see timing results from you if you get it up and running on both Mesh types.
Cody On Mon, Aug 15, 2016 at 12:13 PM Manav Bhatia <bhatiama...@gmail.com> wrote: > Thanks for your comments, Derek. > > We have a few different machines here, all with about 2GB per compute > core. One of the machines has 12 cores per node, and another has 20 cores > per node. > > Why do you say that run time will be saved with ReplicatedMesh? Is this > due to the reduced MPI communication? I don’t see why other operations > would be cheaper. > > -Manav > > On Aug 15, 2016, at 12:22 PM, Derek Gaston <fried...@gmail.com> wrote: > > A little late to the party - but we use all of the options you listed. > However: the one I would recommend is using ReplicatedMesh and just use > less cores per node in your cluster so that it fits. It will save you an > immense amount of development time... and most likely a lot of run time as > well. > > How many cores do you have per node? How much memory per node? > > Derek > > On Thu, Aug 11, 2016 at 5:04 PM Manav Bhatia <bhatiama...@gmail.com> > wrote: > >> Cody, the 2GB was per core (not per node). Sorry for the poor choice of >> terms earlier. >> >> >> >> Maybe I should seek guidance along a different (but related) trajectory... >> >> >> How do folks on this mailing list handle FSI load/displacement mapping >> for large mesh? >> >> I have tried several options at my end, but each one seems to be run into >> a challenge area: >> >> — serial mesh for both fluid and structure: eventually, runs into >> insufficient memory for large models >> >> — parallel mesh with carefully partitioned fluid/structured mesh so that >> adjacent fluid/structural elems live on the same processors: error-prone, >> difficult to manage, inefficient >> >> — parallel mesh with ghosted elements and augmented send list: as >> explained in the previous message, this did not do the trick. >> >> In all of this, I have been depending on the MeshFunction class. >> >> Should I try to move to DTK? If this what other folks use? Any other >> recommendations? >> >> Would greatly appreciate suggestions and guidance here. >> >> Thanks, >> Manav >> >> >> >> > On Aug 11, 2016, at 3:48 PM, Cody Permann <codyperm...@gmail.com> >> wrote: >> > >> > No pointers (no pun intended) but how big is your mesh? Just curious. >> > >> > On Thu, Aug 11, 2016 at 2:47 PM Manav Bhatia <bhatiama...@gmail.com >> <mailto:bhatiama...@gmail.com>> wrote: >> > So, I did manage to get the combination of >> DistributedMesh::add_extra_ghost_elem() and DofMap::augment_send_list() to >> work, but it ended up not giving me the result I was hoping. >> > >> > Basically, my intent is to use MeshFunction to return solution values >> on a specified boundary, and I want this to work on all processors for a >> distributed mesh. >> > >> > I was hoping that if I make the elements at the specified boundary >> available (through ghost elements), along with their their dofs (through >> send list), the point locator would use this info to find the element >> containing a boundary point, and the mesh function would have access to the >> dofs for interpolation. >> > >> > However, the point locator does not seem to work with the ghost >> elements. >> > >> > Is there another way for me to get this desired result? >> > >> > I would appreciate any pointers. >> > >> > Thanks, >> > Manav >> > >> > >> > >> > > On Aug 11, 2016, at 10:16 AM, Roy Stogner <royst...@ices.utexas.edu >> <mailto:royst...@ices.utexas.edu>> wrote: >> > > >> > > >> > > On Thu, 11 Aug 2016, Manav Bhatia wrote: >> > > >> > >> So if I understand correctly, the current implementation of >> > >> DistributedMesh::add_extra_ghost_elem() only provides the G() >> > >> version of your new API >> > > >> > > Not even that. It provides the G() version of the old API - you can >> > > flag current elements that should remain ghosted, when you're ready to >> > > delete remote elements you're responsible for having flagged the extra >> > > ghosted elements first, and IIRC you can't redistribute the mesh after >> > > it's been initally distributed. >> > > >> > >> without an “evaluable” entity? If my interest is in the E() entity, >> > >> would it be best to wait for your API, or is there a way to get it >> > >> to work in current libMesh? >> > > >> > > You can combine the ghost element flagging with calls to >> > > DofMap::augment_send_list() to make sure all dofs on the ghost >> > > elements remain distributed. The new API ought to make that much >> > > easier though. >> > > >> > >> Perhaps follow the implementation in MOOSE? >> > > >> > > That's a good idea. We've also got 2 augment_send_list() uses in the >> > > libMesh examples, IIRC. >> > > --- >> > > Roy >> > >> > >> > >> ------------------------------------------------------------------------------ >> > What NetFlow Analyzer can do for you? Monitors network bandwidth and >> traffic >> > patterns at an interface-level. Reveals which users, apps, and >> protocols are >> > consuming the most bandwidth. Provides multi-vendor support for NetFlow, >> > J-Flow, sFlow and other flows. Make informed decisions using capacity >> > planning reports. http://sdm.link/zohodev2dev < >> http://sdm.link/zohodev2dev> >> > _______________________________________________ >> > Libmesh-users mailing list >> > Libmesh-users@lists.sourceforge.net <mailto: >> Libmesh-users@lists.sourceforge.net> >> > https://lists.sourceforge.net/lists/listinfo/libmesh-users < >> https://lists.sourceforge.net/lists/listinfo/libmesh-users> >> >> >> ------------------------------------------------------------------------------ >> What NetFlow Analyzer can do for you? Monitors network bandwidth and >> traffic >> patterns at an interface-level. Reveals which users, apps, and protocols >> are >> consuming the most bandwidth. Provides multi-vendor support for NetFlow, >> J-Flow, sFlow and other flows. Make informed decisions using capacity >> planning reports. http://sdm.link/zohodev2dev >> _______________________________________________ >> Libmesh-users mailing list >> Libmesh-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/libmesh-users > > > ------------------------------------------------------------------------------ What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic patterns at an interface-level. Reveals which users, apps, and protocols are consuming the most bandwidth. Provides multi-vendor support for NetFlow, J-Flow, sFlow and other flows. Make informed decisions using capacity planning reports. http://sdm.link/zohodev2dev _______________________________________________ Libmesh-users mailing list Libmesh-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/libmesh-users
