On Mar 1, 2011, at 4:27 PM, Roy Stogner wrote: > > Take a look at revision 4214 to src/mesh/mesh_refinement.C > > It's the "implicit PointLocator construction fails" bug again, but > this time with no threads involved. If a problem with periodic > boundaries was run under certain conditions and on the right > partitioning, then some of the MPI ranks (those with partitions > falling on a periodic boundary) would need to construct/update a > PointLocator to do searches across the boundary, while other ranks > would not. PointLocatorTree's constructor needs to run on all ranks > to build a bounding box, and so fails. > > The fix is the same, but I'm not sure how to make sure we catch this > problem in the future. That "libmesh_assert(!Threads::in_threads)" > call is a good safeguard for the "in threaded code" case, but what > about the "in an ordinary local_elements loop" case? > > The library is fixed for now, but it's an easy mistake to repeat. I'm > going to Cc: Derek since his people are probably the most likely to > get bitten by this in user code too. > > The problem: > > point_locator(), and topological_neighbor(), and perhaps an expanding > range of future functions, should not be used inside element loops > unless the loop is preceded by an explicit PointLocator construction. >
Yes - I did in fact run into this in our code. We were puzzled why we needed that extra call in there but it seemed to be necessary for correct functionality. I'm glad you nailed down the problem - I wasn't able to find it when I originally worked on this awhile back. Actually that leads me to the next problem we've encountered. I haven't written the mailing list yet 'cause I don't know yet how to even pose the problem. I'll give it a shot just so you are aware. I hope to dig into this perhaps with John or somebody else's help soon. When using Periodic Boundaries we've noticed that occasionally under the right conditions as we get lots of activity near the nodes that are double constrained, it appears that "sometimes" during the mesh coarsening phase some of the values at those nodes appear to get "bad" values. In 2D, I'm talking about the four corners of a square domain where periodic boundaries are applied at both sets of edges. In 3D I'm talking about the edges (lines) of the cube where periodic boundaries are applied on all 3 sets of sides. I did build a problem in MOOSE that demonstrates the problem. It's actually an adaptation of the existing periodic boundaries example but I moved the forcing function around the domain and approached one of the edges. Some of the nodes eventually get the wrong values. I know that's probably a poor description and probably has more to do with constraints then the mesh adaptivity but it's a bit of a show stopper with some of our advanced Mesoscale runs at this point - well we can always run with adaptivity off but we need a lot of elements in that case. If you even have an idea of where to start looking - I'm happy to hear your thoughts. If I can get an example worked up in libMesh to show the problem - I'll do that as well. > The only solutions I can think of: > > 1. The default: we accept that code with this problem may work fine on > N processors but then die horribly on N+1. > > 2. We could create a convoluted functor-class-based paradigm similar > to Threads::/TBB to enable a similar test... but if we go through that > much trouble we might as well just make every single local_elements > loop multithreaded. > > 3. We could wait for the neat lambda function support in C++0x... or > was that C++1x? > > 4. We always generate a PointLocatorTree when preparing an updated > Mesh. This is too expensive for users who don't need the tree, and it > won't fix things when (as was just the case) the problem is in a class > like MeshRefinement that's in the middle of the Mesh-updating process. > --- > Roy > Thanks, Cody > ------------------------------------------------------------------------------ > Free Software Download: Index, Search & Analyze Logs and other IT data in > Real-Time with Splunk. Collect, index and harness all the fast moving IT data > generated by your applications, servers and devices whether physical, virtual > or in the cloud. Deliver compliance at lower cost and gain new business > insights. http://p.sf.net/sfu/splunk-dev2dev > _______________________________________________ > Libmesh-devel mailing list > Libmesh-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/libmesh-devel ------------------------------------------------------------------------------ Free Software Download: Index, Search & Analyze Logs and other IT data in Real-Time with Splunk. Collect, index and harness all the fast moving IT data generated by your applications, servers and devices whether physical, virtual or in the cloud. Deliver compliance at lower cost and gain new business insights. http://p.sf.net/sfu/splunk-dev2dev _______________________________________________ Libmesh-devel mailing list Libmesh-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/libmesh-devel
