Aha! Just needed a call to gather_neighboring_elements()! We were apparently running without any valid neighbors at all! Weird that it even worked that way!
Derek On Apr 14, 2011, at 9:28 PM, Kirk, Benjamin (JSC-EG311) wrote: > Certainly larger than anything I've done. I think I did 50 million dofs but > that required disabling amr to reduce the memory footprint of the mesh! > > What types of elements are you using (geometric and finite element types)? So > long as a processor gets all its elements ad every element which share at > least one node with a local element I don't see how it could miss... > > Can you figure out how many elements (local, neighbor, and remote) wind up on > each partition for the nemesis and exodus cases? I assume they are the same > mesh? The numbers should be the same, but must not be... > > -Ben > > > > On Apr 14, 2011, at 10:14 PM, "Derek Gaston" <[email protected]> wrote: > >> It appears as though the SparsityPattern is having trouble with ParallelMesh >> when reading Nemesis files (mind jog: Nemesis is the parallel Exodus format >> where each processor reads just its piece of the mesh). We seem to not be >> getting the correct number of nonzeros per row... so that when running a >> problem with -info we see things like this: >> >> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 100 >> >> This doesn't happen when using ParallelMesh to read an Exodus file... only >> when starting with a decomposed Nemesis file. >> >> I've tried several things... like disabling our CouplingMatrix (since that >> uses a different path in SparsityPattern::Build) and even disabling AMR >> (didn't know if the screwy is_child_on_side() with ParallelMesh was messing >> things up... also libMesh currently doesn't compile with AMR disabled! I >> have patches that make it work that I will commit soon though!). None of >> those things work. >> >> I'm thinking that it could be possible that it's not picking up dofs from >> RemoteElems properly... so it's not getting the right sparsity pattern >> around the edges of the parallel pieces of the domain.... but >> SparsityPattern::Build is fairly dense and I can't quite see where it would >> be hosed up... >> >> BTW: Once we get past the first Jacobian evaluation (it's slow because of >> the mallocs) we have fairly large (~300 million DoF) runs going on >> multi-thousands of processors (so far up to about 4k... but we're headed to >> 10k soon!)! What is the largest solve ever done with libMesh? This >> certainly has to be up there... and we're not even to our goal yet! The >> scaling is pretty darned good as well (other than the malloc thing). When I >> have hard numbers I'll share. >> >> Derek >> ------------------------------------------------------------------------------ >> Benefiting from Server Virtualization: Beyond Initial Workload >> Consolidation -- Increasing the use of server virtualization is a top >> priority.Virtualization can reduce costs, simplify management, and improve >> application availability and disaster protection. Learn more about boosting >> the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev >> _______________________________________________ >> Libmesh-devel mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/libmesh-devel > > ------------------------------------------------------------------------------ > Benefiting from Server Virtualization: Beyond Initial Workload > Consolidation -- Increasing the use of server virtualization is a top > priority.Virtualization can reduce costs, simplify management, and improve > application availability and disaster protection. Learn more about boosting > the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev > _______________________________________________ > Libmesh-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/libmesh-devel ------------------------------------------------------------------------------ Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev _______________________________________________ Libmesh-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/libmesh-devel
