Certainly larger than anything I've done. I think I did 50 million dofs but that required disabling amr to reduce the memory footprint of the mesh!
What types of elements are you using (geometric and finite element types)? So long as a processor gets all its elements ad every element which share at least one node with a local element I don't see how it could miss... Can you figure out how many elements (local, neighbor, and remote) wind up on each partition for the nemesis and exodus cases? I assume they are the same mesh? The numbers should be the same, but must not be... -Ben On Apr 14, 2011, at 10:14 PM, "Derek Gaston" <[email protected]> wrote: > It appears as though the SparsityPattern is having trouble with ParallelMesh > when reading Nemesis files (mind jog: Nemesis is the parallel Exodus format > where each processor reads just its piece of the mesh). We seem to not be > getting the correct number of nonzeros per row... so that when running a > problem with -info we see things like this: > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 100 > > This doesn't happen when using ParallelMesh to read an Exodus file... only > when starting with a decomposed Nemesis file. > > I've tried several things... like disabling our CouplingMatrix (since that > uses a different path in SparsityPattern::Build) and even disabling AMR > (didn't know if the screwy is_child_on_side() with ParallelMesh was messing > things up... also libMesh currently doesn't compile with AMR disabled! I > have patches that make it work that I will commit soon though!). None of > those things work. > > I'm thinking that it could be possible that it's not picking up dofs from > RemoteElems properly... so it's not getting the right sparsity pattern around > the edges of the parallel pieces of the domain.... but SparsityPattern::Build > is fairly dense and I can't quite see where it would be hosed up... > > BTW: Once we get past the first Jacobian evaluation (it's slow because of the > mallocs) we have fairly large (~300 million DoF) runs going on > multi-thousands of processors (so far up to about 4k... but we're headed to > 10k soon!)! What is the largest solve ever done with libMesh? This > certainly has to be up there... and we're not even to our goal yet! The > scaling is pretty darned good as well (other than the malloc thing). When I > have hard numbers I'll share. > > Derek > ------------------------------------------------------------------------------ > Benefiting from Server Virtualization: Beyond Initial Workload > Consolidation -- Increasing the use of server virtualization is a top > priority.Virtualization can reduce costs, simplify management, and improve > application availability and disaster protection. Learn more about boosting > the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev > _______________________________________________ > Libmesh-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/libmesh-devel ------------------------------------------------------------------------------ Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev _______________________________________________ Libmesh-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/libmesh-devel
