It appears as though the SparsityPattern is having trouble with ParallelMesh when reading Nemesis files (mind jog: Nemesis is the parallel Exodus format where each processor reads just its piece of the mesh). We seem to not be getting the correct number of nonzeros per row... so that when running a problem with -info we see things like this:
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 100 This doesn't happen when using ParallelMesh to read an Exodus file... only when starting with a decomposed Nemesis file. I've tried several things... like disabling our CouplingMatrix (since that uses a different path in SparsityPattern::Build) and even disabling AMR (didn't know if the screwy is_child_on_side() with ParallelMesh was messing things up... also libMesh currently doesn't compile with AMR disabled! I have patches that make it work that I will commit soon though!). None of those things work. I'm thinking that it could be possible that it's not picking up dofs from RemoteElems properly... so it's not getting the right sparsity pattern around the edges of the parallel pieces of the domain.... but SparsityPattern::Build is fairly dense and I can't quite see where it would be hosed up... BTW: Once we get past the first Jacobian evaluation (it's slow because of the mallocs) we have fairly large (~300 million DoF) runs going on multi-thousands of processors (so far up to about 4k... but we're headed to 10k soon!)! What is the largest solve ever done with libMesh? This certainly has to be up there... and we're not even to our goal yet! The scaling is pretty darned good as well (other than the malloc thing). When I have hard numbers I'll share. Derek ------------------------------------------------------------------------------ Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev _______________________________________________ Libmesh-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/libmesh-devel
