Ok - a bit more info... I can confirm that it is computing _0_ for the number of off-processor nonzeros (n_oz) for every dof....
Derek On Apr 14, 2011, at 9:13 PM, Derek Gaston wrote: > It appears as though the SparsityPattern is having trouble with ParallelMesh > when reading Nemesis files (mind jog: Nemesis is the parallel Exodus format > where each processor reads just its piece of the mesh). We seem to not be > getting the correct number of nonzeros per row... so that when running a > problem with -info we see things like this: > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 100 > > This doesn't happen when using ParallelMesh to read an Exodus file... only > when starting with a decomposed Nemesis file. > > I've tried several things... like disabling our CouplingMatrix (since that > uses a different path in SparsityPattern::Build) and even disabling AMR > (didn't know if the screwy is_child_on_side() with ParallelMesh was messing > things up... also libMesh currently doesn't compile with AMR disabled! I > have patches that make it work that I will commit soon though!). None of > those things work. > > I'm thinking that it could be possible that it's not picking up dofs from > RemoteElems properly... so it's not getting the right sparsity pattern around > the edges of the parallel pieces of the domain.... but SparsityPattern::Build > is fairly dense and I can't quite see where it would be hosed up... > > BTW: Once we get past the first Jacobian evaluation (it's slow because of the > mallocs) we have fairly large (~300 million DoF) runs going on > multi-thousands of processors (so far up to about 4k... but we're headed to > 10k soon!)! What is the largest solve ever done with libMesh? This > certainly has to be up there... and we're not even to our goal yet! The > scaling is pretty darned good as well (other than the malloc thing). When I > have hard numbers I'll share. > > Derek > ------------------------------------------------------------------------------ > Benefiting from Server Virtualization: Beyond Initial Workload > Consolidation -- Increasing the use of server virtualization is a top > priority.Virtualization can reduce costs, simplify management, and improve > application availability and disaster protection. Learn more about boosting > the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev > _______________________________________________ > Libmesh-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/libmesh-devel ------------------------------------------------------------------------------ Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev _______________________________________________ Libmesh-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/libmesh-devel
