Cody, I'm not sure if you saw the graph I uploaded it again here: https://dl.dropboxusercontent.com/u/19391830/scaling.jpg.
In all these runs the NDOFs/Processor is less than 10000. What is bothering me is that the enforce_constraints_exactly is taking up more and more time as number of processor grows for the same problem. Now I can think of an explanation that the NDOFs/Processor is so low that the communication time is becoming a problem! That said my main concern is that I'm using the DirichletBCs API badly and that results in bad scaling. PS: Sorry for multiple copies I forgot to CC libMesh user list. Best, Ata On Jun 10, 2013, at 9:48 AM, Cody Permann <[email protected]> wrote: > Ata, > > You might be scaling past the reasonable limit for libMesh. I don't know > what solver you are using, but for a strong scaling study we generally don't > go below 10,000 local DOFs. This is the recommended floor for PETSc too: > http://www.mcs.anl.gov/petsc/documentation/faq.html#slowerparallel > > Before you start drawing conclusions about scaling, you might start with a > bigger problem and see if it scales well to the ~20,000 local DOF range. > > Cody > > > > On Mon, Jun 10, 2013 at 8:42 AM, Ataollah Mesgarnejad > <[email protected]> wrote: > Dear all, > > I've been doing some scaling tests on my code. When I look at time (or % of > time) spent at each stage in libMesh log I see that the > enforce_constraints_exactly stage in DofMap is scaling very bad. I was > wondering if anyone can comment. > > Here is my EquationSystems.print_info(): > > EquationSystems > n_systems()=2 > System #0, "elasticity_system" > Type "TransientLinearImplicit" > Variables={ "u" "v" } > Finite Element Types="LAGRANGE", "JACOBI_20_00" > Infinite Element Mapping="CARTESIAN" > Approximation Orders="FIRST", "THIRD" > n_dofs()=48660 > n_local_dofs()=930 > n_constrained_dofs()=1048 > n_local_constrained_dofs()=56 > n_vectors()=3 > n_matrices()=1 > DofMap Sparsity > Average On-Processor Bandwidth <= 13.6478 > Average Off-Processor Bandwidth <= 0.904233 > Maximum On-Processor Bandwidth <= 20 > Maximum Off-Processor Bandwidth <= 16 > DofMap Constraints > Number of DoF Constraints = 1048 > Average DoF Constraint Length= 0 > Number of Node Constraints = 0 > System #1, "fracture_system" > Type "TransientNonlinearImplicit" > Variables="psi" > Finite Element Types="LAGRANGE", "JACOBI_20_00" > Infinite Element Mapping="CARTESIAN" > Approximation Orders="FIRST", "THIRD" > n_dofs()=24330 > n_local_dofs()=465 > n_constrained_dofs()=167 > n_local_constrained_dofs()=0 > n_vectors()=3 > n_matrices()=1 > DofMap Sparsity > Average On-Processor Bandwidth <= 6.82388 > Average Off-Processor Bandwidth <= 0.452117 > Maximum On-Processor Bandwidth <= 10 > Maximum Off-Processor Bandwidth <= 8 > DofMap Constraints > Number of DoF Constraints = 167 > Average DoF Constraint Length= 0 > Number of Node Constraints = 0 > > > and here is the how scaling looks for every stage that took > 1% of time: > > > > Best, > Ata > ------------------------------------------------------------------------------ > How ServiceNow helps IT people transform IT departments: > 1. A cloud service to automate IT design, transition and operations > 2. Dashboards that offer high-level views of enterprise services > 3. A single system of record for all IT processes > http://p.sf.net/sfu/servicenow-d2d-j > _______________________________________________ > Libmesh-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/libmesh-users > > ------------------------------------------------------------------------------ How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j _______________________________________________ Libmesh-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/libmesh-users
