It may be a good idea to combine this study with looking at the output of -log_summary (assuming PETSc is your solver engine) to see if there is a correlation with the growth in communication or other components of matrix assembly. Dmitry.
On Mon, Jun 10, 2013 at 10:16 AM, Ataollah Mesgarnejad <[email protected]> wrote: > Cody, > > I'm not sure if you saw the graph I uploaded it again here: > https://dl.dropboxusercontent.com/u/19391830/scaling.jpg. > > In all these runs the NDOFs/Processor is less than 10000. What is bothering > me is that the enforce_constraints_exactly is taking up more and more time as > number of processor grows for the same problem. > > Now I can think of an explanation that the NDOFs/Processor is so low that the > communication time is becoming a problem! That said my main concern is that > I'm using the DirichletBCs API badly and that results in bad scaling. > > PS: Sorry for multiple copies I forgot to CC libMesh user list. > > Best, > Ata > On Jun 10, 2013, at 9:48 AM, Cody Permann <[email protected]> wrote: > >> Ata, >> >> You might be scaling past the reasonable limit for libMesh. I don't know >> what solver you are using, but for a strong scaling study we generally don't >> go below 10,000 local DOFs. This is the recommended floor for PETSc too: >> http://www.mcs.anl.gov/petsc/documentation/faq.html#slowerparallel >> >> Before you start drawing conclusions about scaling, you might start with a >> bigger problem and see if it scales well to the ~20,000 local DOF range. >> >> Cody >> >> >> >> On Mon, Jun 10, 2013 at 8:42 AM, Ataollah Mesgarnejad >> <[email protected]> wrote: >> Dear all, >> >> I've been doing some scaling tests on my code. When I look at time (or % of >> time) spent at each stage in libMesh log I see that the >> enforce_constraints_exactly stage in DofMap is scaling very bad. I was >> wondering if anyone can comment. >> >> Here is my EquationSystems.print_info(): >> >> EquationSystems >> n_systems()=2 >> System #0, "elasticity_system" >> Type "TransientLinearImplicit" >> Variables={ "u" "v" } >> Finite Element Types="LAGRANGE", "JACOBI_20_00" >> Infinite Element Mapping="CARTESIAN" >> Approximation Orders="FIRST", "THIRD" >> n_dofs()=48660 >> n_local_dofs()=930 >> n_constrained_dofs()=1048 >> n_local_constrained_dofs()=56 >> n_vectors()=3 >> n_matrices()=1 >> DofMap Sparsity >> Average On-Processor Bandwidth <= 13.6478 >> Average Off-Processor Bandwidth <= 0.904233 >> Maximum On-Processor Bandwidth <= 20 >> Maximum Off-Processor Bandwidth <= 16 >> DofMap Constraints >> Number of DoF Constraints = 1048 >> Average DoF Constraint Length= 0 >> Number of Node Constraints = 0 >> System #1, "fracture_system" >> Type "TransientNonlinearImplicit" >> Variables="psi" >> Finite Element Types="LAGRANGE", "JACOBI_20_00" >> Infinite Element Mapping="CARTESIAN" >> Approximation Orders="FIRST", "THIRD" >> n_dofs()=24330 >> n_local_dofs()=465 >> n_constrained_dofs()=167 >> n_local_constrained_dofs()=0 >> n_vectors()=3 >> n_matrices()=1 >> DofMap Sparsity >> Average On-Processor Bandwidth <= 6.82388 >> Average Off-Processor Bandwidth <= 0.452117 >> Maximum On-Processor Bandwidth <= 10 >> Maximum Off-Processor Bandwidth <= 8 >> DofMap Constraints >> Number of DoF Constraints = 167 >> Average DoF Constraint Length= 0 >> Number of Node Constraints = 0 >> >> >> and here is the how scaling looks for every stage that took > 1% of time: >> >> >> >> Best, >> Ata >> ------------------------------------------------------------------------------ >> How ServiceNow helps IT people transform IT departments: >> 1. A cloud service to automate IT design, transition and operations >> 2. Dashboards that offer high-level views of enterprise services >> 3. A single system of record for all IT processes >> http://p.sf.net/sfu/servicenow-d2d-j >> _______________________________________________ >> Libmesh-users mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/libmesh-users >> >> > > ------------------------------------------------------------------------------ > How ServiceNow helps IT people transform IT departments: > 1. A cloud service to automate IT design, transition and operations > 2. Dashboards that offer high-level views of enterprise services > 3. A single system of record for all IT processes > http://p.sf.net/sfu/servicenow-d2d-j > _______________________________________________ > Libmesh-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/libmesh-users ------------------------------------------------------------------------------ How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j _______________________________________________ Libmesh-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/libmesh-users
