Hello all,
Out of some curiosity I recently rebased my GMG implementation on to the
upcoming NBX changes in PR #1965 to do some weak scaling analysis.
I ran an np 256 Poisson problem using GMG with ~10k dofs/proc, and in
short it, seems like the NBX changes provide some solid improvement
bringing my total runtime from something like ~19s to ~16s, so a nice
step on the road to weak GMG scaling. Pre-NBX I had a good chunk (30%
total w/sub) of my time being spent in the alltoall() which post-NBX is
down to 2%! This came with a fairly large amount of calls to
possible_receive() and now 15% of my total time being spent in there,
but the overall timing seems to be a win so thanks much for this work!
Despite these improvements, the weak scaling for the GMG implementation
is still a bit lacking unfortunately as np1=~1s. I ran these tests
through gperf in order to gain some more insight and it looks to me that
major components slowing down the setup time are still
refining/coarsening/distributing_dofs which in turn do a lot of nodal
parallel consistency adjusting and setting nonlocal_dof_objects and am
wondering if there are maybe some low hanging fruit to improve on around
those calls.
Logs for these runs and the corresponding gperf outputs can be found at
[1], and come with the minor note that I had some trouble combining all
of the profile_pid logs into a single gperf output file. Basically all
the function names get garbled if I pass in all the profile_pid outputs
to pprof ( hoping maybe Derek Gaston can provide some tips on how to do
this properly? ), but as such the linked 'web' plots are taken from a
single representative gprof_pid output. I cross compared a number of
such outputs and the results seemed fairly consistent between samples,
but obviously this isnt perfect.
Anyways, figured I'd post this as an update, please let me know if I can
provide anything further surrounding these tests that would be helpful.
Thanks as always,
- Boris
[1]: https://drive.google.com/open?id=1dgHKiC8oStpTR0yJFv5uVb8wyJRIFTkk
_______________________________________________
Libmesh-devel mailing list
Libmesh-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libmesh-devel
