> On 29 Sep 2017, at 15:05, Tobin Isaac <[email protected]> wrote: > > On Fri, Sep 29, 2017 at 09:04:47AM -0400, Tobin Isaac wrote: >> On Fri, Sep 29, 2017 at 12:19:54PM +0100, Lawrence Mitchell wrote: >>> Dear all, >>> >>> I'm attempting to understand some results I'm getting for matmult >>> performance. In particular, it looks like I'm obtaining timings that >>> suggest that I'm getting more main memory bandwidth than I think is >>> possible. >>> >>> The run setup is using 2 24 core (dual socket) ivybridge nodes (Xeon >>> E5-2697 v2). The specced main memory bandwidth is 85.3 GB/s per node, and >>> I measure a STREAM triad bandwidth using 48 MPI processes (two nodes) of >>> 148.2 GB/s. The last level cache is 30MB (shared between 12 cores) >> >> One thought: triad has a 1:2 write:read ratio, but with your MatMult() >> for P3 you would have about 1:50. Unless triad used nontemporal >> stores, the reported bandwidth from triad will be about 3./4. of the >> bandwidth available to pure streaming reads, so maybe you actually >> have ~197 GB/s of read bandwidth available. MatMult() would still be >> doing suspiciously well, but it would be within the measurements. How >> confident are you in the specced bandwidth?
I thought I was quite confident, but as you note: > Are you running on archer? I found one site [1] that lists the > bandwidth you gave, which corresponds to DDR3-1333, but other sites > [2] all say the nodes have DDR3-1833, in which case you would be > getting about 80% of spec bandwidth. > > [1]: https://www.archer.ac.uk/documentation/best-practice-guide/arch.php > [2]: > https://www.epcc.ed.ac.uk/blog/2013/11/20/archer-next-national-hpc-service-academic-research I am, yes. I will write to them to confirm, and get them to change their website! Yeah, 85.3 was computed via the assumption of DDR3-1333, whereas DDR3-1866 gives me 102.4. Phew :). Karl wrote: > according to > https://ark.intel.com/products/75283/Intel-Xeon-Processor-E5-2697-v2-30M-Cache-2_70-GHz > you get 59.7 GB/sec of peak memory bandwidth per CPU, so you should get about > 240 GB/sec for your two-node system. That's assuming I have DDR3-2133 RAM chips, but as noted above, it looks like the nodes probably have 1866 RAM. Giving 204.8 GB/s. > If you use PETSc's `make streams`, then processor placement may - > unfortunately - not be ideal and hence underestimating the achievable > performance. Have a look at the new PETSc 3.8 manual [1], Chapter 14, where > Richard and I nailed down some of these performance aspects. Thanks, I am using make streams (or rather, just running the MPIVersion by hand). I have long been led to believe that Cray's aprun does a reasonable job of process placement (and pinning) for pure MPI jobs, so I have just trusted that. On another note, I modified the MPIVersion to not just report the TRIAD number, and got: Copy: 91467.3281 Rate (MB/s) Scale: 63774.9615 Rate (MB/s) Add: 73994.6106 Rate (MB/s) Triad: 73564.8991 Rate (MB/s) Inspecting the assembly, none of these are using non-temporal stores, but the copy has been turned into a call to memcpy (which might be). In any case, those numbers would have led me to question the website-stated spec sheet numbers sooner. Thanks all, Lawrence
signature.asc
Description: Message signed with OpenPGP
