On Thu, 10 Mar 2016, BRADLEY, PETER C PW wrote:
This is an academic exercise, obviously. The curve shown comes from one pair
of ranks running on the same node alternating between MPI_Send and
MPI_Recv. The most likely suspect is a cache effect, but rather than assuming,
I was curious if there might be any other aspects of the implementation
at work.
Pete
Well with some more effort you can get all statistics from the processor
on cache misses...
Your graph doesn't show other effects though i suspect.
A multiple of 5GB/s is a lot of bandwidth for a laptop right now.
Please consider that each buffer gets copied as you ship a message so the
actual bandwidth is a multiple of that 5GB/s.
This test is all shipping it in a FIFO order (first in first out) - trashing
the caches in short.
Ping-pong isn't intended as a bandwidth test at all.
It's a latency test. More useful to run on a supercomputer and figure out
the time it takes to get to a remote node and back, and then it divides
that by 2.
I wrote a few tests some years ago to test the random latency of your RAM
with all cores at the same time.
Yet very few of those tests are busy with bandwidth. They care about
number of of messages a second one can push through. So with latency.
The network plays a larger role when run over multiple nodes, whereas
here you just try to figure out how good your L2/L3 cache is on the CPU.
Let me assure you - that L2/L3 cache works very well :)
Pete,
how did you measure the bandwidth ?
iirc, IMB benchmark does not reuse send and recv buffers, so the results
could be different.
also, you might want to use a logarithmic scale for the message size, so
information for small messages is easier to read.
Cheers,
Gilles
On Thursday, March 10, 2016, BRADLEY, PETER C PW <peter.c.bradley_at_[hidden]>
wrote:
> I’m curious what causes the hump in the pingpong bandwidth curve when
> running on shared memory. Here’s an example running on a fairly antiquated
> single-socket 4 core laptop with linux (2.6.32 kernel). Is this a cache
> effect? Something in OpenMPI itself, or a combination?
>
>
>
>
>
> [image: Macintosh HD:Users:up:Pictures:bandwidth_onepair_onenode.png]
>
>
>
> Pete
>
>
>
