Craig Prescott wrote:
Scott Weitzenkamp (sweitzen) wrote:
But the effect is still clear.
throughput:
64K 128K 1M
SDP 7602.40 7560.57 5791.56
BZCOPY 5454.20 6378.48 7316.28
Looks unclear to me. Sometimes BZCOPY does better, sometimes worse.
Fair enough.
While measuring a broader spectrum of message sizes, I noted a
big variation in throughput and send service demand for the SDP
<snip>
Sorry for the slow follow up. I've tried to get to the bottom of this.
I think what I've found could explain the mixed results that have been
observed when people try to see any benefit from BZCOPY.
In a nutshell, on my setup I found that normal SDP "Bcopy" throughput
and send service demand is highly dependent upon which core the
netperf client runs on (is it the same core handling ib_mthca
interrupts?) and whether or not the pages mapped by the netperf
client are on a node local to the core executing that client.
BZCOPY, on the other hand, shows only weak dependence upon these
variables.
What I did was the following:
1) Fix the netserver and ib_mthca interrupt mappings on specific
cores.
2) On the netperf client machine, run netperf on each core for
message sizes from 64KiB to 4MiB serially while holding
ib_mthca interrupt mappings constant.
3) Repeat step 2) for all possible ib_mthca interrupt mappings.
4) Repeat step 2) and 3) with the client booted with "mem=1024M".
Step 4) forces all pages mapped by the netperf client to come
from node 0 (can see with /proc/<pid>/numa_maps).
On my setup, BZCOPY performance began to be competitive with
Bcopy once the message sizes reached about 256KiB. Somewhere
between 1MiB and 2MiB, BZCOPY send service demand was *always*
less than Bcopy - even under the most advantageous conditions
for Bcopy.
I documented the setup and results here:
http://hpc.ufl.edu/benchmarks/ib_sdp/smp_aff.html
The last plots on the page are the ones to look at if you
don't want to read the whole thing.
So there is a class of conditions for which SDP Bcopy will
remain competitve with BZCOPY, even as message sizes
become quite large. I think this may explain why some
people see it, and some people don't.
Cheers,
Craig
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
