On Feb 6, 2007, at 12:38 PM, Alex Tumanov wrote:
http://www.open-mpi.org/faq/?category=tcp#tcp-routability
The pointer was rather informative. We do have to use non-standard
ranges for IB interfaces, because we're performing automatic IP over
IB configuration based on the eth0 IP and netmask. Given 10.x.y.z/8
configuration for eth0, the IPs assigned to infiniband interfaces will
not only end up on the same subnet ID, but may even conflict with
existing ethernet interface IP addresses. Hence the use of 20.x.y.z
and 30.x.y.z for ib0 & ib1 respectively.
I'm not sure I'm parsing your explanation properly. Are you saying
that your cluster's ethernet addresses are dispersed across all of
10.x.y.z, and therefore you don't want the IPoIB addresses to
conflict? Even being conservative, that's 250^3 IP addresses (over
15 million). There should be plenty of space in there for your
cluster's ethernet and IPoIB addresses to share (and any other
machines that also share your 10.x.y.z address space).
But it doesn't really matter -- this is a minor point. :-)
I actually tried benchmarking with HPLinpack. Specifically, I'm
interested in measuring performance improvements when running OpenMPI
jobs over several available interconnects. However, I have difficulty
interpreting the cryptic HPL output. I've seen members of the list
using xhpl benchmark. Perhaps someone could shed some light on how to
read its output? Also, my understanding is that the only advantage of
I'll defer to others on this one...
multiple interconnect availability is the increased bandwidth for
OpenMPI message striping - correct? In that case, I would probably
That's a big reason, yes.
benefit from a more bandwidth intensive benchmark. If the OpenMPI
community could point me in the right direction for that, it would be
greatly appreciated. I have a feeling that this is not one of HPL's
strongest points.
Actually, it depends on how big your HPL problem size it. HPL can
send very large messages if you set the size high enough. For
example, when we were running HPL at Sandia for its Top500 run, we
were seeing 800MB messages (for 4000+ nodes, lotsa memory -- very
large HPL problem size).
A simple ping-pong benchmark can also be useful to ballpark what
you're seeing for your network performance. My personal favorite is
NetPIPE, but there's others as well.
--
Jeff Squyres
Server Virtualization Business Unit
Cisco Systems
