On Feb 11, 2009, at 4:35 PM, Scott Atchley wrote: > To add to Brian's comments, IB 4X SDR is limited to about 700-750 MB/s > by the fabric. O2IBLND cannot go faster than minimum of either the > fabric or PCI-E connection allow.
Hmmm. I can agree with the second part of that statement but I question the first. We've measured much closer to the 1GByte/sec wire rate of IB using several different tools. 750 GBytes/sec corresponds to roughly 6 GBits/sec. You lose 2 of the 10 Gbits to encoding (8B10) so line rate is really 8GBits/sec or 1 GByte/sec. Yes, you'll lose some more to protocol and swtiching overhead but it is not anywhere near an additional 2 GBits/sec - in our experience. Just ran a quick IMB (formerly Pallas) between a couple of our SDR nodes and got 860 MBytes/sec (ping-pong, 4MB). So I don't think there is anything inherent in SDR IB that limits you to 750 MBytes/ sec. However, running IPoIB will probably limit you to something even less than that which is why you should use the O2IBLND if you want the real benefit of IB. Just our experience, Charlie Taylor UF HPC Center _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
