On Feb 11, 2009, at 8:46 PM, Kevin Van Maren wrote: > Charles Taylor wrote: >> On Feb 11, 2009, at 4:35 PM, Scott Atchley wrote: >> >>> To add to Brian's comments, IB 4X SDR is limited to about 700-750 >>> MB/s >>> by the fabric. O2IBLND cannot go faster than minimum of either the >>> fabric or PCI-E connection allow. >> >> Hmmm. I can agree with the second part of that statement but I >> question the first. We've measured much closer to the 1GByte/sec >> wire rate of IB using several different tools. 750 GBytes/sec >> corresponds to roughly 6 GBits/sec. You lose 2 of the 10 Gbits to >> encoding (8B10) so line rate is really 8GBits/sec or 1 GByte/sec. >> Yes, you'll lose some more to protocol and swtiching overhead but it >> is not anywhere near an additional 2 GBits/sec - in our experience. > > Correct. Infinipath SDR was getting ~980 MB/s, and DDR HCAs in SDR > mode > can also do quite well in an x8 PCIe slot. > > The PCI-X HCAs were limited to around 850MB/s by the bus, and PCIe > HCAs > _are_ likewise limited to around 700-750MB/s -- but only in a PCIe > x4 slot. > > DDR IB (unless using a PCIe gen2 connectX card, or a x16 Infinipath > card) are also > limited to around 1450-1600 MB/s by the PCIe x8 bus, with a wire speed > of 2000 MB/s. > > QDR IB, in a Gen2 x8 PCIe slot, are also going to be limited to << > 4000MB/s line rate > (should expect around twice the BW of the gen1 PCIe slots). > > The IB headers are very small, compared to a 2KB or 4KB packet size, > but > the PCIe > headers (and eg flow-control overhead) are quite large compared to a > typical 256B packet size. > > To clarify one point: IB advertises the "signaling" rate, so the 10Gb > includes the overhead > bits, as 8 bits are encoded in a 10 bit representation for > transmission. So 10Gb/s = 1GB/s, > with 10-bit bytes. Ethernet, on the other hand, always advertises the > "data" rate, so 10Gb > Ethernet is 1.25GB/s (12.5Gb/s signaling rate), as there are 8 bits > in a > byte. Ethernet packet > headers are also effectively a bit larger than for IB (with IFG, > preamble, etc). > > Kevin
Thanks everyone! I get so confused by IB performance claims. :-) Scott _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
