On Thursday 18 January 2007 13:08, Scott Atchley wrote: ... > The best uni-directional performance I have heard of for PCIe 8x IB > DDR is ~1,400 MB/s (11.2 Gb/s)
This is on par with what I have seen. > with Lustre, which is about 55% of the > theoretical 20 Gb/s advertised speed. I think this should be calculated against 16 Gbps, not 20 Gbps. > The ~900 MB/s (7.2 Gb/s) > mentioned above is, of course, ~72% of advertised speed. If any IB > folks have any better numbers, please correct me. Using MPI (over a non idle multi-level switch) I get 940 * 10^6 Bytes/s which is 94% of peak for that IB 4x SDR. > The data throughput limit for 8x PCIe is ~12 Gb/s. The theoretical > limit is 16 Gb/s, but each PCIe packet has a whopping 20 byte > overhead. If the adapter uses 64 byte packets, then you see 1/3 of > the throughput go to overhead. AFAIK the datafield of a pci-express packet is 0-4096 bytes and the header a bit more than 20 bytes (including things such as start/stop frame bytes, LCRC/ECRC..). This gives a maximum speed over 4x PCIe of 993.3 10^6 Bytes/s (8 Gbps after coding minus header waste for a full 4096 byte payload). In short, the SDR IB equipment I have seen has easily reached 90%+ while PCI-express on the platforms I've tried has been limited to ~75%. Current IB DDR HCAs are probably limited by (at least) PCI-express 8x. /Peter
pgphFCXjmUXlv.pgp
Description: PGP signature
