Corey Kovacs wrote: > I am in the early planning stages of putting together a lustre system > and when I look at IB speeds it seems most if not all the 40Gb IB > cards are running over PCIe with 8 lanes. If PCIe lanes are good for > about 200MB/s that means I can only get 1600MB's tops from one of > these cards, but 40Gb xlates to around 4GB/s. Am I reasoning this > correctly or is have the potential of the card wasted? >
PCIe "gen1" x8 will peak at around 1400-1600 MB/s, while "gen2" will peak around 3.2GB/s. So you need to make sure the server supports the new gen2 (5GT/s) PCIe bus with 8 lanes. For example, the Oracle Sun Fire X4170/X4270 servers support the faster PCIe bus. [Note that the raw BW is the same (PCIe x8 and DDR 4x IB; and PCIe x8 gen2 and QDR IB4x), but PCIe implementations have smaller maximum packet sizes so the effective BW is a bit lower due to higher overhead.] > I can see using 40Gb switches to get the backplane speed and reduce > contention on the switch but what are people using on the hosts? > > > Corey > _______________________________________________ > Lustre-discuss mailing list > [email protected] > http://lists.lustre.org/mailman/listinfo/lustre-discuss > _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
