On 4 Feb 2011, at 15:37, Doug Hughes wrote: > On 2/4/2011 9:25 AM, Yves Dorfsman wrote: >>> I have some trouble with infiniband - and admittedly I am an infiniband >>> novice. I get 7GBit/s using TCP (iperf/wget and others). Oddly though >>> because it's a QDR (40GBit/s) switch. >>> Same results with 2.6.36, 2.6.38-rc2 kernel. >>> QLogic PCI-e cards and QLogic switches. >>> >>> I changed kernels/bios/tcp settings and what have you, but no luck. I >>> notice the cpu can be quite busy during these tests. >>> >>> Anyone seen something like this before or knows more about inifiniband >>> than me ? ;)) >> Yes we ran into a similar issue. We had put the IB card in the first >> available >> PCIU slot on the machines. On most machines some PCI slot are faster than >> others, these IB needed to be on a faster slot to be able to attain maximum >> speeds, of course this wasn't documented. >> >> Look for "connector width", on newer machines you typically have slots with >> "x8" and one or two with "x16" width. Move the cards to the faster/widest >> slots. >> > I'm going to reply to a whole bunch of messages in this thread, so just > be advised that I'm not picking on any one in particular. > > Let's assume that the original card in questions is a QDR card. That
correct. > means 40Gbits/sec, but infiniband has an 8/10 overhead for encoding, so > the actual maximum achievable bandwidth is 32Gbits/sec. (Anybody that > says 35 is trying to pull something). DDR > infiniband is 20 gbits/sec (actually 16). In the next version of > Infiniband, later this year, they eliminate the 8/10 off-the-top > overhead, for what it's worth. good point. I checked back myself and found this by mellanox: http://www.mellanox.com/content/pages.php?pg=performance_infiniband where the list 2950MB/s (23600Mb/s). Makes it easier to believe - but even though I am far away from that yet ; [...] > > Re: 7gbits/etc. As others have said, you're over-running your CPU. IP > over IB works just fine, but the packet sizes aren't matched up. What > you really want to do is test it using actual native infiniband > protocols. I recommend the Intel Benchmark Suite (IBM - nee Pallas). It > will test infiniband typical MPI operations. Fair enough, problem is, I am trying to provide highspeed internet access to clients and they need tcp/ip (only). So I got to improve the speed of IPoIB. And yes, we have >20GBit/s internet connectivity. Conrad h _______________________________________________ Tech mailing list [email protected] https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech This list provided by the League of Professional System Administrators http://lopsa.org/
