How exactly do you set the MTU for ipoib? I am running the latest unpatched git branch of vofed kernel 1.2.5 and I get "SIOCSIFMTU: Invalid argument" when I try ifconfig ib0 mtu 65520. Anything above the preset 2044 returns this issue.
-----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Rick Jones Sent: Wednesday, September 12, 2007 11:28 AM To: Arlin Davis Cc: general; Davis,Arlin R Subject: Re: [ofa-general] scp performance over IPoIB Arlin Davis wrote: > Rick Jones wrote: > >> Davis, Arlin R wrote: >> >>> Can someone explain why scp performance over IPoIB would be 10x >>> slower then on GBE? The netperf numbers look normal. >> >> >> >> So, you could try tweaking the MTU on the IPoIB interfaces. >> >> > > Rick, > > Thanks for the suggestion. Looks like we may need to change the > default MTU for IPoIB. It would be interesting to see results from > other distributions. > > (Woodcrest, Xeon 5160, DDR, RHEL4U4) > > MTU SCP NetPerf > > 1024 41 MB/s 151 MB/s > 2048 50 MB/s 313 MB/s > 4096 50 MB/s 485 MB/s > 8192 50 MB/s 641 MB/s > 16384 25 MB/s 761 MB/s > 32768 50 MB/s 700 MB/s > 65520 8 MB/s 440 MB/s I'm actually a triffle surprised that netperf was affected by the 65520 MTU - I'm guessing you were using all defaults, which on "linux" IIRC means netperf was making 16KB (K == 1024) sends. I suspect that if you were to make 64K sends from netperf (test specific -m 64K) that the numbers for 64420 might be better. I'm really shakey on scp behaviour knowledge, but suspect that perhaps with the "HPN" (High Performance Network) patches in place (check the archives pointed-to by: https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev) it might be possible to get good SCP performance out of a 65520 byte MTU. I'm _guessing" that by default scp isn't trying to put-out > 65520 bytes worth of data in the sum of its sends with its own windowing and so gets hit by issues with Nagle. Ie it is doing write, write, read and the second write at least is sub-MSS. Some strace tracing of the scp transfer could confirm/deny that hypothesis. So, it may not be necessary to shrink the MTU. rick jones _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
