What does "cat /sys/class/net/ib0/mode" report? If "datagram", you need to run "echo connected > /sys/class/net/ib0/mode", then you can raise the MTU.
Scott Weitzenkamp SQA and Release Manager Server Virtualization Business Unit Cisco Systems > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of > Sufficool, Stanley > Sent: Wednesday, September 12, 2007 2:15 PM > To: Arlin Davis; general > Subject: RE: [ofa-general] scp performance over IPoIB > > How exactly do you set the MTU for ipoib? > > I am running the latest unpatched git branch of vofed kernel > 1.2.5 and I > get "SIOCSIFMTU: Invalid argument" when I try ifconfig ib0 mtu 65520. > Anything above the preset 2044 returns this issue. > > > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Rick Jones > Sent: Wednesday, September 12, 2007 11:28 AM > To: Arlin Davis > Cc: general; Davis,Arlin R > Subject: Re: [ofa-general] scp performance over IPoIB > > Arlin Davis wrote: > > Rick Jones wrote: > > > >> Davis, Arlin R wrote: > >> > >>> Can someone explain why scp performance over IPoIB would be 10x > >>> slower then on GBE? The netperf numbers look normal. > >> > >> > >> > >> So, you could try tweaking the MTU on the IPoIB interfaces. > >> > >> > > > > Rick, > > > > Thanks for the suggestion. Looks like we may need to change the > > default MTU for IPoIB. It would be interesting to see results from > > other distributions. > > > > (Woodcrest, Xeon 5160, DDR, RHEL4U4) > > > > MTU SCP NetPerf > > > > 1024 41 MB/s 151 MB/s > > 2048 50 MB/s 313 MB/s > > 4096 50 MB/s 485 MB/s > > 8192 50 MB/s 641 MB/s > > 16384 25 MB/s 761 MB/s > > 32768 50 MB/s 700 MB/s > > 65520 8 MB/s 440 MB/s > > I'm actually a triffle surprised that netperf was affected by > the 65520 > MTU - I'm guessing you were using all defaults, which on "linux" IIRC > means netperf was making 16KB (K == 1024) sends. I suspect > that if you > were to make 64K sends from netperf (test specific -m 64K) that the > numbers for 64420 might be better. > > I'm really shakey on scp behaviour knowledge, but suspect that perhaps > with the "HPN" (High Performance Network) patches in place (check the > archives pointed-to by: > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev) > it might be > possible to get good SCP performance out of a 65520 byte MTU. I'm > _guessing" that by default scp isn't trying to put-out > 65520 bytes > worth of data in the sum of its sends with its own windowing > and so gets > hit by issues with Nagle. Ie it is doing write, write, read and the > second write at least is sub-MSS. Some strace tracing of the scp > transfer could confirm/deny that hypothesis. > > So, it may not be necessary to shrink the MTU. > > rick jones > _______________________________________________ > general mailing list > [email protected] > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > _______________________________________________ > general mailing list > [email protected] > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
