Jagga, I think this is more a function of scp than than ib in general. Have you tried using the HPN-SSH patches? http://www.psc.edu/networking/projects/hpn-ssh/
You could also try using apache to serve the iso over HTTP to see if SCP is the bottleneck. Also I think IPoIB should be able to use a much larger MTU. It may help for large files. Do this on both machines: ifconfig ib0 mtu 65520 To benchmark the raw infiniband performance see the man pages for ib_read_bw and ib_write_bw Erik On Wed, Feb 10, 2010 at 7:48 PM, Jagga Soorma <[email protected]> wrote: > > Hi Guys, > > I have setup a new cluster with a infiniband interconnect and I am a bit > confused with the performance I am getting as far as transfer rates. Is > there something that I am missing here? Shouldn't the transfer rate over the > ib interface be much faster and not the same compared with the 1G (bonded > mode=6) interface? > > (I am new to infiniband but this does not seem right. My routing is also > setup correctly. Not sure what I might be missing here): > > -- > hpc101:/var/tmp # time scp SLES-11-DVD-x86_64-GM-DVD1.iso > r...@hpc103:/var/tmp/ > SLES-11-DVD-x86_64-GM-DVD1.iso > 100% 2749MB 62.5MB/s 00:44 > > real 0m43.882s > user 0m29.482s > sys 0m6.736s > > hpc101:/var/tmp # time scp SLES-11-DVD-x86_64-GM-DVD1.iso > r...@hpc103-ib:/var/tmp/ > SLES-11-DVD-x86_64-GM-DVD1.iso > 100% 2749MB 80.9MB/s 00:34 > > real 0m35.757s > user 0m24.498s > sys 0m6.292s > > hpc101:/var/tmp # netstat -rn > Kernel IP routing table > Destination Gateway Genmask Flags MSS Window irtt Iface > 10.0.250.0 0.0.0.0 255.255.255.0 U 0 0 0 ib0 > 128.137.126.0 0.0.0.0 255.255.255.0 U 0 0 0 bond0 > 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo > 0.0.0.0 128.137.126.253 0.0.0.0 UG 0 0 0 bond0 > > hpc101:/var/tmp # ifconfig ib0 > ib0 Link encap:UNSPEC HWaddr > 80-00-00-48-FE-80-00-00-00-00-00-00-00-00-00-00 > inet addr:10.0.250.5 Bcast:10.0.250.255 Mask:255.255.255.0 > inet6 addr: fe80::223:7dff:ff93:bf4d/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 > RX packets:493685 errors:0 dropped:0 overruns:0 frame:0 > TX packets:890402 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:256 > RX bytes:22848079 (21.7 Mb) TX bytes:7349365631 (7008.9 Mb) > -- > > Thanks in advance for your assistance. > > Regards, > -J > > _______________________________________________ > Lustre-discuss mailing list > [email protected] > http://lists.lustre.org/mailman/listinfo/lustre-discuss > _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
