On Thursday 11 February 2010, Erik Froese wrote: > Jagga, > I think this is more a function of scp than than ib in general. Have > you tried using the HPN-SSH patches? > http://www.psc.edu/networking/projects/hpn-ssh/ > > You could also try using apache to serve the iso over HTTP to see if > SCP is the bottleneck.
Iperf and netpipe are examples of tools made for this purpose. All this is, however, IPoIB which is _not_ what you'd typically use to run IB at high speed. Running Lustre on IB is typically done on verbs (lnet o2ib, not tcp). /Peter > Also I think IPoIB should be able to use a much larger MTU. It may > help for large files. > > Do this on both machines: ifconfig ib0 mtu 65520 > To benchmark the raw infiniband performance see the man pages for > ib_read_bw and ib_write_bw > > Erik > > On Wed, Feb 10, 2010 at 7:48 PM, Jagga Soorma <[email protected]> wrote: > > Hi Guys, > > > > I have setup a new cluster with a infiniband interconnect and I am a bit > > confused with the performance I am getting as far as transfer rates. Is > > there something that I am missing here? Shouldn't the transfer rate over > > the ib interface be much faster and not the same compared with the 1G > > (bonded mode=6) interface? > > > > (I am new to infiniband but this does not seem right. My routing is also > > setup correctly. Not sure what I might be missing here): > > > > -- > > hpc101:/var/tmp # time scp SLES-11-DVD-x86_64-GM-DVD1.iso > > r...@hpc103:/var/tmp/ > > SLES-11-DVD-x86_64-GM-DVD1.iso > > > > 100% 2749MB > > 62.5MB/s 00:44 > > > > real 0m43.882s > > user 0m29.482s > > sys 0m6.736s > > > > hpc101:/var/tmp # time scp SLES-11-DVD-x86_64-GM-DVD1.iso > > r...@hpc103-ib:/var/tmp/ > > SLES-11-DVD-x86_64-GM-DVD1.iso > > > > 100% 2749MB > > 80.9MB/s 00:34 > > > > real 0m35.757s > > user 0m24.498s > > sys 0m6.292s ...
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
