With my setup. I have been able to get 10MBps write speed, 40MBps read speed while writing multiple files (ranging a few Bytes to 100MB) into SequenceFiles, and reading them back. The cluster has 1Gbps backbone.
On Tue, Feb 10, 2009 at 5:53 PM, Mark Kerzner <markkerz...@gmail.com> wrote: > Brian, I have a similar question: why does transfer from a local filesystem > to SequenceFile takes so long (about 1 second per Meg)? > Thank you, > Mark > > On Tue, Feb 10, 2009 at 4:46 PM, Brian Bockelman <bbock...@cse.unl.edu > >wrote: > > > > > On Feb 10, 2009, at 4:10 PM, Wasim Bari wrote: > > > > Hi, > >> Could someone help me to find some real Figures (transfer rate) about > >> Hadoop File transfer from local filesystem to HDFS, S3 etc and among > >> Storage Systems (HDFS to S3 etc) > >> > >> Thanks, > >> > >> Wasim > >> > > > > What are you looking for? Maximum possible transfer rate? Maximum > > possible transfer rate per client? Generally, if you're using the Java > > client, transfer rate to/from HDFS is limited by the hardware you have > and > > the network connection (if you have 1Gbps per client). > > > > I could give you a graph showing a peak of 9Gbps from our Hadoop instance > > to the WAN, but that's not very interesting if you don't have a 10Gbps > > pipe... > > > > Brian > > > > >
