Brian, large files using command-line hadoop go fast, so it is something about my computer or network. I won't worry about this now, especially in light of Amit reporting fast writes and reads.
Mark On Tue, Feb 10, 2009 at 5:00 PM, Brian Bockelman <[email protected]>wrote: > > On Feb 10, 2009, at 4:53 PM, Mark Kerzner wrote: > > Brian, I have a similar question: why does transfer from a local >> filesystem >> to SequenceFile takes so long (about 1 second per Meg)? >> > > Hey Mark, > > I saw your question about speed the other day ... unfortunately, I didn't > have any specific advice so I stayed quiet :) > > In a correctly configured cluster, performance is mostly limited by > available hardware. If it's obvious that performance is well below hardware > limits (such as in your case), it's usually (a) you're not generating files > fast enough or (b) something is configured wrong. > > Have you just tried hadoop fs -put .... for some large file hanging around > locally? If that doesn't go more than 5MB/s or so (when your hardware can > obviously do such a rate), then there's probably a configuration issue. > > Brian > > > >> Thank you, >> Mark >> >> On Tue, Feb 10, 2009 at 4:46 PM, Brian Bockelman <[email protected] >> >wrote: >> >> >>> On Feb 10, 2009, at 4:10 PM, Wasim Bari wrote: >>> >>> Hi, >>> >>>> Could someone help me to find some real Figures (transfer rate) about >>>> Hadoop File transfer from local filesystem to HDFS, S3 etc and among >>>> Storage Systems (HDFS to S3 etc) >>>> >>>> Thanks, >>>> >>>> Wasim >>>> >>>> >>> What are you looking for? Maximum possible transfer rate? Maximum >>> possible transfer rate per client? Generally, if you're using the Java >>> client, transfer rate to/from HDFS is limited by the hardware you have >>> and >>> the network connection (if you have 1Gbps per client). >>> >>> I could give you a graph showing a peak of 9Gbps from our Hadoop instance >>> to the WAN, but that's not very interesting if you don't have a 10Gbps >>> pipe... >>> >>> Brian >>> >>> >>> >
