> I actually tested it with a simple Java test loader I quickly put together, > which ran on each machine and continuously has written random data to DFS. I > tuned the writing rate until I got ~77Mb/s - above it the iowait loads on > each disk (measured by iostat) became above 50% - 60%, which is quite close > to disks limits.
How many DNs are you using? How many copies of the benchmark are you running? What results do you get just running a single copy of the benchmark? I see ~46 MB/s hadoop fs put'ing a local 1gb file from one DN, using 3-way replication. Running the test on three DNs I get around 30 MB/s. This is a little less than half the theoretical limit (using three hosts each with a single gigabit nic). In these tests I purged the buffer cache before running the test, with the input file cached in memory (more similar to your test) I get 92 MB/s on one host but about the same rate for three hosts (we're network bound). This is about 3x faster than what you're seeing so I suspect something's up with your test. Would be useful for you to see what results you get running the same test I did. > You mentioned some TestDFSIO, any idea if it's present in 0.18.3? It's in 0.18.3 See src/test/org/apache/hadoop/fs/TestDFSIO.java Thanks, Eli
