On Thu, Apr 9, 2009 at 9:30 PM, Brian Bockelman <[email protected]> wrote: > > On Apr 9, 2009, at 5:45 PM, Stas Oskin wrote: > >> Hi. >> >> I have 2 questions about HDFS performance: >> >> 1) How fast are the read and write operations over network, in Mbps per >> second? >> > > Depends. What hardware? How much hardware? Is the cluster under load? > What does your I/O load look like? As a rule of thumb, you'll probably > expect very close to hardware speed.
For comparison, on a 1400 node cluster, I can checksum 100 TB in around 10 minutes, which means I'm seeing read averages of roughly 166 GB/sec. For writes with replication of 3, I see roughly 40-50 minutes to write 100TB, so roughly 33 GB/sec average. Of course the peaks are much higher. Each node has 4 SATA disks, dual quad core, and 8 GB of ram. -- Owen
