also, try dfstestio and then restart one datanode and see how it behaves http://answers.oreilly.com/topic/460-how-to-benchmark-a-hadoop-cluster/
On Tue, Dec 28, 2010 at 9:45 PM, Rajat Goel <rajatgoe...@gmail.com> wrote: > Hi, > > I want to measure read/write rates to HDFS under various conditions such as > under heavy load or one data node goes down etc? Is there some profiler > already available for such purpose? > > I am pushing data at high rate to HDFS, reads are also happening in > parallel and I suddenly reboot one datanode. I observe that I am no longer > able to write to HDFS (from live datanodes) at the same higher rate. This > happens for few minutes (around 30 mins), after which things go back to > normal again. I want to find out why HDFS becomes slow, what is the main > contributor of this latency and can I improve this behavior by changing some > configuration parameters. > > Thanks & Regards, > Rajat > -- Connect to me at http://www.facebook.com/dhruba
