I got JMX counters hooked up to JConsole (couple of them opened). Do you have any advice from your experience on what metrics I should focus on to spot this issue?
On Jul 9, 2012, at 22:19 PM, Stack wrote: > On Mon, Jul 9, 2012 at 8:35 PM, Asaf Mesika <[email protected]> wrote: >> Hi, >> >> My cluster started being incredibly slow in the past 2 days. >> I've seen many Blocking updates on the region server logs, which lead me to >> believe HDFS creates is the bottleneck. >> I ran a small test (hadoop fs -copyFromLocal big3_3Giga.tz.gz /tmp) which >> copies a 3.3G file, and I was surprised to see a rate of 11MB/sec, when it >> was previously around 28MB/sec. >> >> I have 3 RS computers, hosting DataNode and embedded zookeeper as well. >> And 1 Master computer, hosting NameNode as well. >> >> The slowest computer has Core 2 Duo 2.93GHz, 4GB. >> >> What steps can I take to find out the source of this slowdown? > > Install ganglia or opentsdb and figure what your cluster is up to when its > slow. > > St.Ack
