You might have run into HBASE-9428 -- Lars
________________________________ From: OpenSource Dev <[email protected]> To: [email protected] Sent: Wednesday, September 11, 2013 1:49 PM Subject: High cpu usage on a region server Hi, I'm using HBase 0.94.6 (CDH 4.3) for Opentsdb. So far I have had no issues with writes/puts. System is handles upto 800k puts per seconds without issue. On average we do 250k puts per second. I am having the problem with Reads, I've also isolated where the problem is but not been able to find the root cause. I have 16 machines running hbase-region server, each has ~35 regions. Once in a while cpu goes flatout 80% in 1 region server. These are the things i've noticed in ganglia: hbase.regionserver.request - evenly distributed. Not seeing any spikes on the busy server hbase.regionserver.blockCacheSize - between 500MB and 1000MB hbase.regionserver.compactionQueueSize - avg 2 or less hbase.regionserver.blockCacheHitRatio - 30% on busy node, >60% on other nodes JVM Heap size is set to 16GB and I'm using -XX:+UseParNewGC -XX:+UseConcMarkSweepGC I've noticed the system load moves to a different region, sometimes within a minute, if the busy region is restarted. Any suggestion what could be causing the load and/or what other metrics should I check ? Thank you!
