It might be a larger scan (maybe gathering many data points for a metric) hitting many regions, in that case you'd see only a single region server being busy at a given time, since HBase scans only a region at a time for a single client scan.
A thread dump would give us a better idea. J-D specifically mentions OpenTSDB in that jira. -- Lars ________________________________ From: OpenSource Dev <[email protected]> To: [email protected]; lars hofhansl <[email protected]> Sent: Wednesday, September 11, 2013 8:59 PM Subject: Re: High cpu usage on a region server Hi Lars, All the read & write requests are equally distributed across all region-servers. If it is caused by the HBASE-9428 bug, any idea why it would impact only 1 reason server at a given time ? Thx On Wed, Sep 11, 2013 at 1:55 PM, lars hofhansl <[email protected]> wrote: > You might have run into HBASE-9428 > > -- Lars > > > > ________________________________ > From: OpenSource Dev <[email protected]> > To: [email protected] > Sent: Wednesday, September 11, 2013 1:49 PM > Subject: High cpu usage on a region server > > > Hi, > > I'm using HBase 0.94.6 (CDH 4.3) for Opentsdb. So far I have had no > issues with writes/puts. System is handles upto 800k puts per seconds > without issue. On average we do 250k puts per second. > > I am having the problem with Reads, I've also isolated where the > problem is but not been able to find the root cause. > > I have 16 machines running hbase-region server, each has ~35 regions. > Once in a while cpu goes flatout 80% in 1 region server. These are the > things i've noticed in ganglia: > > hbase.regionserver.request - evenly distributed. Not seeing any spikes > on the busy server > hbase.regionserver.blockCacheSize - between 500MB and 1000MB > hbase.regionserver.compactionQueueSize - avg 2 or less > hbase.regionserver.blockCacheHitRatio - 30% on busy node, >60% on other nodes > > > JVM Heap size is set to 16GB and I'm using -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC > > I've noticed the system load moves to a different region, sometimes > within a minute, if the busy region is restarted. > > Any suggestion what could be causing the load and/or what other > metrics should I check ? > > > Thank you!
