A server started getting busy last night, but this time it took ~5 hrs to get from 15% busy to 75% busy. It is not running 80% flat-out yet. But this is still very high compared to other servers that are running under ~25% cpu usage. Only change that I made yesterday was the addition of "-XX:+UseParNewGC" to hbase startup command.
http://pastebin.com/VRmujgyH On Wed, Sep 11, 2013 at 2:28 PM, Stack <[email protected]> wrote: > Can you thread dump the busy server and pastebin it? > Thanks, > St.Ack > > > On Wed, Sep 11, 2013 at 1:49 PM, OpenSource Dev > <[email protected]>wrote: > >> Hi, >> >> I'm using HBase 0.94.6 (CDH 4.3) for Opentsdb. So far I have had no >> issues with writes/puts. System is handles upto 800k puts per seconds >> without issue. On average we do 250k puts per second. >> >> I am having the problem with Reads, I've also isolated where the >> problem is but not been able to find the root cause. >> >> I have 16 machines running hbase-region server, each has ~35 regions. >> Once in a while cpu goes flatout 80% in 1 region server. These are the >> things i've noticed in ganglia: >> >> hbase.regionserver.request - evenly distributed. Not seeing any spikes >> on the busy server >> hbase.regionserver.blockCacheSize - between 500MB and 1000MB >> hbase.regionserver.compactionQueueSize - avg 2 or less >> hbase.regionserver.blockCacheHitRatio - 30% on busy node, >60% on other >> nodes >> >> >> JVM Heap size is set to 16GB and I'm using -XX:+UseParNewGC >> -XX:+UseConcMarkSweepGC >> >> I've noticed the system load moves to a different region, sometimes >> within a minute, if the busy region is restarted. >> >> Any suggestion what could be causing the load and/or what other >> metrics should I check ? >> >> >> Thank you! >>
