Re: Region Server Hotspot/CPU Problem

2017-03-02 Thread Stack
Could try CMS GC; the thread local collection is less prominent when CMS is in place it seems (Try thread dumping and comparing to thread dumps posted to HBASE-17072 and related issues; the original poster did a nice job describing the problem). St.Ack On Wed, Mar 1, 2017 at 2:49 PM, Saad Mufti

Re: Region Server Hotspot/CPU Problem

2017-03-01 Thread Sean Busbey
Several of those jiras are fixed in later versions of CDH. Since the inclusion of jiras in packaging by particular vendors is a vendor specific issue, please seek out help from the vendor (e.g. on the community forum you've just mentioned). On Wed, Mar 1, 2017 at 8:49 AM, Saad Mufti

Re: Region Server Hotspot/CPU Problem

2017-03-01 Thread Saad Mufti
Someone in our team found this: http://community.cloudera.com/t5/Storage-Random-Access-HDFS/CPU-Usage-high-when-using-G1GC/td-p/48101 Looks like we're bitten by this bug. Unfortunately this is only fixed in HBase 1.4.0 so we'll have to undertake a version upgrade which is not trivial. -

Re: Region Server Hotspot/CPU Problem

2017-03-01 Thread Sudhir Babu Pothineni
First obvious thing to check is "major compaction" happening at the same time when it goes to 100% CPU? See this helps: https://community.hortonworks.com/articles/52616/hbase-compaction-tuning-tips.html Sent from my iPhone > On Mar 1, 2017, at 6:06 AM, Saad Mufti