that kind of messages may happen when there were queries that utilize memory manager (usually joins and group by) and they were timed out or failed due to some reason. So the message itself is hardly related to CPU usage or GC. BUT. That may mean that your region servers are unable to handle properly such kind of workload. Since you say that this issue started after Yarn work I would suggest checking swappiness and huge pages (there are quite a lot of resources over the Internet how they affect HBase). It might be the case when you just run out of HW resources.
Thanks, Sergey On Wed, Jan 31, 2018 at 6:40 PM, Jins George <jins.geo...@aeris.net> wrote: > Hi, > > On analyzing a prod issue of High CPU usage on Hbase Region server, I came > across warning messages from region server logs complaining about Orphaned > chunk of memory. > > 2018-01-30 19:16:31,565 WARN org.apache.phoenix.memory.GlobalMemoryManager: > Orphaned chunk of 104000 bytes found during finalize > 2018-01-30 19:16:31,565 WARN org.apache.phoenix.memory.GlobalMemoryManager: > Orphaned chunk of 104000 bytes found during finalize > > > The high CPU usage looks like due to garbage collection and it lasted for > almost 6 hours. And throughout 6 hours, region server logs had these > warning messages logged. > > Cluster Details: > 4 node( 1 master + 3 slaves) cdh cluster > Hbase version 1.2 > Phoenix version 4.7 > Region Server Heap : 4G > Total Regions: ~135 > Total tables : ~35 > > Out of 3 region servers, 2 of them had the warning logs and both suffered > high CPU. Third region server nither had High CPU nor the warning logs. Any > idea why these messages are logged and can that trigger continuous GC ? > > Before this issue started( or around the same time) huge application log > files were copied to HDFS by Yarn.. But can't think of that causing issue > on Hbase Region server. > > Any help is appreciated. > > Thanks, > Jins George >
