Re: High CPU Utilization by meta region

We discovered that we were running the canary on all of our RegionServers. These were reading from meta and causing the CPU usage. When we stopped the canary, the CPU utilization on the RS that only contained the meta region looked like the usage on the other RegionServers. On Thu, Dec 1, 2016 at

Re: High CPU Utilization by meta region

On Thu, Dec 1, 2016 at 3:48 PM, Timothy Brown wrote: > > > > > > If you turn on RPC-level TRACE logging for a minute or so, anything about > > the client addresses that seems interesting? > > > Nothing seemed interesting to me but you may have a different opinion. > Here's

Re: High CPU Utilization by meta region

Thanks for the help. I've added some more responses inline. On Tue, Nov 29, 2016 at 9:51 PM, Stack wrote: > On Mon, Nov 28, 2016 at 10:25 AM, Timothy Brown > wrote: > > > Responses inlined. > > > > ... > > > > > > > > > What is the difference when you

Re: High CPU Utilization by meta region

On Mon, Nov 28, 2016 at 10:25 AM, Timothy Brown wrote: > Responses inlined. > > ... > > > > > > What is the difference when you compare servers? More requests? More > i/o? > > Thread dump the metadata server and let us see a link in here? (What you > > attached below is

Re: High CPU Utilization by meta region

Responses inlined. On Mon, Nov 28, 2016 at 12:45 PM, Stack wrote: > On Sun, Nov 27, 2016 at 6:53 PM, Timothy Brown > wrote: > > > Hi Everyone, > > > > I apologize for starting an additional thread about this but I wasn't > > subscribed to the users

Re: High CPU Utilization by meta region

On Sun, Nov 27, 2016 at 6:53 PM, Timothy Brown wrote: > Hi Everyone, > > I apologize for starting an additional thread about this but I wasn't > subscribed to the users mailing list when I sent the original and can't > figure out how to respond to the original :( > >

Re: High CPU Utilization by meta region

Hi Ted, The region server hosting hbase:meta only has the meta region on it so it has 1 region while other region servers can have more than 100 regions on them. I didn't notice anything interesting in the logs in my opinion. Is there anything in particular I should watch out for? The hbase:meta

Re: High CPU Utilization by meta region

Does the region server hosting hbase:meta have roughly the same number of regions as the other servers ? Did you find anything interesting in the server log (where hbase:meta is hosted) ? Have you tried major compacting the hbase:meta table ? In 1.2, DEFAULT_HBASE_META_VERSIONS is still 10. See

High CPU Utilization by meta region

Hi Everyone, I apologize for starting an additional thread about this but I wasn't subscribed to the users mailing list when I sent the original and can't figure out how to respond to the original :( Original Message: We are seeing about 80% CPU utilization on the Region Server that solely

Re: High CPU utilization by meta region

To add to what Stack asked, do you have the metrics for your META vs he other regions? Is the meta hot-spotted, which might create an increase on the CPU usage? Not just the requests per seconds, but also the number of calls. Does the META have way more? Or almost the same? Or less? thanks, JMS

Re: High CPU utilization by meta region

Can we see configs -- encodings? -- and a thread dump? Any I/O? If you look in HDFS, many files under hbase:meta? Is it big? When was last time it major compacted? Thanks, S On Mon, Nov 21, 2016 at 5:50 PM, Timothy Brown wrote: > Hi, > > We are seeing about 80% CPU

High CPU utilization by meta region

Hi, We are seeing about 80% CPU utilization on the Region Server that solely serves the meta table while other region servers typically have under 50% CPU utilization. Is this expected? Here's some more info about our cluster: HBase version 1.2 Number of regions: 72 Number of tables: 97 Approx.