Hi Sandeep,
Can you take a jstack of the RegionServer process when this happens the next
time?(DebugDump on the RegionServer UI page, or log into the box and run jstack
yourself)
Otherwise we'll be guessing :)
-- Lars
From: Sandeep Reddy <[email protected]>
To: "[email protected]" <[email protected]>
Sent: Thursday, July 7, 2016 11:06 PM
Subject: Very High CPU usage in RegionServer
Hi,
We are observing very high CPU load(400 to 600%) in one of our RegionServer to
the point where the machine is becoming unresponsive.
At this point the whole cluster of 20+ RegionServers becoming unresponsive.
Before cluster becomes unresponsive we observed following symptoms:
* Huge bandwidth spike
* CPU spikes vertically form normal load to very high usage only in one
RegionServer
* Few times even though machine is unresponsive, it sending heartbeats to
master
* There is no spike in number of requests to HBase
* We are observed this pattern at least twice is last week
* We don't have any co-processors in any of the region servers
What could be the possible reasons for this kind of behaviour.
We are using hbase-0.98.7, hadoop-2.5.1 versions.
Its production cluster so upgrading to latest version will not be possible
right away.
Thanks,
Sandeep.
Thanks,
Sandeep.
