We are using HBase 1.0 on CDH 5.5.2 . We have taken great care to avoid
hotspotting due to inadvertent data patterns by prepending an MD5 based 4
digit hash prefix to all our data keys. This works fine most of the times,
but more and more (as much as once or twice a day) recently we have
occasions where one region server suddenly becomes "hot" (CPU above or
around 95% in various monitoring tools). When it happens it lasts for
hours, occasionally the hotspot might jump to another region server as the
master decide the region is unresponsive and gives its region to another

For the longest time, we thought this must be some single rogue key in our
input data that is being hammered. All attempts to track this down have
failed though, and the following behavior argues against this being
application based:

1. plotted Get and Put rate by region on the "hot" region server in
Cloudera Manager Charts, shows no single region is an outlier.

2. cleanly restarting just the region server process causes its regions to
randomly migrate to other region servers, then it gets new ones from the
HBase master, basically a sort of shuffling, then the hotspot goes away. If
it were application based, you'd expect the hotspot to just jump to another
region server.

3. have pored through region server logs and can't see anything out of the
ordinary happening

The only other pertinent thing to mention might be that we have a special
process of our own running outside the cluster that does cluster wide major
compaction in a rolling fashion, where each batch consists of one region
from each region server, and it waits before one batch is completely done
before starting another. We have seen no real impact on the hotspot from
shutting this down and in normal times it doesn't impact our read or write
performance much.

We are at our wit's end, anyone have experience with a scenario like this?
Any help/guidance would be most appreciated.


Reply via email to