[
https://issues.apache.org/jira/browse/HBASE-21436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zephyr Guo updated HBASE-21436:
-------------------------------
Attachment: HBASE-21436-UT.patch
> Getting OOM frequently if hold many regions
> --------------------------------------------
>
> Key: HBASE-21436
> URL: https://issues.apache.org/jira/browse/HBASE-21436
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 3.0.0, 1.4.8, 2.0.2
> Reporter: Zephyr Guo
> Priority: Major
> Attachments: HBASE-21436-UT.patch
>
>
> Recently, some feedback reached me from a customer which complains about
> NotServingRegionException thrown out at intevals. I examined his cluster and
> found there were quite a lot of OOM logs there but metric
> "readDataPerSecondKB/writeDataPerSecondKB" is in quite low level. In this
> customer's case, each RS has 3k regions and heap size of 4G. I dumped heap
> when OOM took place, and found that a lot of Chunk objects (counts as much as
> 1700) was there.
> Eventually, piecing all these evidences together, I came to the conclusion
> that: 1. The root cause is that global flush is triggered by size of all
> memstores, rather than size of all chunks. 2. A chunk is always allocated for
> each region, even we only write a few data to the region.
> And in this case, a total of 3.4G memory was consumed by 1700 chunks,
> although throughput is very low.
> Although 3K regions is too much for RS with 4G memory, it is still wise to
> improve RS stability in such scenario (In fact, most customers buy a small
> size HBase on cloud side).
>
> I provide a patch (only contain UT) to reproduce this case (just send a
> batch).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
