[
https://issues.apache.org/jira/browse/HBASE-15950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-15950:
----------------------------------
Release Note:
The estimates of heap usage by the memstore objects (KeyValue, object and array
header sizes, etc) have been made more accurate for heap sizes up to 32G (using
CompressedOops), resulting in them dropping by 10-50% in practice. This also
results in less number of flushes and compactions due to "fatter" flushes.
YMMV. As a result, the actual heap usage of the memstore before being flushed
may increase by up to 100%. If configured memory limits for the region server
had been tuned based on observed usage, this change could result in worse GC
behavior or even OutOfMemory errors. Set the environment property (not
hbase-site.xml) "hbase.memorylayout.use.unsafe" to false to disable.
was:Made the object/array header sizes to be CompressedOops aware and fixed
object heap size calculations for important objects like KeyValue and
ConcurrentSkipListMap and . For heap sizes up to 32GB, depending on average
cell sizes and total memstore size, %10-50 reduction in memstore size and
flushes and compactions might be observed. YMMV. Due to more tighter than
before size estimates, total heap space is expected to be utilized more,
slightly increasing the chance to get OOM if misconfigured. Set the environment
property (not hbase-site.xml) "hbase.memorylayout.use.unsafe" to false to
disable.
> Fix memstore size estimates to be more tighter
> ----------------------------------------------
>
> Key: HBASE-15950
> URL: https://issues.apache.org/jira/browse/HBASE-15950
> Project: HBase
> Issue Type: Bug
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 2.0.0
>
> Attachments: Screen Shot 2016-06-02 at 8.48.27 PM.png,
> hbase-15950-v0.patch, hbase-15950-v1.patch
>
>
> While testing something else, I was loading a region with a lot of data.
> Writing 30M cells in 1M rows, with 1 byte values.
> The memstore size turned out to be estimated as 4.5GB, while with the JFR
> profiling I can see that we are using 2.8GB for all the objects in the
> memstore (KV + KV byte[] + CSLM.Node + CSLM.Index).
> This obviously means that there is room in the write cache that we are not
> effectively using.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
