[
https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16259239#comment-16259239
]
Eshcar Hillel commented on HBASE-18294:
---------------------------------------
With the new property the user is not obligated to set flush size or heap flush
size -- both of them are optional and will take the default value if not set by
the application.
Other than that, why is it so unreasonable to have 2 thresholds, after all we
have 2 parameters to set the heap size used by the application: (1) JVM heap:
-Xmx, (2) off-heap -XX:MaxDirectMemorySize.
Consider a case in the future when off-heap is fully supported the admin
decides to allocate 100GB off-heap and only 1.6GB on-heap and assume the
machine has about 800 regions. We would like to allow the admin to set
different thresholds on the occupancy of each region on- and off-heap, say 2MB
and 128MB, respectively.
It is true that we have global heap size as an upper limit but this is just as
a safe guard.
Reaching global pressure too frequently means the RS is not "healthy". Do we
agree on this??
We have region level threshold to trigger the vast majority of flushes. And
from what I have noticed in the benchmarks I did this is not always the case.
I plan to do an additional write-only experiment to count number of times we
reach global pressure. The goal of the current patch is to reduce this number.
Let me get back when I have the numbers.
> Reduce global heap pressure: flush based on heap occupancy
> ----------------------------------------------------------
>
> Key: HBASE-18294
> URL: https://issues.apache.org/jira/browse/HBASE-18294
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Eshcar Hillel
> Assignee: Eshcar Hillel
> Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch,
> HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch,
> HBASE-18294.06.patch
>
>
> A region is flushed if its memory component exceed a threshold (default size
> is 128MB).
> A flush policy decides whether to flush a store by comparing the size of the
> store to another threshold (that can be configured with
> hbase.hregion.percolumnfamilyflush.size.lower.bound).
> Currently the implementation (in both cases) compares the data size
> (key-value only) to the threshold where it should compare the heap size
> (which includes index size, and metadata).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
