[ 
https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eshcar Hillel updated HBASE-18294:
----------------------------------
    Release Note: 
A region is flushed if its memory component exceeds the region flush threshold.
A flush policy decides which stores to flush by comparing the size of the store 
to a column-family-flush threshold.
If the overall size of all memstores in the machine exceeds the bounds defined 
by the administrator (denoted global pressure) a region is selected and 
flushed. 
HBASE-18294 changes flush decisions to be based on heap-occupancy and not data 
(key-value) size, consistently across levels. This rolls back some of the 
changes by HBASE-16747. Specifically,
(1) RSs, Regions and stores track their overall on-heap and off-heap occupancy,
(2) A region is flushed when its on-heap+off-heap size exceeds the region flush 
threshold specified in hbase.hregion.memstore.flush.size,
(3) The store to be flushed is chosen based on its on-heap+off-heap size 
(4) At the RS level, a flush is triggered when the overall on-heap exceeds the 
on-heap limit, or when the overall off-heap size exceeds the off-heap limit 
(low/high water marks).

Note that when the region flush size is set to XXmb a region flush may be 
triggered even before writing keys and values of size XX because the total heap 
occupancy of the region which includes additional metadata exceeded the 
threshold.

  was:
A region is flushed if its memory component exceeds the region flush threshold.
A flush policy decides which stores to flush by comparing the size of the store 
to a column-family-flush threshold.
If the overall size of all memstores in the machine exceeds the bounds defined 
by the administrator (denoted global pressure) a region is selected and flush. 
HBASE-18294 changes flush decisions to be based on heap-occupancy and not data 
(key-value) size, consistently across levels. This rolls back some of the 
changes by HBASE-16747. Specifically,
(1) RSs, Regions and stores track their overall on-heap and off-heap occupancy,
(2) A region is flushed when its on-heap+off-heap size exceeds the region flush 
threshold,
(3) The store to be flushed is chosen based on its on-heap+off-heap size 
(4) At the RS level, a flush is triggered when the overall on-heap exceeds the 
on-heap limit, or when the overall off-heap size exceeds the off-heap limit 
(low/high water marks).

Note that when the region flush size is set to XXmb a region flush may be 
triggered even before writing keys and values of size XX because the total heap 
occupancy of the region which includes additional metadata exceeded the 
threshold.


> Reduce global heap pressure: flush based on heap occupancy
> ----------------------------------------------------------
>
>                 Key: HBASE-18294
>                 URL: https://issues.apache.org/jira/browse/HBASE-18294
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 3.0.0
>            Reporter: Eshcar Hillel
>            Assignee: Eshcar Hillel
>            Priority: Major
>             Fix For: 2.0.0-beta-2
>
>         Attachments: HBASE-18294.01.patch, HBASE-18294.01.patch, 
> HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.01.patch, 
> HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.02.patch, 
> HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch, 
> HBASE-18294.06.patch, HBASE-18294.07.patch, HBASE-18294.07.patch, 
> HBASE-18294.08.patch, HBASE-18294.09.patch, HBASE-18294.10.patch, 
> HBASE-18294.11.patch, HBASE-18294.11.patch, HBASE-18294.12.patch, 
> HBASE-18294.13.patch, HBASE-18294.15.patch, HBASE-18294.16.patch, 
> HBASE-18294.master.01.patch, HBASE-18294.master.01.patch
>
>
> A region is flushed if its memory component exceed a threshold (default size 
> is 128MB).
> A flush policy decides whether to flush a store by comparing the size of the 
> store to another threshold (that can be configured with 
> hbase.hregion.percolumnfamilyflush.size.lower.bound).
> Currently the implementation (in both cases) compares the data size 
> (key-value only) to the threshold where it should compare the heap size 
> (which includes index size, and metadata).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to