[
https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16305325#comment-16305325
]
Eshcar Hillel commented on HBASE-18294:
---------------------------------------
Thanks [~stack] for the document, I think its a good summary of the issue.
I think it is important to have symmetric behavior for on-heap and off-heap
clusters. Even if users do special tuning for off-heap clusters it is not
reasonable to expect them understand that on-heap flushes when total occupancy
exceeds 128MB and off-heap flushes when key-values are exceeding 128MB.
I would like to avoid checking heapsize >128MB OR datasize > 128MB since this
is exactly defining unsymmetrical behavior for on-heap and off-heap clusters.
I understand also that [~anoop.hbase] and [~ram_krish] are objecting to change
the implementation of {{heapSize()}} so it accounts also for off-heap size,
either because they feel it is confusing or because they think it will cause
some real problems in the off-heap implementation
So here is a suggestion:
(1) I will change the name of the counter in {{MemStoreSize}} from {{heapSize}}
to something like {{totalAllocationSize}} or {{totalOccupancy}}
(2) I will add methods {{estimatedTotalOccupancyOf(final Cell cell)}} and
{{totalOccupancy()}} to compute total occupancy of data+metadata no matter were
they are allocated java-heap or off-heap (the way I suggested above) and will
use them to update the counters in {{MemStoreSize}}
(3) only changing region level flush decision, not changing RS global
accounting and decisions
If this is acceptable, I can make a new patch in a day or two.
> Reduce global heap pressure: flush based on heap occupancy
> ----------------------------------------------------------
>
> Key: HBASE-18294
> URL: https://issues.apache.org/jira/browse/HBASE-18294
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Eshcar Hillel
> Assignee: Eshcar Hillel
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch,
> HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch,
> HBASE-18294.06.patch, HBASE-18294.07.patch, HBASE-18294.07.patch,
> HBASE-18294.08.patch, HBASE-18294.09.patch, HBASE-18294.10.patch,
> HBASE-18294.11.patch, HBASE-18294.11.patch, HBASE-18294.12.patch,
> HBASE-18294.13.patch
>
>
> A region is flushed if its memory component exceed a threshold (default size
> is 128MB).
> A flush policy decides whether to flush a store by comparing the size of the
> store to another threshold (that can be configured with
> hbase.hregion.percolumnfamilyflush.size.lower.bound).
> Currently the implementation (in both cases) compares the data size
> (key-value only) to the threshold where it should compare the heap size
> (which includes index size, and metadata).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
