[
https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16304514#comment-16304514
]
Eshcar Hillel commented on HBASE-18294:
---------------------------------------
My previous solution had three different counters in MemStoreSize: (1) one for
data size (2) one for on-heap size (3) one for off-heap size.
The main reason I rolled back this change is the understanding that in the near
future HBase is going to support only clusters that are either entirely on-heap
or entirely off-heap and so there is no need to measure on- and off-heap
occupancy separately in any level (cluster, RS, region or memstore).
This means that when we say heap: heap size, heap occupancy, heap space, etc.
we mean all bytes allocated on the heap including JVM and OS heap.
If an implementation of the {{Cell}} class the method {{heapSize()}} does not
return the number of bytes allocated to store data that is related to this cell
(key, value, metadata, etc.) then I would say the implementation does not
satisfy the specification of the method and needs to be changed.
Is there any reason the method was implemented this way??
How about the following change adding {{KeyValueUtil.length(this)}} to the
computation in the implementation of {{ByteBufferKV}}
I think it will fix the problem. Do you see any reason it shouldn't work?
{code}
public long heapSize() {
if (this.buf.hasArray()) {
return ClassSize.align(FIXED_OVERHEAD + length);
}
return ClassSize.align(FIXED_OVERHEAD) + KeyValueUtil.length(this);
}
{code}
> Reduce global heap pressure: flush based on heap occupancy
> ----------------------------------------------------------
>
> Key: HBASE-18294
> URL: https://issues.apache.org/jira/browse/HBASE-18294
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Eshcar Hillel
> Assignee: Eshcar Hillel
> Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch,
> HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch,
> HBASE-18294.06.patch, HBASE-18294.07.patch, HBASE-18294.07.patch,
> HBASE-18294.08.patch, HBASE-18294.09.patch, HBASE-18294.10.patch,
> HBASE-18294.11.patch, HBASE-18294.11.patch, HBASE-18294.12.patch,
> HBASE-18294.13.patch
>
>
> A region is flushed if its memory component exceed a threshold (default size
> is 128MB).
> A flush policy decides whether to flush a store by comparing the size of the
> store to another threshold (that can be configured with
> hbase.hregion.percolumnfamilyflush.size.lower.bound).
> Currently the implementation (in both cases) compares the data size
> (key-value only) to the threshold where it should compare the heap size
> (which includes index size, and metadata).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
