[
https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752297#comment-15752297
]
Hudson commented on HBASE-15787:
--------------------------------
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2137 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2137/])
HBASE-15787 Change the flush related heuristics to work with offheap
(ramkrishna: rev d1147eeb7e1d5f41161c7cf5bc5ddb4744ca5b57)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultHeapMemoryTuner.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HeapMemoryManager.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreChunkPool.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerAccounting.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsHeapMemoryManager.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/io/util/MemorySizeUtil.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerAccounting.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHeapMemoryManager.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
> Change the flush related heuristics to work with offheap size configured
> ------------------------------------------------------------------------
>
> Key: HBASE-15787
> URL: https://issues.apache.org/jira/browse/HBASE-15787
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-15787.patch, HBASE-15787_1.patch,
> HBASE-15787_4.patch, HBASE-15787_5.patch, HBASE-15787_6.patch,
> HBASE-15787_7.patch, HBASE-15787_8.patch, HBASE-15787_9.patch
>
>
> With offheap MSLAB in place we may have to change the flush related
> heuristics to work with offheap size configured rather than the java heap
> size.
> Since we now have clear seperation of the memstore data size and memstore
> heap size, for offheap memstore
> -> Decide if the global.offheap.memstore.size is breached for blocking
> updates and force flushes.
> -> If the onheap global.memstore.size is breached (due to heap overhead) even
> then block updates and force flushes.
> -> The global.memstore.size.lower.limit is now by default 95% of the
> global.memstore.size. So now we apply this 95% on the
> global.offheap.memstore.size and also on global.memstore.size (as it was done
> for onheap case).
> -> We will have new FlushTypes introduced
> {code}
> ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap
> memstore settings
> An offheap memstore can even breach the
> onheap_lower_mark*/
> ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap
> memstore settings
> An offheap memstore can even breach the
> onheap_higher_mark*/
> ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap
> memstore settings*/
> ABOVE_OFFHEAP_HIGHER_MARK;
> {code}
> -> regionServerAccounting does all the accounting.
> -> HeapMemoryTuner is what is litte tricky here. First thing to note is that
> at no point it will try to increase or decrease the
> global.offheap.memstore.size. If there is a heap pressure then it will try to
> increase the memstore heap limit.
> In case of offheap memstore there is always a chance that the heap pressure
> does not increase. In that case we could ideally decrease the heap limit for
> memstore. The current logic of heapmemory tuner is such that things will
> naturally settle down. But on discussion what we thought is let us include
> the flush count that happens due to offheap pressure but give that a lesser
> weightage and thus ensure that the initial decrease on memstore heap limit
> does not happen. Currently that fraction is set as 0.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
