[
https://issues.apache.org/jira/browse/HBASE-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15619926#comment-15619926
]
Hudson commented on HBASE-16747:
--------------------------------
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1882 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1882/])
HBASE-16747 Track memstore data size and heap overhead separately.
(anoopsamjohn: rev ba6d95232401ce533fb8c121ade7f4a864d06f12)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerAccounting.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushLargeStoresPolicy.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractMemStore.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWalAndCompactingMemStoreFlush.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java
* (edit)
hbase-common/src/main/java/org/apache/hadoop/hbase/SizeCachedKeyValue.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHMobStore.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/OffheapKeyValue.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HeapMemoryManager.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java
* (edit)
hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFlusher.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreFlusher.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Region.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingMemStore.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionWrapperImpl.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreChunkPool.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactingMemStore.java
* (add)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemstoreSize.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MutableSegment.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestPerColumnFamilyFlush.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransaction.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/AbstractTestWALReplay.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingToCellArrayMapMemStore.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/ExtendedCell.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServicesForStores.java
> Track memstore data size and heap overhead separately
> ------------------------------------------------------
>
> Key: HBASE-16747
> URL: https://issues.apache.org/jira/browse/HBASE-16747
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
> Reporter: Anoop Sam John
> Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16747.patch, HBASE-16747.patch,
> HBASE-16747_V2.patch, HBASE-16747_V2.patch, HBASE-16747_V3.patch,
> HBASE-16747_V3.patch, HBASE-16747_V3.patch, HBASE-16747_V4.patch,
> HBASE-16747_WIP.patch
>
>
> We track the memstore size in 3 places.
> 1. Global at RS level in RegionServerAccounting. This tracks all memstore's
> size and used to calculate whether forced flushes needed because of global
> heap pressure
> 2. At region level in HRegion. This is sum of sizes of all memstores within
> this region. This is used to decide whether region reaches flush size (128 MB)
> 3. Segment level. This tracks the in memory flush/compaction decisions.
> All these use the Cell's heap size which include the data bytes# as well as
> Cell object heap overhead. Also we include the overhead because of addition
> of Cells into Segment's data structures (Like CSLM).
> Once we have off heap memstore, we will keep the cell data bytes in off heap
> area. So we can not track both data size and heap overhead as one entity. We
> need to separate them and track.
> Proposal here is to track both cell data size and heap overhead separately at
> global accounting layer. As of now we have only on heap memstore. So the
> global memstore boundary checks will consider both (adds up and check against
> global max memstore size)
> Track cell data size alone (This can be on heap or off heap) in region level.
> Region flushes use cell data size alone for the region flush decision. A
> user configuring 128 MB as flush size, normally he will expect to get a 128MB
> data flush size. But as we were including the heap overhead also, once the
> flush happens, the actual data size getting flushed is way behind this 128
> MB. Now with this change we will behave more like what a user thinks.
> Segment level in memory flush/compaction also considers cell data size alone.
> But we will need to track the heap overhead also. (Once the in memory flush
> or normal flush happens, we will have to adjust both cell data size and heap
> overhead)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
