[
https://issues.apache.org/jira/browse/HBASE-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13822308#comment-13822308
]
Liang Xie commented on HBASE-8163:
----------------------------------
I did a apple-to-apple test with this patch. Here is the test result just for
refer:
Env: Xmx=Xms=8G,Xmn1G, memestore lower/upper limit = 0.2/0.3
each test began with empty table, then write 20million records each with three
fields, and each field has 200 bytes:
1)original config(-XX:PretenureSizeThreshold=4m)
YGC YGCT FGC FGCT GCT
6970 318.592 8 0.884 319.476
2)set hbase.hregion.memstore.mslab.chunksize to
4194320(-XX:PretenureSizeThreshold=4m)
YGC YGCT FGC FGCT GCT
6973 253.891 8 0.522 254.413
3)set -XX:PretenureSizeThreshold=2097088(hbase.hregion.memstore.mslab.chunksize
is by default 2M)
6960 260.642 8 1.427 262.069
4)set hbase.hregion.memstore.chunkpool.maxsize=0.6, means enable
MemStoreChunkPool feature(log said maxCount=706),
-XX:PretenureSizeThreshold=2097088(hbase.hregion.memstore.mslab.chunksize is by
default 2M)
7028 258.598 2 0.401 258.999
To me, this MemStorechunkPool feature is useful for heavy FGC scenarios caused
by write request. If the YGC is the big hurt by write request, personally i
recommend to config "hbase.hregion.memstore.mslab.chunksize" or
"-XX:PretenureSizeThreshold", considering risk:)
Hope this test result will be helpful for some guys
> MemStoreChunkPool: An improvement for JAVA GC when using MSLAB
> --------------------------------------------------------------
>
> Key: HBASE-8163
> URL: https://issues.apache.org/jira/browse/HBASE-8163
> Project: HBase
> Issue Type: New Feature
> Components: Performance, regionserver
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.98.0, 0.95.0
>
> Attachments: hbase-0.95-8163v6.patch, hbase-8163v1.patch,
> hbase-8163v2.patch, hbase-8163v3.patch, hbase-8163v4.patch,
> hbase-8163v5.patch, hbase-8163v6.patch
>
>
> *Usage:*
> Disable pool(default): configure "hbase.hregion.memstore.chunkpool.maxsize"
> as 0
> Enable pool: configure "hbase.hregion.memstore.chunkpool.maxsize" as a
> percentage of global memstore size (between 0.0 and 1.0, recommend to set it
> as the gap between min and max sizes of memstore, e.g. 0.5)
> *Background*：
> When we use mslab,we will copy the keyvalue together in a structure called
> *MemStoreLAB$Chunk*, therefore we could decrease the heap fragment.
> *Problem*：
> When one chunk is full, we will create a new chunk, and then the old chunk
> will be reclaimed by JVM if no reference to it.
> Mostly the chunk object would be promoted when doing Young GC, cause
> increasing the cost of YGC
> When a Chunk object has no reference? It should meet two following condition:
> 1.Memstore which this chunk belongs to is flushed
> 2.No scanner is opening on the memstore which this chunk belongs to
> *Solution:*
> 1.Create a chunk pool to manage the no-reference chunks, instead of being
> reclaimed by JVM
> 2.When a Chunk has no reference, put it back to the pool
> 3.The pool has a max capacity, it will skip the chunks when achieve the max
> size
> 4.When we need new Chunk to store KeyValue, get it from the pool if exists,
> else create new one by pool, so we could be able to reuse the old chunks
> *Test results:*
> Environment:
> hbase-version:0.94
> -Xms4G -Xmx4G -Xmn2G
> Row size=50 bytes, Value size=1024 bytes
> 50 concurrent theads per client, insert 10,000,000 rows
> Before:
> Avg write request per second:12953
> After testing, final result of jstat -gcutil :
> YGC YGCT FGC FGCT GCT
> 747 36.503 48 2.492 38.995
> After:
> Avg write request per second:14025
> After testing, final result of jstat -gcutil :
> YGC YGCT FGC FGCT GCT
> 711 20.344 4 0.284 20.628
> *Improvement: YGC 40+%; WPS 5%+*
> review board :
> https://reviews.apache.org/r/10056/
--
This message was sent by Atlassian JIRA
(v6.1#6144)
