[
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389043#comment-15389043
]
ramkrishna.s.vasudevan commented on HBASE-14921:
------------------------------------------------
Did the testing with compacting memstore.
Based on our write path offheaping work we had arrived at certain config params
and gc tuning.
Onheap case : Xmx : 32 G
Offheap case Xmx : 12 G
Offheap memory for memstore : 32 G
global memstore size : 0.42
GC - G1GC
"-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=60
-XX:G1HeapWastePercent=20 -XX:G1MixedGCCountTarget=8"
PE : ./hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred
--presplit=50 --size=150 --columns=50 --valueSize=200 --writeToWAL=false
--bloomFilter=NONE --inmemoryCompaction=false/true randomWrite 50
We were able to run and get results for the default memstore (onheap), default
memstore(offheap), Pipelinememstore (Internal impl where all segments are
flushed with no in memory compaction) - flatten to ARRAY_MAP and flatten to
CHUNK_MAP.
But with compacting memstore we are not able to run and get results as with the
offheap case 12G is just not enough because we have duplicates while doing in
memory compaction. The segments are read once again to create a compacted
segment and it is creating lot of garbage and more memory need.
With onheap case I increased the heap size to 50G (as with 32G I was not able
to run) still the compaction was taking lot of memory and most importantly the
disk flushes and disk compactions were getting piled up and finally got 'Too
many open files' exception.
So I copied the latest trunk patch and gave a run and it had other exception
due to invalid region size settings as it was going negative.
{code}
2016-07-22 16:14:51,066 ERROR
[RpcServer.FifoWFPBQ.default.handler=60,queue=0,port=16041-inmemoryCompactions-1469184056544]
regionserver.HRegion: Asked to modify this region's
(TestTable,00000000000000000005033136,1469184040624.26d47b5614da8885fd0768b66a5d6457.)
memstoreSize to a negative value which is incorrect. Current
memstoreSize=33534312, delta=-33570288
java.lang.Exception
at
org.apache.hadoop.hbase.regionserver.HRegion.addAndGetGlobalMemstoreSize(HRegion.java:1125)
at
org.apache.hadoop.hbase.regionserver.RegionServicesForStores.addAndGetGlobalMemstoreSize(RegionServicesForStores.java:72)
at
org.apache.hadoop.hbase.regionserver.CompactionPipeline.swap(CompactionPipeline.java:111)
at
org.apache.hadoop.hbase.regionserver.CompactingMemStore.swapCompactedSegments(CompactingMemStore.java:208)
at
org.apache.hadoop.hbase.regionserver.MemStoreCompactor.doCompaction(MemStoreCompactor.java:173)
at
org.apache.hadoop.hbase.regionserver.MemStoreCompactor.start(MemStoreCompactor.java:114)
at
org.apache.hadoop.hbase.regionserver.CompactingMemStore.flushInMemory(CompactingMemStore.java:311)
at
org.apache.hadoop.hbase.regionserver.CompactingMemStore$InMemoryFlushRunnable.run(CompactingMemStore.java:384)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{code}
So in order to avoid compaction I just ensure that only flattening happens
(because in the default case that is what you are trying to do). Even here the
problem is that since only the flush happens on the tail part we keep
pressurizing the memstore and we have lot of blocked regions. Also the tail
flush is creating lot of small files and this is leading to more compactions
and hence again getting 'Too many file open' exception.
When i did the above change to the offheap memstore (allow only flattening)
case the problem is similar but here the tests are taking longer time to
complete as every time only a small portion of the memstore is getting flushed
out.
So we feel that if we go only with flattening then we should ensure that entire
segments in pipeline has to be flushed.
> Memory optimizations
> --------------------
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
> Issue Type: Sub-task
> Affects Versions: 2.0.0
> Reporter: Eshcar Hillel
> Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf,
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch,
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch,
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch,
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf,
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap
> allocations
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
