[
https://issues.apache.org/jira/browse/HBASE-15999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331611#comment-15331611
]
Eshcar Hillel commented on HBASE-15999:
---------------------------------------
patch _4 looks correct to me, unless any other issues come up.
+1
> NPE in MemstoreCompactor
> ------------------------
>
> Key: HBASE-15999
> URL: https://issues.apache.org/jira/browse/HBASE-15999
> Project: HBase
> Issue Type: Sub-task
> Affects Versions: 2.0.0
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15999.patch, HBASE-15999_1.patch,
> HBASE-15999_2.patch, HBASE-15999_3.patch, HBASE-15999_4.patch
>
>
> In the INMEMORY_COMPACTION_POOL ThreadPoolExecutor for every thread the
> current thread name is also appended. Since we are using a pool the name gets
> appended and we end up in names like this
> {code}
> "B.defaultRpcServer.handler=89,queue=9,port=16041-inmemoryCompactions-1465492533442-inmemoryCompactions-1465492548754-inmemoryCompactions-1465492548913-inmemoryCompactions-1465492549625-inmemoryCompactions-1465492549956-inmemoryCompactions-1465492567040-inmemoryCompactions-1465492567160-inmemoryCompactions-1465492578465-inmemoryCompactions-1465492578707-inmemoryCompactions-1465492579292-inmemoryCompactions-1465492579357-inmemoryCompactions-1465492579786-inmemoryCompactions-1465492580059-inmemoryCompactions-1465492589975-inmemoryCompactions-1465492590192-inmemoryCompactions-1465492590484-inmemoryCompactions-1465492591144-inmemoryCompactions-1465492592603-inmemoryCompactions-1465492592799-inmemoryCompactions-1465492597106-inmemoryCompactions-1465492602925-inmemoryCompactions-1465492606620-inmemoryCompactions-1465492651478-inmemoryCompactions-1465492653460-inmemoryCompactions-1465492677020-inmemoryCompactions-1465492680857-inmemoryCompactions-1465492681989-inmemoryCompactions-1465492721818-inmemoryCompactions-1465492723562-inmemoryCompactions-1465492724801-inmemoryCompactions-1465492726665-inmemoryCompactions-1465492745750-inmemoryCompactions-1465492745964-inmemoryCompactions-1465492746578-inmemoryCompactions-1465492756867-inmemoryCompactions-1465492764727-inmemoryCompactions-1465492766944-inmemoryCompactions-1465492767098-inmemoryCompactions-1465492785298-inmemoryCompactions-1465492788334-inmemoryCompactions-1465492795954-inmemoryCompactions-1465493047265-inmemoryCompactions-1465493091530-inmemoryCompactions-1465493185684"
> #6006 daemon prio=5 os_prio=0 tid=0x000000000daa6800 nid=0x454a runnable
> [0x00007f50fd0b9000]
> {code}
> As we were surprised to see why so many threads are getting created as Anoop
> pointed out the pool size is 10 and there is no setting for the thread to
> die, the reason for this issue is that we have an multi threaded issue in
> MemstoreCompactor. The memstoreCompactor has the StoreScanner and
> MemstoreScanner as the state variable and every time we just instantiate a
> new one when a new inmemory flush request comes. Finally we try to release
> the resource where the scanner is nullified and closed. But the instance
> would have already been updated or nullified by another thread when there are
> multiple requests. So this causes an NPE in releaseResources.
> {code}
> Exception in thread
> "B.defaultRpcServer.handler=76,queue=6,port=16041-inmemoryCompactions-1465906554314"
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.MemStoreCompactor.releaseResources(MemStoreCompactor.java:108)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreCompactor.doCompaction(MemStoreCompactor.java:144)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreCompactor.startCompaction(MemStoreCompactor.java:88)
> at
> org.apache.hadoop.hbase.regionserver.CompactingMemStore.flushInMemory(CompactingMemStore.java:287)
> at
> org.apache.hadoop.hbase.regionserver.CompactingMemStore$InMemoryFlushRunnable.run(CompactingMemStore.java:356)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> {code}
> So every time the thread dies and a new is created which I found from the
> stack traces by adding some logs. So this is why we were creating lot of
> threads and the name was simply getting appened. Now from this we can see
> that adding the Handler name is fine but the main issue is the NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
