[ https://issues.apache.org/jira/browse/HBASE-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135116#comment-13135116 ]
Hudson commented on HBASE-4367: ------------------------------- Integrated in HBase-0.92 #79 (See [https://builds.apache.org/job/HBase-0.92/79/]) HBASE-4367 Deadlock in MemStore flusher due to JDK internally synchronizing on current thread stack : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/Chore.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/Leases.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HasThread.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/JVMClusterUtil.java > Deadlock in MemStore flusher due to JDK internally synchronizing on current > thread > ---------------------------------------------------------------------------------- > > Key: HBASE-4367 > URL: https://issues.apache.org/jira/browse/HBASE-4367 > Project: HBase > Issue Type: Bug > Components: regionserver > Affects Versions: 0.90.4 > Reporter: Todd Lipcon > Assignee: Todd Lipcon > Priority: Critical > Fix For: 0.92.0 > > Attachments: 4367.txt, hbase-4367.txt > > > We observed a deadlock in production between the following threads: > - IPC handler thread holding the monitor lock on MemStoreFlusher inside > reclaimMemStoreMemory, waiting to obtain MemStoreFlusher.lock (the reentrant > lock member) > - cacheFlusher thread inside flushRegion holds MemStoreFlusher.lock, and then > calls PriorityCompactionQueue.add, which calls > PriorityCompactionQueue.addToRegionsInQueue, which calls > CompactionRequest.toString(), which calls Date.toString. If this occurs just > after a GC under memory pressure, Date.toString needs to reload locale > information (stored in a soft reference), so it calls > ResourceBundle.loadBundle, which uses Thread.currentThread() as a > synchronizer (see sun bug http://bugs.sun.com/view_bug.do?bug_id=6915621). > Since the current thread is the MemStoreFlusher itself, we have a lock order > inversion and a deadlock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira