Deadlock in MemStore flusher due to JDK internally synchronizing on current
thread
----------------------------------------------------------------------------------
Key: HBASE-4367
URL: https://issues.apache.org/jira/browse/HBASE-4367
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Critical
Fix For: 0.92.0
We observed a deadlock in production between the following threads:
- IPC handler thread holding the monitor lock on MemStoreFlusher inside
reclaimMemStoreMemory, waiting to obtain MemStoreFlusher.lock (the reentrant
lock member)
- cacheFlusher thread inside flushRegion holds MemStoreFlusher.lock, and then
calls PriorityCompactionQueue.add, which calls
PriorityCompactionQueue.addToRegionsInQueue, which calls
CompactionRequest.toString(), which calls Date.toString. If this occurs just
after a GC under memory pressure, Date.toString needs to reload locale
information (stored in a soft reference), so it calls
ResourceBundle.loadBundle, which uses Thread.currentThread() as a synchronizer
(see sun bug http://bugs.sun.com/view_bug.do?bug_id=6915621). Since the current
thread is the MemStoreFlusher itself, we have a lock order inversion and a
deadlock.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
