[
https://issues.apache.org/jira/browse/HBASE-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15046526#comment-15046526
]
Stephen Yuan Jiang commented on HBASE-14943:
--------------------------------------------
+1 on the patch.
This is a good found. Now I know why sometimes the procedure hangs during
system tests.
{noformat}
Found one Java-level deadlock:
=============================
"ProcedureExecutor-0":
waiting for ownable synchronizer 0x00000004410fa5b8, (a
java/util/concurrent/locks/ReentrantLock$NonfairSync),
which is held by "WALProcedureStoreSyncThread"
"WALProcedureStoreSyncThread":
waiting to lock Monitor@0x00007f90f0847de8 (Object@0x00000004410f9ca8, a
org/apache/hadoop/hbase/procedure2/store/ProcedureStoreTracker),
which is held by "ProcedureExecutor-0"
{noformat}
For the "WALProcedureStoreSyncThread" - syncLoop() holds "lock.lock()" and wait
for "synchronized (storeTracker)" by calling periodicRoll().
For the "ProcedureExecutor-0" - insert/delete/update holds the "synchronized
(storeTracker)", then checkAndTryRoll() calls rollWriter(), which trying to do
"lock.lock()".
> Deadlock in WALProcedureStore
> -----------------------------
>
> Key: HBASE-14943
> URL: https://issues.apache.org/jira/browse/HBASE-14943
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
> Affects Versions: 2.0.0, 1.2.0, 1.1.3
> Reporter: Ashu Pachauri
> Assignee: Matteo Bertozzi
> Priority: Blocker
> Attachments: HBASE-14943-v0.patch, deadlock-jstack.out
>
>
> Found deadlock in WALProcedureStore between the WALProcedureSyncThread and
> WALProcedureStore#update/delete/insert API which is used by the
> ProcedureExecutor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
