[
https://issues.apache.org/jira/browse/HBASE-7728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13569272#comment-13569272
]
stack commented on HBASE-7728:
------------------------------
Looks like I just ran into this:
{code}
Java stack information for the threads listed above:
===================================================
"IPC Server handler 49 on 10304":
at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:1121)
- waiting to lock <0x0000000583d75420> (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.appendNoSync(HLog.java:1167)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2212)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1960)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3454)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)
"regionserver10304.logRoller":
at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1302)
- waiting to lock <0x0000000583d75438> (a java.lang.Object)
at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1279)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1432)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.cleanupCurrentWriter(HLog.java:865)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:646)
- locked <0x0000000583d75420> (a java.lang.Object)
at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
at java.lang.Thread.run(Thread.java:662)
"IPC Server handler 46 on 10304":
at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1312)
- waiting to lock <0x0000000583d75420> (a java.lang.Object)
- locked <0x0000000583d75438> (a java.lang.Object)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1436)
at
org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5176)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2232)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1960)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3454)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)
Found 1 deadlock.
{code}
This is a 0.94.3 or so.
> deadlock occurs between hlog roller and hlog syncer
> ---------------------------------------------------
>
> Key: HBASE-7728
> URL: https://issues.apache.org/jira/browse/HBASE-7728
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.2
> Environment: Linux 2.6.18-164.el5 x86_64 GNU/Linux
> Reporter: Wang Qiang
> Assignee: Ted Yu
> Priority: Blocker
> Fix For: 0.96.0, 0.94.5
>
> Attachments: 7728-0.94-simplified.txt, 7728-0.94.txt,
> 7728-0.94-v2.txt, 7728-suggest-0.96.txt, 7728-suggest.txt, 7728-v1.txt,
> 7728-v2.txt, 7728-v3.txt, 7728-v4.txt
>
>
> the hlog roller thread and hlog syncer thread may occur dead lock with the
> 'flushLock' and 'updateLock', and then cause all 'IPC Server handler' thread
> blocked on hlog append. the jstack info is as follow :
> "regionserver60020.logRoller":
> at
> org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1305)
> - waiting to lock <0x000000067bf88d58> (a java.lang.Object)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1283)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1456)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLog.cleanupCurrentWriter(HLog.java:876)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:657)
> - locked <0x000000067d54ace0> (a java.lang.Object)
> at
> org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> at java.lang.Thread.run(Thread.java:662)
> "regionserver60020.logSyncer":
> at
> org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1314)
> - waiting to lock <0x000000067d54ace0> (a java.lang.Object)
> - locked <0x000000067bf88d58> (a java.lang.Object)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1283)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1456)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1235)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
