[
https://issues.apache.org/jira/browse/HBASE-27919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17760920#comment-17760920
]
Haiping lv commented on HBASE-27919:
------------------------------------
Setup a 5 slave nodes cluster, and run with a 10B ITBLL test.
HMaster encountered a deadlock.
The handler thread of HMaster holds the synchronized lock of SnapshotManager
and competes for the readLock of takingSnapshotLock.
{code:java}
Thread 57 (RpcServer.default.FPBQ.Fifo.handler=22,queue=1,port=16000):
State: WAITING
Blocked count: 14
Waited count: 206
Waiting on java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@a7a58d4
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.takeSnapshot(SnapshotManager.java:675)
org.apache.hadoop.hbase.master.MasterRpcServices.snapshot(MasterRpcServices.java:1765)
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:437)
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82) {code}
The Master Snapshot cleaning thread holds the writeLock of takingSnapshotLock
to compete for the synchronized lock of SnapshotManager.
{code:java}
Thread 5405 (hfile_cleaner-dir-scan-pool-2):
State: BLOCKED
Blocked count: 50535
Waited count: 92200
Blocked on org.apache.hadoop.hbase.master.snapshot.SnapshotManager@21a16019
Blocked by 57 (RpcServer.default.FPBQ.Fifo.handler=22,queue=1,port=16000)
Stack:
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isTakingAnySnapshot(SnapshotManager.java:655)
org.apache.hadoop.hbase.master.snapshot.SnapshotFileCache.getUnreferencedFiles(SnapshotFileCache.java:197)
org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner.getDeletableFiles(SnapshotHFileCleaner.java:78)
org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:375)
org.apache.hadoop.hbase.master.cleaner.CleanerChore.lambda$traverseAndDelete$1(CleanerChore.java:483)
org.apache.hadoop.hbase.master.cleaner.CleanerChore$$Lambda$788/1121007598.act(Unknown
Source)
org.apache.hadoop.hbase.master.cleaner.CleanerChore.deleteAction(CleanerChore.java:565)
org.apache.hadoop.hbase.master.cleaner.CleanerChore.traverseAndDelete(CleanerChore.java:483)
org.apache.hadoop.hbase.master.cleaner.CleanerChore.lambda$traverseAndDelete$2(CleanerChore.java:495)
org.apache.hadoop.hbase.master.cleaner.CleanerChore$$Lambda$794/1250580839.run(Unknown
Source)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750) {code}
> Run ITBLL for branch-3
> ----------------------
>
> Key: HBASE-27919
> URL: https://issues.apache.org/jira/browse/HBASE-27919
> Project: HBase
> Issue Type: Sub-task
> Components: integration tests
> Reporter: Duo Zhang
> Assignee: Haiping lv
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
