[
https://issues.apache.org/jira/browse/ZOOKEEPER-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15343049#comment-15343049
]
Hadoop QA commented on ZOOKEEPER-2380:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12812315/ZOOKEEPER-2380-07.patch
against trunk revision 1748630.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 8 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3236//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3236//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3236//console
This message is automatically generated.
> Deadlock between leader shutdown and forwarding ACK to the leader
> -----------------------------------------------------------------
>
> Key: ZOOKEEPER-2380
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2380
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Reporter: Arshad Mohammad
> Assignee: Arshad Mohammad
> Priority: Blocker
> Fix For: 3.5.2, 3.6.0
>
> Attachments: ZOOKEEPER-2380-01.patch, ZOOKEEPER-2380-02.patch,
> ZOOKEEPER-2380-03.patch, ZOOKEEPER-2380-04.patch, ZOOKEEPER-2380-05.patch,
> ZOOKEEPER-2380-06.patch, ZOOKEEPER-2380-07.patch, ZOOKEEPER-2380-fail.out
>
>
> Zookeeper enters into deadlock while shutting down itself, thus making
> zookeeper service unavailable as deadlocked server is a leader. Here is the
> thread dump:
> {code}
> "QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled)" #25 prio=5
> os_prio=0 tid=0x00007fbc502a6800 nid=0x834 in Object.wait()
> [0x00007fbc4d9a8000] java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method) at
> java.lang.Thread.join(Thread.java:1245) - locked <
> 0x00000000feb78000> (a org.apache.zookeeper.server.SyncRequestProcessor)
> at java.lang.Thread.join(Thread.java:1319) at
> org.apache.zookeeper.server.SyncRequestProcessor.shutdown(SyncRequestProcessor.java:196)
> at
> org.apache.zookeeper.server.quorum.ProposalRequestProcessor.shutdown(ProposalRequestProcessor.java:90)
> at
> org.apache.zookeeper.server.PrepRequestProcessor.shutdown(PrepRequestProcessor.java:1016)
> at
> org.apache.zookeeper.server.quorum.LeaderRequestProcessor.shutdown(LeaderRequestProcessor.java:78)
> at
> org.apache.zookeeper.server.ZooKeeperServer.shutdown(ZooKeeperServer.java:561)
> - locked <
> 0x00000000feb61e20> (a
> org.apache.zookeeper.server.quorum.LeaderZooKeeperServer) at
> org.apache.zookeeper.server.quorum.QuorumZooKeeperServer.shutdown(QuorumZooKeeperServer.java:169)
> - locked <
> 0x00000000feb61e20> (a
> org.apache.zookeeper.server.quorum.LeaderZooKeeperServer) at
> org.apache.zookeeper.server.quorum.LeaderZooKeeperServer.shutdown(LeaderZooKeeperServer.java:102)
> - locked <
> 0x00000000feb61e20> (a
> org.apache.zookeeper.server.quorum.LeaderZooKeeperServer) at
> org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:637) at
> org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:590) - locked
> <
> 0x00000000feb781a0> (a org.apache.zookeeper.server.quorum.Leader) at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1108)
> "SyncThread:1" #46 prio=5 os_prio=0 tid=0x00007fbc5848f000 nid=0x867 waiting
> for monitor entry [0x00007fbc4ca90000] java.lang.Thread.State: BLOCKED
> (on object monitor) at
> org.apache.zookeeper.server.quorum.Leader.processAck(Leader.java:784) -
> waiting to lock <0x00000000feb781a0> (a
> org.apache.zookeeper.server.quorum.Leader) at
> org.apache.zookeeper.server.quorum.AckRequestProcessor.processRequest(AckRequestProcessor.java:46)
> at
> org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:183)
> at
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:113)
> {code}
> Leader.lead() calls shutdown() from the synchronized block, it acquired lock
> on Leader.java instance
> {code}
> while (true) {
> synchronized (this) {
> long start = Time.currentElapsedTime();
> .....
> {code}
> In the shutdown flow SyncThread is trying to acquire lock on the same
> Leader.java instance.
> Leader thread acquired lock and waiting for SyncThread shutdown. SyncThread
> waiting for the lock to complete its shutdown. This is how ZooKeeper entered
> into deadlock
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
