Arshad Mohammad created ZOOKEEPER-2380:
------------------------------------------
Summary: Deadlock while shutting down the zookeeper
Key: ZOOKEEPER-2380
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2380
Project: ZooKeeper
Issue Type: Bug
Components: server
Affects Versions: 3.5.0
Reporter: Arshad Mohammad
Assignee: Arshad Mohammad
Priority: Blocker
Zookeeper enters into deadlock while shutting down itself, thus making
zookeeper service unavailable as deadlocked server is a leader. Here is the
thread dump:
{code}
"QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled)" #25 prio=5
os_prio=0 tid=0x00007fbc502a6800 nid=0x834 in Object.wait()
[0x00007fbc4d9a8000] java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method) at
java.lang.Thread.join(Thread.java:1245) - locked <
0x00000000feb78000> (a org.apache.zookeeper.server.SyncRequestProcessor)
at java.lang.Thread.join(Thread.java:1319) at
org.apache.zookeeper.server.SyncRequestProcessor.shutdown(SyncRequestProcessor.java:196)
at
org.apache.zookeeper.server.quorum.ProposalRequestProcessor.shutdown(ProposalRequestProcessor.java:90)
at
org.apache.zookeeper.server.PrepRequestProcessor.shutdown(PrepRequestProcessor.java:1016)
at
org.apache.zookeeper.server.quorum.LeaderRequestProcessor.shutdown(LeaderRequestProcessor.java:78)
at
org.apache.zookeeper.server.ZooKeeperServer.shutdown(ZooKeeperServer.java:561)
- locked <
0x00000000feb61e20> (a
org.apache.zookeeper.server.quorum.LeaderZooKeeperServer) at
org.apache.zookeeper.server.quorum.QuorumZooKeeperServer.shutdown(QuorumZooKeeperServer.java:169)
- locked <
0x00000000feb61e20> (a
org.apache.zookeeper.server.quorum.LeaderZooKeeperServer) at
org.apache.zookeeper.server.quorum.LeaderZooKeeperServer.shutdown(LeaderZooKeeperServer.java:102)
- locked <
0x00000000feb61e20> (a
org.apache.zookeeper.server.quorum.LeaderZooKeeperServer) at
org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:637) at
org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:590) - locked <
0x00000000feb781a0> (a org.apache.zookeeper.server.quorum.Leader) at
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1108)
"SyncThread:1" #46 prio=5 os_prio=0 tid=0x00007fbc5848f000 nid=0x867 waiting
for monitor entry [0x00007fbc4ca90000] java.lang.Thread.State: BLOCKED (on
object monitor) at
org.apache.zookeeper.server.quorum.Leader.processAck(Leader.java:784) -
waiting to lock <0x00000000feb781a0> (a
org.apache.zookeeper.server.quorum.Leader) at
org.apache.zookeeper.server.quorum.AckRequestProcessor.processRequest(AckRequestProcessor.java:46)
at
org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:183)
at
org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:113)
{code}
Leader.lead() calls shutdown() from the synchronized block, it acquired lock on
Leader.java instance
{code}
while (true) {
synchronized (this) {
long start = Time.currentElapsedTime();
.....
{code}
In the shutdown flow SyncThread is trying to acquire lock on the same
Leader.java instance.
Leader thread acquired lock and waiting for SyncThread shutdown. SyncThread
waiting for the lock to complete its shutdown. This is how ZooKeeper entered
into deadlock
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
