Michael Han created ZOOKEEPER-2486:
--------------------------------------

             Summary: Flaky Test: 
org.apache.zookeeper.test.QuorumZxidSyncTest.testBehindLeader
                 Key: ZOOKEEPER-2486
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2486
             Project: ZooKeeper
          Issue Type: Bug
          Components: tests
    Affects Versions: 3.4.8
            Reporter: Michael Han
             Fix For: 3.5.2


>From https://builds.apache.org/job/ZooKeeper_branch34_jdk7/1156

{noformat}
Error Message

waiting for server up
Stacktrace

junit.framework.AssertionFailedError: waiting for server up
        at 
org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:183)
        at 
org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:113)
        at 
org.apache.zookeeper.test.QuorumZxidSyncTest.testBehindLeader(QuorumZxidSyncTest.java:67)
        at 
org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
Standard Output

2016-07-21 08:11:45,722 [myid:] - INFO  [main:PortAssignment@32] - assigning 
port 11221
2016-07-21 08:11:45,729 [myid:] - INFO  [main:ZKTestCase$1@50] - STARTING 
testBehindLeader
2016-07-21 08:11:45,729 [myid:] - INFO  [main:QuorumBase@69] - QuorumBase.setup 
null
2016-07-21 08:11:45,926 [myid:] - INFO  [main:PortAssignment@32] - assigning 
port 11222
2016-07-21 08:11:45,926 [myid:] - INFO  [main:PortAssignment@32] - assigning 
port 11223
2016-07-21 08:11:45,927 [myid:] - INFO  [main:PortAssignment@32] - assigning 
port 11224
2016-07-21 08:11:45,927 [myid:] - INFO  [main:PortAssignment@32] - assigning 
port 11225
2016-07-21 08:11:45,927 [myid:] - INFO  [main:PortAssignment@32] - assigning 
port 11226
2016-07-21 08:11:45,928 [myid:] - INFO  [main:PortAssignment@32] - assigning 
port 11227
2016-07-21 08:11:45,928 [myid:] - INFO  [main:PortAssignment@32] - assigning 
port 11228
2016-07-21 08:11:45,928 [myid:] - INFO  [main:PortAssignment@32] - assigning 
port 11229
2016-07-21 08:11:45,928 [myid:] - INFO  [main:PortAssignment@32] - assigning 
port 11230
2016-07-21 08:11:45,929 [myid:] - INFO  [main:PortAssignment@32] - assigning 
port 11231
2016-07-21 08:11:45,929 [myid:] - INFO  [main:QuorumBase@93] - Ports are: 
127.0.0.1:11222,127.0.0.1:11223,127.0.0.1:11224,127.0.0.1:11225,127.0.0.1:11226
2016-07-21 08:11:45,946 [myid:] - INFO  [main:QuorumPeer$QuorumServer@149] - 
Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 08:11:45,946 [myid:] - INFO  [main:QuorumPeer$QuorumServer@149] - 
Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 08:11:45,948 [myid:] - INFO  [main:QuorumPeer$QuorumServer@149] - 
Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 08:11:45,948 [myid:] - INFO  [main:QuorumPeer$QuorumServer@149] - 
Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 08:11:45,949 [myid:] - INFO  [main:QuorumPeer$QuorumServer@149] - 
Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 08:11:45,949 [myid:] - INFO  [main:QuorumBase@142] - creating 
QuorumPeer 1 port 11222
2016-07-21 08:11:45,961 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - 
binding to port 0.0.0.0/0.0.0.0:11222
2016-07-21 08:11:45,979 [myid:] - INFO  [main:QuorumBase@145] - creating 
QuorumPeer 2 port 11223
2016-07-21 08:11:45,979 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - 
binding to port 0.0.0.0/0.0.0.0:11223
2016-07-21 08:11:45,980 [myid:] - INFO  [main:QuorumBase@148] - creating 
QuorumPeer 3 port 11224
2016-07-21 08:11:45,980 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - 
binding to port 0.0.0.0/0.0.0.0:11224
2016-07-21 08:11:45,981 [myid:] - INFO  [main:QuorumBase@151] - creating 
QuorumPeer 4 port 11225
2016-07-21 08:11:45,981 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - 
binding to port 0.0.0.0/0.0.0.0:11225
2016-07-21 08:11:45,981 [myid:] - INFO  [main:QuorumBase@154] - creating 
QuorumPeer 5 port 11226
2016-07-21 08:11:45,982 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - 
binding to port 0.0.0.0/0.0.0.0:11226
2016-07-21 08:11:45,982 [myid:] - INFO  [main:QuorumBase@163] - QuorumPeer 1 
voting view: 
{1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 
2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 
3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 
4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 
5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
2016-07-21 08:11:45,983 [myid:] - INFO  [main:QuorumBase@164] - QuorumPeer 2 
voting view: 
{1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 
2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 
3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 
4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 
5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
2016-07-21 08:11:45,983 [myid:] - INFO  [main:QuorumBase@165] - QuorumPeer 3 
voting view: 
{1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 
2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 
3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 
4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 
5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
2016-07-21 08:11:45,983 [myid:] - INFO  [main:QuorumBase@166] - QuorumPeer 4 
voting view: 
{1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 
2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 
3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 
4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 
5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
2016-07-21 08:11:45,983 [myid:] - INFO  [main:QuorumBase@167] - QuorumPeer 5 
voting view: 
{1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 
2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 
3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 
4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 
5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
2016-07-21 08:11:45,983 [myid:] - INFO  [main:QuorumBase@169] - start 
QuorumPeer 1
2016-07-21 08:11:45,988 [myid:] - INFO  [main:QuorumPeer@533] - currentEpoch 
not found! Creating with a reasonable default of 0. This should only happen 
when you are upgrading your installation
2016-07-21 08:11:45,990 [myid:] - INFO  [main:QuorumPeer@548] - acceptedEpoch 
not found! Creating with a reasonable default of 0. This should only happen 
when you are upgrading your installation
2016-07-21 08:11:45,996 [myid:] - INFO  
[ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: 
/127.0.0.1:12227
2016-07-21 08:11:46,001 [myid:] - INFO  [main:QuorumBase@171] - start 
QuorumPeer 2
2016-07-21 08:11:46,002 [myid:] - INFO  [main:QuorumPeer@533] - currentEpoch 
not found! Creating with a reasonable default of 0. This should only happen 
when you are upgrading your installation
2016-07-21 08:11:46,003 [myid:] - INFO  [main:QuorumPeer@548] - acceptedEpoch 
not found! Creating with a reasonable default of 0. This should only happen 
when you are upgrading your installation
2016-07-21 08:11:46,004 [myid:] - INFO  
[ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: 
/127.0.0.1:12228
2016-07-21 08:11:46,006 [myid:] - INFO  [main:QuorumBase@173] - start 
QuorumPeer 3
2016-07-21 08:11:46,006 [myid:] - INFO  [main:QuorumPeer@533] - currentEpoch 
not found! Creating with a reasonable default of 0. This should only happen 
when you are upgrading your installation
2016-07-21 08:11:46,007 [myid:] - INFO  [main:QuorumPeer@548] - acceptedEpoch 
not found! Creating with a reasonable default of 0. This should only happen 
when you are upgrading your installation
2016-07-21 08:11:46,007 [myid:] - INFO  
[QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11222:QuorumPeer@774] - LOOKING
2016-07-21 08:11:46,008 [myid:] - INFO  
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11223:QuorumPeer@774] - LOOKING
2016-07-21 08:11:46,009 [myid:] - INFO  
[ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: 
/127.0.0.1:12229
2016-07-21 08:11:46,009 [myid:] - INFO  
[QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11222:FastLeaderElection@818] - New 
election. My id =  1, proposed zxid=0x0
2016-07-21 08:11:46,009 [myid:] - INFO  
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11223:FastLeaderElection@818] - New 
election. My id =  2, proposed zxid=0x0
2016-07-21 08:11:46,009 [myid:] - INFO  [main:QuorumBase@175] - start 
QuorumPeer 4
2016-07-21 08:11:46,010 [myid:] - INFO  [main:QuorumPeer@533] - currentEpoch 
not found! Creating with a reasonable default of 0. This should only happen 
when you are upgrading your installation
2016-07-21 08:11:46,011 [myid:] - INFO  [main:QuorumPeer@548] - acceptedEpoch 
not found! Creating with a reasonable default of 0. This should only happen 
when you are upgrading your installation
2016-07-21 08:11:46,011 [myid:] - INFO  
[QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11224:QuorumPeer@774] - LOOKING
2016-07-21 08:11:46,011 [myid:] - INFO  
[/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:38924
2016-07-21 08:11:46,011 [myid:] - INFO  
[QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11224:FastLeaderElection@818] - New 
election. My id =  3, proposed zxid=0x0
2016-07-21 08:11:46,011 [myid:] - INFO  
[/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:43415
2016-07-21 08:11:46,011 [myid:] - INFO  
[WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, 
so dropping the connection: (2, 1)
2016-07-21 08:11:46,011 [myid:] - INFO  
[WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message 
format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,013 [myid:] - INFO  
[WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, 
so dropping the connection: (3, 1)
2016-07-21 08:11:46,014 [myid:] - INFO  
[/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:51548
2016-07-21 08:11:46,014 [myid:] - INFO  [main:QuorumBase@177] - start 
QuorumPeer 5
2016-07-21 08:11:46,015 [myid:] - INFO  
[ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: 
/127.0.0.1:12230
2016-07-21 08:11:46,015 [myid:] - INFO  [main:QuorumPeer@533] - currentEpoch 
not found! Creating with a reasonable default of 0. This should only happen 
when you are upgrading your installation
2016-07-21 08:11:46,015 [myid:] - WARN  
[WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 4 at 
election address /127.0.0.1:12230
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
        at 
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
        at 
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
        at java.lang.Thread.run(Thread.java:745)
2016-07-21 08:11:46,018 [myid:] - INFO  
[WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message 
format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,018 [myid:] - INFO  
[/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:51551
2016-07-21 08:11:46,016 [myid:] - INFO  
[/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:43417
2016-07-21 08:11:46,016 [myid:] - INFO  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:QuorumPeer@774] - LOOKING
2016-07-21 08:11:46,016 [myid:] - INFO  [main:QuorumPeer@548] - acceptedEpoch 
not found! Creating with a reasonable default of 0. This should only happen 
when you are upgrading your installation
2016-07-21 08:11:46,019 [myid:] - INFO  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:FastLeaderElection@818] - New 
election. My id =  4, proposed zxid=0x0
2016-07-21 08:11:46,019 [myid:] - INFO  
[/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:46102
2016-07-21 08:11:46,018 [myid:] - INFO  
[WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message 
format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,018 [myid:] - INFO  
[WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, 
so dropping the connection: (3, 2)
2016-07-21 08:11:46,018 [myid:] - INFO  
[WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname: 
127.0.0.1 to address: /127.0.0.1
2016-07-21 08:11:46,018 [myid:] - INFO  
[/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:38928
2016-07-21 08:11:46,021 [myid:] - INFO  
[WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, 
so dropping the connection: (4, 2)
2016-07-21 08:11:46,021 [myid:] - INFO  
[WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message 
format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,021 [myid:] - INFO  
[/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:51553
2016-07-21 08:11:46,020 [myid:] - INFO  
[WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message 
format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,019 [myid:] - INFO  
[WorkerSender[myid=3]:QuorumCnxManager@199] - Have smaller server identifier, 
so dropping the connection: (4, 3)
2016-07-21 08:11:46,022 [myid:] - INFO  
[WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message 
format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,021 [myid:] - WARN  
[WorkerSender[myid=2]:QuorumCnxManager@400] - Cannot open channel to 5 at 
election address /127.0.0.1:12231
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
        at 
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
        at 
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
        at java.lang.Thread.run(Thread.java:745)
2016-07-21 08:11:46,021 [myid:] - INFO  
[WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message 
format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,021 [myid:] - WARN  
[WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 5 at 
election address /127.0.0.1:12231
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
        at 
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
        at 
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
        at java.lang.Thread.run(Thread.java:745)
2016-07-21 08:11:46,023 [myid:] - INFO  
[WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message 
format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,023 [myid:] - INFO  
[WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message 
format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,023 [myid:] - INFO  
[WorkerSender[myid=2]:QuorumPeer$QuorumServer@149] - Resolved hostname: 
127.0.0.1 to address: /127.0.0.1
2016-07-21 08:11:46,024 [myid:] - INFO  
[/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:46104
2016-07-21 08:11:46,022 [myid:] - INFO  
[ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: 
/127.0.0.1:12231
2016-07-21 08:11:46,022 [myid:] - WARN  
[WorkerSender[myid=3]:QuorumCnxManager@400] - Cannot open channel to 5 at 
election address /127.0.0.1:12231
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
        at 
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
        at 
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
        at java.lang.Thread.run(Thread.java:745)
2016-07-21 08:11:46,025 [myid:] - INFO  
[WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message 
format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,025 [myid:] - INFO  
[WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message 
format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,025 [myid:] - INFO  [main:QuorumBase@179] - started 
QuorumPeer 5
2016-07-21 08:11:46,026 [myid:] - INFO  [main:QuorumBase@181] - Checking ports 
127.0.0.1:11222,127.0.0.1:11223,127.0.0.1:11224,127.0.0.1:11225,127.0.0.1:11226
2016-07-21 08:11:46,025 [myid:] - INFO  
[WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message 
format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,025 [myid:] - INFO  
[/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:43428
2016-07-21 08:11:46,027 [myid:] - INFO  
[/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:46109
2016-07-21 08:11:46,025 [myid:] - INFO  
[WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message 
format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,024 [myid:] - INFO  
[WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname: 
127.0.0.1 to address: /127.0.0.1
2016-07-21 08:11:46,027 [myid:] - INFO  
[WorkerSender[myid=4]:QuorumCnxManager@199] - Have smaller server identifier, 
so dropping the connection: (5, 4)
2016-07-21 08:11:46,027 [myid:] - INFO  
[WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,026 [myid:] - INFO  
[/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:38938
2016-07-21 08:11:46,025 [myid:] - INFO  
[WorkerSender[myid=3]:QuorumPeer$QuorumServer@149] - Resolved hostname: 
127.0.0.1 to address: /127.0.0.1
2016-07-21 08:11:46,025 [myid:] - INFO  
[WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, 
so dropping the connection: (4, 2)
2016-07-21 08:11:46,028 [myid:] - INFO  [main:FourLetterWordMain@62] - 
connecting to 127.0.0.1 11222
2016-07-21 08:11:46,028 [myid:] - WARN  
[RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my 
id = 4, error = 
java.net.SocketException: Socket closed
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.net.SocketInputStream.read(SocketInputStream.java:210)
        at java.io.DataInputStream.readInt(DataInputStream.java:387)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:11:46,029 [myid:] - WARN  
[RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:11:46,028 [myid:] - INFO  
[WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,027 [myid:] - INFO  
[QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:QuorumPeer@774] - LOOKING
2016-07-21 08:11:46,027 [myid:] - INFO  
[WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message 
format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,027 [myid:] - INFO  
[/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:52086
2016-07-21 08:11:46,030 [myid:] - WARN  
[RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4, my 
id = 2, error = 
java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:11:46,030 [myid:] - WARN  
[SendWorker:4:QuorumCnxManager$SendWorker@732] - Exception when using channel: 
for id 4 my id = 2 error = java.net.SocketException: Broken pipe
2016-07-21 08:11:46,029 [myid:] - INFO  
[WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message 
format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,031 [myid:] - INFO  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory@192] - 
Accepted socket connection from /127.0.0.1:52363
2016-07-21 08:11:46,029 [myid:] - INFO  
[/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:38940
2016-07-21 08:11:46,029 [myid:] - INFO  
[WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, 
so dropping the connection: (5, 1)
2016-07-21 08:11:46,029 [myid:] - INFO  
[WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message 
format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,032 [myid:] - INFO  
[WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message 
format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,032 [myid:] - INFO  
[WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,029 [myid:] - WARN  
[SendWorker:2:QuorumCnxManager$SendWorker@732] - Exception when using channel: 
for id 2 my id = 4 error = java.net.SocketException: Socket closed
2016-07-21 08:11:46,033 [myid:] - WARN  
[SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:11:46,029 [myid:] - INFO  
[WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, 
so dropping the connection: (5, 2)
2016-07-21 08:11:46,028 [myid:] - INFO  
[WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message 
format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,033 [myid:] - INFO  
[WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, 
so dropping the connection: (5, 1)
2016-07-21 08:11:46,032 [myid:] - INFO  
[WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message 
format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,031 [myid:] - INFO  
[/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:52088
2016-07-21 08:11:46,031 [myid:] - INFO  
[WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,031 [myid:] - INFO  
[/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:46117
2016-07-21 08:11:46,030 [myid:] - WARN  
[SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:11:46,030 [myid:] - WARN  
[RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:11:46,030 [myid:] - INFO  
[WorkerSender[myid=3]:QuorumCnxManager@199] - Have smaller server identifier, 
so dropping the connection: (5, 3)
2016-07-21 08:11:46,030 [myid:] - INFO  
[QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:FastLeaderElection@818] - New 
election. My id =  5, proposed zxid=0x0
2016-07-21 08:11:46,030 [myid:] - INFO  
[WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,034 [myid:] - INFO  
[/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:38947
2016-07-21 08:11:46,034 [myid:] - INFO  
[WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message 
format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,034 [myid:] - INFO  
[WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message 
format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,033 [myid:] - INFO  
[WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,035 [myid:] - INFO  
[/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:52089
2016-07-21 08:11:46,035 [myid:] - INFO  
[WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,036 [myid:] - INFO  
[WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,036 [myid:] - INFO  
[WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message 
format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,037 [myid:] - INFO  
[/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:43442
2016-07-21 08:11:46,036 [myid:] - INFO  
[WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,036 [myid:] - INFO  
[WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,037 [myid:] - INFO  
[WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,037 [myid:] - INFO  
[WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,038 [myid:] - INFO  
[/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:52091
2016-07-21 08:11:46,037 [myid:] - INFO  
[WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message 
format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,036 [myid:] - INFO  
[WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, 
so dropping the connection: (5, 1)
2016-07-21 08:11:46,038 [myid:] - INFO  
[WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,038 [myid:] - INFO  
[WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,038 [myid:] - INFO  
[WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,038 [myid:] - INFO  
[/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:51573
2016-07-21 08:11:46,038 [myid:] - INFO  
[WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,037 [myid:] - INFO  
[WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,039 [myid:] - INFO  
[WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,039 [myid:] - INFO  
[WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,039 [myid:] - INFO  
[WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,040 [myid:] - INFO  
[WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,039 [myid:] - INFO  
[WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,039 [myid:] - INFO  
[WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,039 [myid:] - INFO  
[/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:52093
2016-07-21 08:11:46,040 [myid:] - INFO  
[WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,040 [myid:] - INFO  
[WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,040 [myid:] - INFO  
[WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,040 [myid:] - INFO  
[WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,041 [myid:] - INFO  
[WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,041 [myid:] - INFO  
[/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:43444
2016-07-21 08:11:46,041 [myid:] - INFO  
[WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,041 [myid:] - WARN  
[RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my 
id = 5, error = 
java.net.SocketException: Socket closed
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.io.DataInputStream.readFully(DataInputStream.java:195)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:805)
2016-07-21 08:11:46,042 [myid:] - WARN  
[RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:11:46,041 [myid:] - INFO  
[WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,041 [myid:] - WARN  
[SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:11:46,041 [myid:] - INFO  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11222:NIOServerCnxn@827] - Processing 
stat command from /127.0.0.1:52363
2016-07-21 08:11:46,041 [myid:] - WARN  
[RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my 
id = 1, error = 
java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:11:46,043 [myid:] - WARN  
[RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:11:46,041 [myid:] - INFO  
[WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,041 [myid:] - INFO  
[WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,043 [myid:] - INFO  
[WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,043 [myid:] - WARN  
[SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:11:46,042 [myid:] - INFO  
[WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,042 [myid:] - WARN  
[SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:11:46,044 [myid:] - WARN  
[SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:11:46,042 [myid:] - INFO  
[WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,042 [myid:] - INFO  
[/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:52096
2016-07-21 08:11:46,041 [myid:] - INFO  
[WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message 
format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,045 [myid:] - WARN  
[SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:11:46,045 [myid:] - WARN  
[SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:11:46,045 [myid:] - INFO  [Thread-2:NIOServerCnxn@1008] - Closed 
socket connection for client /127.0.0.1:52363 (no session established for 
client)
2016-07-21 08:11:46,044 [myid:] - INFO  
[WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,044 [myid:] - INFO  
[WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,044 [myid:] - INFO  
[WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,046 [myid:] - INFO  
[WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,045 [myid:] - INFO  
[/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request 
/127.0.0.1:43446
2016-07-21 08:11:46,045 [myid:] - INFO  
[WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,045 [myid:] - WARN  
[RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my 
id = 1, error = 
java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:11:46,047 [myid:] - WARN  
[RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:11:46,045 [myid:] - WARN  
[RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my 
id = 5, error = 
java.net.SocketException: Socket closed
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.net.SocketInputStream.read(SocketInputStream.java:210)
        at java.io.DataInputStream.readInt(DataInputStream.java:387)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:11:46,047 [myid:] - WARN  
[RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:11:46,047 [myid:] - INFO  
[WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,047 [myid:] - WARN  
[SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:11:46,048 [myid:] - WARN  
[SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:11:46,046 [myid:] - INFO  
[WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,048 [myid:] - INFO  
[WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,048 [myid:] - INFO  
[WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,049 [myid:] - INFO  
[WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,049 [myid:] - INFO  
[WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,049 [myid:] - INFO  
[WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message 
format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 
1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,245 [myid:] - INFO  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:QuorumPeer@844] - FOLLOWING
2016-07-21 08:11:46,246 [myid:] - INFO  
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11223:QuorumPeer@844] - FOLLOWING
2016-07-21 08:11:46,247 [myid:] - INFO  
[QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11224:QuorumPeer@844] - FOLLOWING
2016-07-21 08:11:46,249 [myid:] - INFO  
[QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11222:QuorumPeer@844] - FOLLOWING
2016-07-21 08:11:46,249 [myid:] - INFO  
[QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:QuorumPeer@856] - LEADING
2016-07-21 08:11:46,251 [myid:] - INFO  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Learner@86] - TCP NoDelay set to: true
2016-07-21 08:11:46,255 [myid:] - INFO  
[QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:Leader@59] - TCP NoDelay set to: true
2016-07-21 08:11:46,259 [myid:] - INFO  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server 
environment:zookeeper.version=3.4.9-SNAPSHOT-1753645, built on 07/21/2016 07:46 
GMT
2016-07-21 08:11:46,259 [myid:] - INFO  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server 
environment:host.name=asf907.gq1.ygridcore.net
2016-07-21 08:11:46,260 [myid:] - INFO  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server 
environment:java.version=1.7.0_80
2016-07-21 08:11:46,260 [myid:] - INFO  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server 
environment:java.vendor=Oracle Corporation
2016-07-21 08:11:46,260 [myid:] - INFO  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server 
environment:java.home=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/jre
2016-07-21 08:11:46,260 [myid:] - INFO  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server 
environment:java.class.path=/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/antlr-2.7.6.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/checkstyle-5.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-beanutils-core-1.7.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-cli-1.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-lang-1.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-logging-1.0.3.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/google-collections-0.9.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/junit-4.8.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/jline-0.9.94.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/log4j-1.2.16.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/netty-3.10.5.Final.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/slf4j-api-1.6.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/slf4j-log4j12-1.6.1.jar:/home/jenkins/tools/ant/apache-ant-1.9.4/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar
2016-07-21 08:11:46,260 [myid:] - INFO  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server 
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-07-21 08:11:46,260 [myid:] - INFO  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server 
environment:java.io.tmpdir=/tmp
2016-07-21 08:11:46,260 [myid:] - INFO  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server 
environment:java.compiler=<NA>
2016-07-21 08:11:46,260 [myid:] - INFO  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server 
environment:os.name=Linux
2016-07-21 08:11:46,260 [myid:] - INFO  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server 
environment:os.arch=amd64
2016-07-21 08:11:46,260 [myid:] - INFO  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server 
environment:os.version=3.13.0-36-lowlatency
2016-07-21 08:11:46,260 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:
...[truncated 1420555 chars]...
ager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:21,678 [myid:] - WARN  
[SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:21,672 [myid:] - WARN  
[RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my 
id = 5, error = 
java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:21,679 [myid:] - WARN  
[RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:21,672 [myid:] - WARN  
[SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:21,679 [myid:] - WARN  
[SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:21,679 [myid:] - WARN  
[SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:21,679 [myid:] - WARN  
[SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:21,678 [myid:] - WARN  
[SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:21,679 [myid:] - WARN  
[SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:21,677 [myid:] - WARN  
[SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:21,680 [myid:] - WARN  
[SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:21,675 [myid:] - WARN  
[SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:21,680 [myid:] - WARN  
[SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:22,000 [myid:] - INFO  [SessionTracker:SessionTrackerImpl@162] 
- SessionTrackerImpl exited loop!
2016-07-21 08:14:22,317 [myid:] - INFO  
[/127.0.0.1:12241:QuorumCnxManager$Listener@560] - Leaving listener
2016-07-21 08:14:22,320 [myid:] - INFO  
[/127.0.0.1:12242:QuorumCnxManager$Listener@560] - Leaving listener
2016-07-21 08:14:22,579 [myid:] - INFO  
[QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
        at 
org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
        at 
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
2016-07-21 08:14:22,579 [myid:] - INFO  
[QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:FollowerZooKeeperServer@140] - 
Shutting down
2016-07-21 08:14:22,579 [myid:] - INFO  
[QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:SyncRequestProcessor@209] - Shutting 
down
2016-07-21 08:14:22,579 [myid:] - WARN  
[QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:QuorumPeer@874] - QuorumPeer main 
thread exited
2016-07-21 08:14:22,582 [myid:] - INFO  [main:QuorumBase@306] - Shutting down 
quorum peer QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234
2016-07-21 08:14:22,582 [myid:] - INFO  [main:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
        at 
org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
        at 
org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891)
        at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307)
        at 
org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:298)
        at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285)
        at 
org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
        at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
        at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
        at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
        at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
        at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
        at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
        at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
        at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
        at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
        at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
2016-07-21 08:14:22,582 [myid:] - INFO  [main:FollowerZooKeeperServer@140] - 
Shutting down
2016-07-21 08:14:22,582 [myid:] - INFO  [main:ZooKeeperServer@469] - shutting 
down
2016-07-21 08:14:22,582 [myid:] - INFO  [main:FollowerRequestProcessor@107] - 
Shutting down
2016-07-21 08:14:22,582 [myid:] - INFO  [main:CommitProcessor@184] - Shutting 
down
2016-07-21 08:14:22,582 [myid:] - INFO  
[FollowerRequestProcessor:2:FollowerRequestProcessor@97] - 
FollowerRequestProcessor exited loop!
2016-07-21 08:14:22,582 [myid:] - INFO  [CommitProcessor:2:CommitProcessor@153] 
- CommitProcessor exited loop!
2016-07-21 08:14:22,582 [myid:] - INFO  [main:FinalRequestProcessor@402] - 
shutdown of request processor complete
2016-07-21 08:14:22,583 [myid:] - INFO  [main:SyncRequestProcessor@209] - 
Shutting down
2016-07-21 08:14:22,583 [myid:] - INFO  [SyncThread:2:SyncRequestProcessor@187] 
- SyncRequestProcessor exited!
2016-07-21 08:14:22,584 [myid:] - INFO  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11234:NIOServerCnxnFactory@219] - 
NIOServerCnxn factory exited run method
2016-07-21 08:14:22,585 [myid:] - WARN  
[SendWorker:3:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:22,585 [myid:] - WARN  
[RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my 
id = 4, error = 
java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:22,585 [myid:] - WARN  
[RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:22,585 [myid:] - WARN  
[RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id 3, my 
id = 2, error = 
java.net.SocketException: Socket closed
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.net.SocketInputStream.read(SocketInputStream.java:210)
        at java.io.DataInputStream.readInt(DataInputStream.java:387)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:22,586 [myid:] - WARN  
[RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:22,585 [myid:] - WARN  
[RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my 
id = 5, error = 
java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:22,586 [myid:] - WARN  
[RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:22,585 [myid:] - WARN  
[RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my 
id = 2, error = 
java.net.SocketException: Socket closed
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.net.SocketInputStream.read(SocketInputStream.java:210)
        at java.io.DataInputStream.readInt(DataInputStream.java:387)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:22,587 [myid:] - WARN  
[RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:22,585 [myid:] - WARN  
[SendWorker:3:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:22,585 [myid:] - WARN  
[RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my 
id = 3, error = 
java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:22,587 [myid:] - WARN  
[RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:22,585 [myid:] - WARN  
[SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:22,587 [myid:] - WARN  
[SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:22,585 [myid:] - ERROR 
[/127.0.0.1:12239:QuorumCnxManager$Listener@547] - Exception while listening
java.net.SocketException: Socket closed
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at 
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
        at java.net.ServerSocket.accept(ServerSocket.java:498)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
2016-07-21 08:14:22,587 [myid:] - WARN  
[SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:22,588 [myid:] - WARN  
[SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:22,587 [myid:] - WARN  
[SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:22,589 [myid:] - WARN  
[SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:22,586 [myid:] - WARN  
[SendWorker:4:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:22,589 [myid:] - WARN  
[SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:22,586 [myid:] - WARN  
[RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4, my 
id = 2, error = 
java.net.SocketException: Socket closed
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.net.SocketInputStream.read(SocketInputStream.java:210)
        at java.io.DataInputStream.readInt(DataInputStream.java:387)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:22,589 [myid:] - WARN  
[RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:22,586 [myid:] - INFO  [main:QuorumBase@310] - Shutting down 
leader election QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234
2016-07-21 08:14:22,589 [myid:] - INFO  [main:QuorumBase@315] - Waiting for 
QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234 to exit thread
2016-07-21 08:14:22,585 [myid:] - WARN  
[SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:22,590 [myid:] - WARN  
[SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:22,673 [myid:] - INFO  
[/127.0.0.1:12238:QuorumCnxManager$Listener@560] - Leaving listener
2016-07-21 08:14:23,079 [myid:] - INFO  
[WorkerSender[myid=3]:FastLeaderElection$Messenger$WorkerSender@438] - 
WorkerSender is down
2016-07-21 08:14:23,080 [myid:] - INFO  
[WorkerSender[myid=4]:FastLeaderElection$Messenger$WorkerSender@438] - 
WorkerSender is down
2016-07-21 08:14:23,084 [myid:] - INFO  
[WorkerSender[myid=5]:FastLeaderElection$Messenger$WorkerSender@438] - 
WorkerSender is down
2016-07-21 08:14:23,088 [myid:] - INFO  
[WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@407] - 
WorkerReceiver is down
2016-07-21 08:14:23,088 [myid:] - INFO  
[WorkerReceiver[myid=4]:FastLeaderElection$Messenger$WorkerReceiver@407] - 
WorkerReceiver is down
2016-07-21 08:14:23,089 [myid:] - INFO  
[WorkerReceiver[myid=5]:FastLeaderElection$Messenger$WorkerReceiver@407] - 
WorkerReceiver is down
2016-07-21 08:14:23,578 [myid:] - INFO  
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
        at 
org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
        at 
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
2016-07-21 08:14:23,578 [myid:] - INFO  
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:FollowerZooKeeperServer@140] - 
Shutting down
2016-07-21 08:14:23,578 [myid:] - INFO  
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:SyncRequestProcessor@209] - Shutting 
down
2016-07-21 08:14:23,578 [myid:] - WARN  
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:QuorumPeer@874] - QuorumPeer main 
thread exited
2016-07-21 08:14:23,579 [myid:] - INFO  [main:QuorumBase@306] - Shutting down 
quorum peer QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235
2016-07-21 08:14:23,579 [myid:] - INFO  [main:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
        at 
org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
        at 
org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891)
        at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307)
        at 
org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:299)
        at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285)
        at 
org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
        at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
        at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
        at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
        at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
        at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
        at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
        at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
        at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
        at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
        at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
2016-07-21 08:14:23,579 [myid:] - INFO  [main:FollowerZooKeeperServer@140] - 
Shutting down
2016-07-21 08:14:23,579 [myid:] - INFO  [main:ZooKeeperServer@469] - shutting 
down
2016-07-21 08:14:23,580 [myid:] - INFO  [main:FollowerRequestProcessor@107] - 
Shutting down
2016-07-21 08:14:23,580 [myid:] - INFO  [main:CommitProcessor@184] - Shutting 
down
2016-07-21 08:14:23,580 [myid:] - INFO  
[FollowerRequestProcessor:3:FollowerRequestProcessor@97] - 
FollowerRequestProcessor exited loop!
2016-07-21 08:14:23,580 [myid:] - INFO  [CommitProcessor:3:CommitProcessor@153] 
- CommitProcessor exited loop!
2016-07-21 08:14:23,580 [myid:] - INFO  [main:FinalRequestProcessor@402] - 
shutdown of request processor complete
2016-07-21 08:14:23,581 [myid:] - INFO  [main:SyncRequestProcessor@209] - 
Shutting down
2016-07-21 08:14:23,581 [myid:] - INFO  [SyncThread:3:SyncRequestProcessor@187] 
- SyncRequestProcessor exited!
2016-07-21 08:14:23,582 [myid:] - INFO  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11235:NIOServerCnxnFactory@219] - 
NIOServerCnxn factory exited run method
2016-07-21 08:14:23,583 [myid:] - ERROR 
[/127.0.0.1:12240:QuorumCnxManager$Listener@547] - Exception while listening
java.net.SocketException: Socket closed
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at 
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
        at java.net.ServerSocket.accept(ServerSocket.java:498)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
2016-07-21 08:14:23,583 [myid:] - WARN  
[SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:23,585 [myid:] - WARN  
[SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:23,584 [myid:] - WARN  
[RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4, my 
id = 3, error = 
java.net.SocketException: Socket closed
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.net.SocketInputStream.read(SocketInputStream.java:210)
        at java.io.DataInputStream.readInt(DataInputStream.java:387)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:23,585 [myid:] - WARN  
[RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:23,584 [myid:] - WARN  
[SendWorker:4:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:23,585 [myid:] - WARN  
[SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:23,584 [myid:] - WARN  
[RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my 
id = 3, error = 
java.net.SocketException: Socket closed
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.net.SocketInputStream.read(SocketInputStream.java:210)
        at java.io.DataInputStream.readInt(DataInputStream.java:387)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:23,585 [myid:] - WARN  
[RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:23,584 [myid:] - WARN  
[RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id 3, my 
id = 5, error = 
java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:23,586 [myid:] - WARN  
[RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:23,584 [myid:] - WARN  
[RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id 3, my 
id = 4, error = 
java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:23,586 [myid:] - WARN  
[RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:23,584 [myid:] - INFO  [main:QuorumBase@310] - Shutting down 
leader election QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235
2016-07-21 08:14:23,586 [myid:] - INFO  [main:QuorumBase@315] - Waiting for 
QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235 to exit thread
2016-07-21 08:14:23,586 [myid:] - WARN  
[SendWorker:3:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:23,586 [myid:] - WARN  
[SendWorker:3:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:23,586 [myid:] - WARN  
[SendWorker:3:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:23,587 [myid:] - WARN  
[SendWorker:3:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:23,588 [myid:] - INFO  
[/127.0.0.1:12239:QuorumCnxManager$Listener@560] - Leaving listener
2016-07-21 08:14:24,355 [myid:] - INFO  
[WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@438] - 
WorkerSender is down
2016-07-21 08:14:24,358 [myid:] - INFO  
[WorkerSender[myid=1]:FastLeaderElection$Messenger$WorkerSender@438] - 
WorkerSender is down
2016-07-21 08:14:24,360 [myid:] - INFO  
[WorkerSender[myid=3]:FastLeaderElection$Messenger$WorkerSender@438] - 
WorkerSender is down
2016-07-21 08:14:24,361 [myid:] - INFO  
[WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@407] - 
WorkerReceiver is down
2016-07-21 08:14:24,361 [myid:] - INFO  
[WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@407] - 
WorkerReceiver is down
2016-07-21 08:14:24,362 [myid:] - INFO  
[WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@407] - 
WorkerReceiver is down
2016-07-21 08:14:24,578 [myid:] - INFO  
[QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
        at 
org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
        at 
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
2016-07-21 08:14:24,578 [myid:] - INFO  
[QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:FollowerZooKeeperServer@140] - 
Shutting down
2016-07-21 08:14:24,578 [myid:] - INFO  
[QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:SyncRequestProcessor@209] - Shutting 
down
2016-07-21 08:14:24,578 [myid:] - WARN  
[QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:QuorumPeer@874] - QuorumPeer main 
thread exited
2016-07-21 08:14:24,579 [myid:] - INFO  [main:QuorumBase@306] - Shutting down 
quorum peer QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236
2016-07-21 08:14:24,579 [myid:] - INFO  [main:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
        at 
org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
        at 
org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891)
        at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307)
        at 
org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:300)
        at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285)
        at 
org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
        at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
        at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
        at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
        at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
        at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
        at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
        at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
        at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
        at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
        at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
2016-07-21 08:14:24,580 [myid:] - INFO  [main:FollowerZooKeeperServer@140] - 
Shutting down
2016-07-21 08:14:24,581 [myid:] - INFO  [main:ZooKeeperServer@469] - shutting 
down
2016-07-21 08:14:24,581 [myid:] - INFO  [main:FollowerRequestProcessor@107] - 
Shutting down
2016-07-21 08:14:24,581 [myid:] - INFO  [main:CommitProcessor@184] - Shutting 
down
2016-07-21 08:14:24,581 [myid:] - INFO  [main:FinalRequestProcessor@402] - 
shutdown of request processor complete
2016-07-21 08:14:24,581 [myid:] - INFO  
[FollowerRequestProcessor:4:FollowerRequestProcessor@97] - 
FollowerRequestProcessor exited loop!
2016-07-21 08:14:24,581 [myid:] - INFO  [main:SyncRequestProcessor@209] - 
Shutting down
2016-07-21 08:14:24,581 [myid:] - INFO  [CommitProcessor:4:CommitProcessor@153] 
- CommitProcessor exited loop!
2016-07-21 08:14:24,582 [myid:] - INFO  [SyncThread:4:SyncRequestProcessor@187] 
- SyncRequestProcessor exited!
2016-07-21 08:14:24,583 [myid:] - INFO  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11236:NIOServerCnxnFactory@219] - 
NIOServerCnxn factory exited run method
2016-07-21 08:14:24,585 [myid:] - INFO  [main:QuorumBase@310] - Shutting down 
leader election QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236
2016-07-21 08:14:24,586 [myid:] - ERROR 
[/127.0.0.1:12241:QuorumCnxManager$Listener@547] - Exception while listening
java.net.SocketException: Socket closed
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at 
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
        at java.net.ServerSocket.accept(ServerSocket.java:498)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
2016-07-21 08:14:24,586 [myid:] - WARN  
[SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:24,586 [myid:] - WARN  
[SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:24,586 [myid:] - INFO  [main:QuorumBase@315] - Waiting for 
QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236 to exit thread
2016-07-21 08:14:24,586 [myid:] - WARN  
[RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4, my 
id = 5, error = 
java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:24,587 [myid:] - WARN  
[RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:24,585 [myid:] - INFO  
[/127.0.0.1:12240:QuorumCnxManager$Listener@560] - Leaving listener
2016-07-21 08:14:24,585 [myid:] - WARN  
[RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my 
id = 4, error = 
java.net.SocketException: Socket closed
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.net.SocketInputStream.read(SocketInputStream.java:210)
        at java.io.DataInputStream.readInt(DataInputStream.java:387)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:24,588 [myid:] - WARN  
[RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:24,588 [myid:] - WARN  
[SendWorker:4:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for 
message on queue
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at 
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:24,588 [myid:] - WARN  
[SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:25,579 [myid:] - INFO  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
        at 
org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
        at 
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
2016-07-21 08:14:25,579 [myid:] - INFO  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:FollowerZooKeeperServer@140] - 
Shutting down
2016-07-21 08:14:25,579 [myid:] - INFO  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:SyncRequestProcessor@209] - Shutting 
down
2016-07-21 08:14:25,579 [myid:] - WARN  
[QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:QuorumPeer@874] - QuorumPeer main 
thread exited
2016-07-21 08:14:25,579 [myid:] - INFO  [main:QuorumBase@306] - Shutting down 
quorum peer QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237
2016-07-21 08:14:25,579 [myid:] - INFO  [main:Leader@496] - Shutting down
2016-07-21 08:14:25,580 [myid:] - INFO  [main:Leader@502] - Shutdown called
java.lang.Exception: shutdown Leader! reason: quorum Peer shutdown
        at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:502)
        at 
org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:888)
        at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307)
        at 
org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:301)
        at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285)
        at 
org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
        at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
        at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
        at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
        at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
        at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
        at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
        at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
        at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
        at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
        at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
2016-07-21 08:14:25,581 [myid:] - INFO  [main:ZooKeeperServer@469] - shutting 
down
2016-07-21 08:14:25,581 [myid:] - INFO  
[LearnerCnxAcceptor-/127.0.0.1:12237:Leader$LearnerCnxAcceptor@325] - exception 
while shutting down acceptor: java.net.SocketException: Socket closed
2016-07-21 08:14:25,582 [myid:] - INFO  [main:SessionTrackerImpl@225] - 
Shutting down
2016-07-21 08:14:25,582 [myid:] - INFO  [main:PrepRequestProcessor@765] - 
Shutting down
2016-07-21 08:14:25,582 [myid:] - INFO  [main:ProposalRequestProcessor@88] - 
Shutting down
2016-07-21 08:14:25,582 [myid:] - INFO  [main:CommitProcessor@184] - Shutting 
down
2016-07-21 08:14:25,582 [myid:] - INFO  [ProcessThread(sid:5 
cport:-1)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2016-07-21 08:14:25,583 [myid:] - INFO  [CommitProcessor:5:CommitProcessor@153] 
- CommitProcessor exited loop!
2016-07-21 08:14:25,582 [myid:] - INFO  
[main:Leader$ToBeAppliedRequestProcessor@661] - Shutting down
2016-07-21 08:14:25,583 [myid:] - INFO  [main:FinalRequestProcessor@402] - 
shutdown of request processor complete
2016-07-21 08:14:25,583 [myid:] - INFO  [main:SyncRequestProcessor@209] - 
Shutting down
2016-07-21 08:14:25,583 [myid:] - INFO  [SyncThread:5:SyncRequestProcessor@187] 
- SyncRequestProcessor exited!
2016-07-21 08:14:25,585 [myid:] - WARN  
[LearnerHandler-/127.0.0.1:43485:LearnerHandler@644] - ******* GOODBYE 
/127.0.0.1:43485 ********
2016-07-21 08:14:25,585 [myid:] - WARN  
[LearnerHandler-/127.0.0.1:43486:LearnerHandler@644] - ******* GOODBYE 
/127.0.0.1:43486 ********
2016-07-21 08:14:25,585 [myid:] - WARN  
[LearnerHandler-/127.0.0.1:43486:LearnerHandler@656] - Ignoring unexpected 
exception
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
        at 
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
        at 
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
        at 
org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
        at 
org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
2016-07-21 08:14:25,585 [myid:] - WARN  
[LearnerHandler-/127.0.0.1:43485:LearnerHandler@656] - Ignoring unexpected 
exception
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
        at 
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
        at 
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
        at 
org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
        at 
org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
2016-07-21 08:14:25,585 [myid:] - WARN  
[LearnerHandler-/127.0.0.1:43488:LearnerHandler@644] - ******* GOODBYE 
/127.0.0.1:43488 ********
2016-07-21 08:14:25,586 [myid:] - INFO  
[/127.0.0.1:12241:QuorumCnxManager$Listener@560] - Leaving listener
2016-07-21 08:14:25,586 [myid:] - WARN  
[LearnerHandler-/127.0.0.1:43487:LearnerHandler@644] - ******* GOODBYE 
/127.0.0.1:43487 ********
2016-07-21 08:14:25,586 [myid:] - WARN  
[LearnerHandler-/127.0.0.1:43487:LearnerHandler@656] - Ignoring unexpected 
exception
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
        at 
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
        at 
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
        at 
org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
        at 
org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
2016-07-21 08:14:25,586 [myid:] - INFO  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11237:NIOServerCnxnFactory@219] - 
NIOServerCnxn factory exited run method
2016-07-21 08:14:25,586 [myid:] - WARN  
[LearnerHandler-/127.0.0.1:43488:LearnerHandler@656] - Ignoring unexpected 
exception
java.lang.InterruptedException
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
        at 
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
        at 
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
        at 
org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
        at 
org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
2016-07-21 08:14:25,587 [myid:] - WARN  
[QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237:QuorumPeer@862] - Unexpected exception
java.lang.InterruptedException: sleep interrupted
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:456)
        at 
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:859)
2016-07-21 08:14:25,588 [myid:] - INFO  
[QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237:Leader@496] - Shutting down
2016-07-21 08:14:25,588 [myid:] - INFO  [main:QuorumBase@310] - Shutting down 
leader election QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237
2016-07-21 08:14:25,588 [myid:] - ERROR 
[/127.0.0.1:12242:QuorumCnxManager$Listener@547] - Exception while listening
java.net.SocketException: Socket closed
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at 
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
        at java.net.ServerSocket.accept(ServerSocket.java:498)
        at 
org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
2016-07-21 08:14:25,588 [myid:] - INFO  [main:QuorumBase@315] - Waiting for 
QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237 to exit thread
2016-07-21 08:14:25,588 [myid:] - WARN  
[QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237:QuorumPeer@874] - QuorumPeer main 
thread exited
2016-07-21 08:14:25,589 [myid:] - INFO  [main:FourLetterWordMain@62] - 
connecting to 127.0.0.1 11233
2016-07-21 08:14:25,589 [myid:] - INFO  [main:QuorumBase@291] - 127.0.0.1:11233 
is no longer accepting client connections
2016-07-21 08:14:25,589 [myid:] - INFO  [main:FourLetterWordMain@62] - 
connecting to 127.0.0.1 11234
2016-07-21 08:14:25,589 [myid:] - INFO  [main:QuorumBase@291] - 127.0.0.1:11234 
is no longer accepting client connections
2016-07-21 08:14:25,590 [myid:] - INFO  [main:FourLetterWordMain@62] - 
connecting to 127.0.0.1 11235
2016-07-21 08:14:25,590 [myid:] - INFO  [main:QuorumBase@291] - 127.0.0.1:11235 
is no longer accepting client connections
2016-07-21 08:14:25,590 [myid:] - INFO  [main:FourLetterWordMain@62] - 
connecting to 127.0.0.1 11236
2016-07-21 08:14:25,590 [myid:] - INFO  [main:QuorumBase@291] - 127.0.0.1:11236 
is no longer accepting client connections
2016-07-21 08:14:25,590 [myid:] - INFO  [main:FourLetterWordMain@62] - 
connecting to 127.0.0.1 11237
2016-07-21 08:14:25,590 [myid:] - INFO  [main:QuorumBase@291] - 127.0.0.1:11237 
is no longer accepting client connections
2016-07-21 08:14:25,592 [myid:] - INFO  [main:ZKTestCase$1@60] - SUCCEEDED 
testLateLogs
2016-07-21 08:14:25,592 [myid:] - INFO  [main:ZKTestCase$1@55] - FINISHED 
testLateLogs
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to