[
https://issues.apache.org/jira/browse/ZOOKEEPER-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Han updated ZOOKEEPER-2486:
-----------------------------------
Issue Type: Test (was: Bug)
> Flaky Test: org.apache.zookeeper.test.QuorumZxidSyncTest.testBehindLeader
> -------------------------------------------------------------------------
>
> Key: ZOOKEEPER-2486
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2486
> Project: ZooKeeper
> Issue Type: Test
> Components: tests
> Affects Versions: 3.4.8
> Reporter: Michael Han
> Labels: flaky, flaky-test
> Fix For: 3.5.2
>
>
> From https://builds.apache.org/job/ZooKeeper_branch34_jdk7/1156
> {noformat}
> Error Message
> waiting for server up
> Stacktrace
> junit.framework.AssertionFailedError: waiting for server up
> at
> org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:183)
> at
> org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:113)
> at
> org.apache.zookeeper.test.QuorumZxidSyncTest.testBehindLeader(QuorumZxidSyncTest.java:67)
> at
> org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
> Standard Output
> 2016-07-21 08:11:45,722 [myid:] - INFO [main:PortAssignment@32] - assigning
> port 11221
> 2016-07-21 08:11:45,729 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING
> testBehindLeader
> 2016-07-21 08:11:45,729 [myid:] - INFO [main:QuorumBase@69] -
> QuorumBase.setup null
> 2016-07-21 08:11:45,926 [myid:] - INFO [main:PortAssignment@32] - assigning
> port 11222
> 2016-07-21 08:11:45,926 [myid:] - INFO [main:PortAssignment@32] - assigning
> port 11223
> 2016-07-21 08:11:45,927 [myid:] - INFO [main:PortAssignment@32] - assigning
> port 11224
> 2016-07-21 08:11:45,927 [myid:] - INFO [main:PortAssignment@32] - assigning
> port 11225
> 2016-07-21 08:11:45,927 [myid:] - INFO [main:PortAssignment@32] - assigning
> port 11226
> 2016-07-21 08:11:45,928 [myid:] - INFO [main:PortAssignment@32] - assigning
> port 11227
> 2016-07-21 08:11:45,928 [myid:] - INFO [main:PortAssignment@32] - assigning
> port 11228
> 2016-07-21 08:11:45,928 [myid:] - INFO [main:PortAssignment@32] - assigning
> port 11229
> 2016-07-21 08:11:45,928 [myid:] - INFO [main:PortAssignment@32] - assigning
> port 11230
> 2016-07-21 08:11:45,929 [myid:] - INFO [main:PortAssignment@32] - assigning
> port 11231
> 2016-07-21 08:11:45,929 [myid:] - INFO [main:QuorumBase@93] - Ports are:
> 127.0.0.1:11222,127.0.0.1:11223,127.0.0.1:11224,127.0.0.1:11225,127.0.0.1:11226
> 2016-07-21 08:11:45,946 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] -
> Resolved hostname: 127.0.0.1 to address: /127.0.0.1
> 2016-07-21 08:11:45,946 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] -
> Resolved hostname: 127.0.0.1 to address: /127.0.0.1
> 2016-07-21 08:11:45,948 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] -
> Resolved hostname: 127.0.0.1 to address: /127.0.0.1
> 2016-07-21 08:11:45,948 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] -
> Resolved hostname: 127.0.0.1 to address: /127.0.0.1
> 2016-07-21 08:11:45,949 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] -
> Resolved hostname: 127.0.0.1 to address: /127.0.0.1
> 2016-07-21 08:11:45,949 [myid:] - INFO [main:QuorumBase@142] - creating
> QuorumPeer 1 port 11222
> 2016-07-21 08:11:45,961 [myid:] - INFO [main:NIOServerCnxnFactory@89] -
> binding to port 0.0.0.0/0.0.0.0:11222
> 2016-07-21 08:11:45,979 [myid:] - INFO [main:QuorumBase@145] - creating
> QuorumPeer 2 port 11223
> 2016-07-21 08:11:45,979 [myid:] - INFO [main:NIOServerCnxnFactory@89] -
> binding to port 0.0.0.0/0.0.0.0:11223
> 2016-07-21 08:11:45,980 [myid:] - INFO [main:QuorumBase@148] - creating
> QuorumPeer 3 port 11224
> 2016-07-21 08:11:45,980 [myid:] - INFO [main:NIOServerCnxnFactory@89] -
> binding to port 0.0.0.0/0.0.0.0:11224
> 2016-07-21 08:11:45,981 [myid:] - INFO [main:QuorumBase@151] - creating
> QuorumPeer 4 port 11225
> 2016-07-21 08:11:45,981 [myid:] - INFO [main:NIOServerCnxnFactory@89] -
> binding to port 0.0.0.0/0.0.0.0:11225
> 2016-07-21 08:11:45,981 [myid:] - INFO [main:QuorumBase@154] - creating
> QuorumPeer 5 port 11226
> 2016-07-21 08:11:45,982 [myid:] - INFO [main:NIOServerCnxnFactory@89] -
> binding to port 0.0.0.0/0.0.0.0:11226
> 2016-07-21 08:11:45,982 [myid:] - INFO [main:QuorumBase@163] - QuorumPeer 1
> voting view:
> {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981,
> 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4,
> 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e,
> 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0,
> 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
> 2016-07-21 08:11:45,983 [myid:] - INFO [main:QuorumBase@164] - QuorumPeer 2
> voting view:
> {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981,
> 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4,
> 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e,
> 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0,
> 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
> 2016-07-21 08:11:45,983 [myid:] - INFO [main:QuorumBase@165] - QuorumPeer 3
> voting view:
> {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981,
> 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4,
> 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e,
> 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0,
> 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
> 2016-07-21 08:11:45,983 [myid:] - INFO [main:QuorumBase@166] - QuorumPeer 4
> voting view:
> {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981,
> 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4,
> 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e,
> 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0,
> 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
> 2016-07-21 08:11:45,983 [myid:] - INFO [main:QuorumBase@167] - QuorumPeer 5
> voting view:
> {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981,
> 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4,
> 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e,
> 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0,
> 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
> 2016-07-21 08:11:45,983 [myid:] - INFO [main:QuorumBase@169] - start
> QuorumPeer 1
> 2016-07-21 08:11:45,988 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch
> not found! Creating with a reasonable default of 0. This should only happen
> when you are upgrading your installation
> 2016-07-21 08:11:45,990 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch
> not found! Creating with a reasonable default of 0. This should only happen
> when you are upgrading your installation
> 2016-07-21 08:11:45,996 [myid:] - INFO
> [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port:
> /127.0.0.1:12227
> 2016-07-21 08:11:46,001 [myid:] - INFO [main:QuorumBase@171] - start
> QuorumPeer 2
> 2016-07-21 08:11:46,002 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch
> not found! Creating with a reasonable default of 0. This should only happen
> when you are upgrading your installation
> 2016-07-21 08:11:46,003 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch
> not found! Creating with a reasonable default of 0. This should only happen
> when you are upgrading your installation
> 2016-07-21 08:11:46,004 [myid:] - INFO
> [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port:
> /127.0.0.1:12228
> 2016-07-21 08:11:46,006 [myid:] - INFO [main:QuorumBase@173] - start
> QuorumPeer 3
> 2016-07-21 08:11:46,006 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch
> not found! Creating with a reasonable default of 0. This should only happen
> when you are upgrading your installation
> 2016-07-21 08:11:46,007 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch
> not found! Creating with a reasonable default of 0. This should only happen
> when you are upgrading your installation
> 2016-07-21 08:11:46,007 [myid:] - INFO
> [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11222:QuorumPeer@774] - LOOKING
> 2016-07-21 08:11:46,008 [myid:] - INFO
> [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11223:QuorumPeer@774] - LOOKING
> 2016-07-21 08:11:46,009 [myid:] - INFO
> [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port:
> /127.0.0.1:12229
> 2016-07-21 08:11:46,009 [myid:] - INFO
> [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11222:FastLeaderElection@818] - New
> election. My id = 1, proposed zxid=0x0
> 2016-07-21 08:11:46,009 [myid:] - INFO
> [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11223:FastLeaderElection@818] - New
> election. My id = 2, proposed zxid=0x0
> 2016-07-21 08:11:46,009 [myid:] - INFO [main:QuorumBase@175] - start
> QuorumPeer 4
> 2016-07-21 08:11:46,010 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch
> not found! Creating with a reasonable default of 0. This should only happen
> when you are upgrading your installation
> 2016-07-21 08:11:46,011 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch
> not found! Creating with a reasonable default of 0. This should only happen
> when you are upgrading your installation
> 2016-07-21 08:11:46,011 [myid:] - INFO
> [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11224:QuorumPeer@774] - LOOKING
> 2016-07-21 08:11:46,011 [myid:] - INFO
> [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:38924
> 2016-07-21 08:11:46,011 [myid:] - INFO
> [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11224:FastLeaderElection@818] - New
> election. My id = 3, proposed zxid=0x0
> 2016-07-21 08:11:46,011 [myid:] - INFO
> [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:43415
> 2016-07-21 08:11:46,011 [myid:] - INFO
> [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier,
> so dropping the connection: (2, 1)
> 2016-07-21 08:11:46,011 [myid:] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message
> format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,013 [myid:] - INFO
> [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier,
> so dropping the connection: (3, 1)
> 2016-07-21 08:11:46,014 [myid:] - INFO
> [/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:51548
> 2016-07-21 08:11:46,014 [myid:] - INFO [main:QuorumBase@177] - start
> QuorumPeer 5
> 2016-07-21 08:11:46,015 [myid:] - INFO
> [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port:
> /127.0.0.1:12230
> 2016-07-21 08:11:46,015 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch
> not found! Creating with a reasonable default of 0. This should only happen
> when you are upgrading your installation
> 2016-07-21 08:11:46,015 [myid:] - WARN
> [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 4 at
> election address /127.0.0.1:12230
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
> at
> org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
> at
> org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
> at java.lang.Thread.run(Thread.java:745)
> 2016-07-21 08:11:46,018 [myid:] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message
> format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,018 [myid:] - INFO
> [/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:51551
> 2016-07-21 08:11:46,016 [myid:] - INFO
> [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:43417
> 2016-07-21 08:11:46,016 [myid:] - INFO
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:QuorumPeer@774] - LOOKING
> 2016-07-21 08:11:46,016 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch
> not found! Creating with a reasonable default of 0. This should only happen
> when you are upgrading your installation
> 2016-07-21 08:11:46,019 [myid:] - INFO
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:FastLeaderElection@818] - New
> election. My id = 4, proposed zxid=0x0
> 2016-07-21 08:11:46,019 [myid:] - INFO
> [/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:46102
> 2016-07-21 08:11:46,018 [myid:] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message
> format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,018 [myid:] - INFO
> [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier,
> so dropping the connection: (3, 2)
> 2016-07-21 08:11:46,018 [myid:] - INFO
> [WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname:
> 127.0.0.1 to address: /127.0.0.1
> 2016-07-21 08:11:46,018 [myid:] - INFO
> [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:38928
> 2016-07-21 08:11:46,021 [myid:] - INFO
> [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier,
> so dropping the connection: (4, 2)
> 2016-07-21 08:11:46,021 [myid:] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message
> format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,021 [myid:] - INFO
> [/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:51553
> 2016-07-21 08:11:46,020 [myid:] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message
> format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,019 [myid:] - INFO
> [WorkerSender[myid=3]:QuorumCnxManager@199] - Have smaller server identifier,
> so dropping the connection: (4, 3)
> 2016-07-21 08:11:46,022 [myid:] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message
> format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,021 [myid:] - WARN
> [WorkerSender[myid=2]:QuorumCnxManager@400] - Cannot open channel to 5 at
> election address /127.0.0.1:12231
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
> at
> org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
> at
> org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
> at java.lang.Thread.run(Thread.java:745)
> 2016-07-21 08:11:46,021 [myid:] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message
> format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,021 [myid:] - WARN
> [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 5 at
> election address /127.0.0.1:12231
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
> at
> org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
> at
> org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
> at java.lang.Thread.run(Thread.java:745)
> 2016-07-21 08:11:46,023 [myid:] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message
> format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,023 [myid:] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message
> format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,023 [myid:] - INFO
> [WorkerSender[myid=2]:QuorumPeer$QuorumServer@149] - Resolved hostname:
> 127.0.0.1 to address: /127.0.0.1
> 2016-07-21 08:11:46,024 [myid:] - INFO
> [/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:46104
> 2016-07-21 08:11:46,022 [myid:] - INFO
> [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port:
> /127.0.0.1:12231
> 2016-07-21 08:11:46,022 [myid:] - WARN
> [WorkerSender[myid=3]:QuorumCnxManager@400] - Cannot open channel to 5 at
> election address /127.0.0.1:12231
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
> at
> org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
> at
> org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
> at java.lang.Thread.run(Thread.java:745)
> 2016-07-21 08:11:46,025 [myid:] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message
> format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,025 [myid:] - INFO
> [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message
> format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,025 [myid:] - INFO [main:QuorumBase@179] - started
> QuorumPeer 5
> 2016-07-21 08:11:46,026 [myid:] - INFO [main:QuorumBase@181] - Checking
> ports
> 127.0.0.1:11222,127.0.0.1:11223,127.0.0.1:11224,127.0.0.1:11225,127.0.0.1:11226
> 2016-07-21 08:11:46,025 [myid:] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message
> format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,025 [myid:] - INFO
> [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:43428
> 2016-07-21 08:11:46,027 [myid:] - INFO
> [/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:46109
> 2016-07-21 08:11:46,025 [myid:] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message
> format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,024 [myid:] - INFO
> [WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname:
> 127.0.0.1 to address: /127.0.0.1
> 2016-07-21 08:11:46,027 [myid:] - INFO
> [WorkerSender[myid=4]:QuorumCnxManager@199] - Have smaller server identifier,
> so dropping the connection: (5, 4)
> 2016-07-21 08:11:46,027 [myid:] - INFO
> [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,026 [myid:] - INFO
> [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:38938
> 2016-07-21 08:11:46,025 [myid:] - INFO
> [WorkerSender[myid=3]:QuorumPeer$QuorumServer@149] - Resolved hostname:
> 127.0.0.1 to address: /127.0.0.1
> 2016-07-21 08:11:46,025 [myid:] - INFO
> [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier,
> so dropping the connection: (4, 2)
> 2016-07-21 08:11:46,028 [myid:] - INFO [main:FourLetterWordMain@62] -
> connecting to 127.0.0.1 11222
> 2016-07-21 08:11:46,028 [myid:] - WARN
> [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2,
> my id = 4, error =
> java.net.SocketException: Socket closed
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.net.SocketInputStream.read(SocketInputStream.java:210)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:11:46,029 [myid:] - WARN
> [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:11:46,028 [myid:] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,027 [myid:] - INFO
> [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:QuorumPeer@774] - LOOKING
> 2016-07-21 08:11:46,027 [myid:] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message
> format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,027 [myid:] - INFO
> [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:52086
> 2016-07-21 08:11:46,030 [myid:] - WARN
> [RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4,
> my id = 2, error =
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:11:46,030 [myid:] - WARN
> [SendWorker:4:QuorumCnxManager$SendWorker@732] - Exception when using
> channel: for id 4 my id = 2 error = java.net.SocketException: Broken pipe
> 2016-07-21 08:11:46,029 [myid:] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message
> format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,031 [myid:] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory@192] -
> Accepted socket connection from /127.0.0.1:52363
> 2016-07-21 08:11:46,029 [myid:] - INFO
> [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:38940
> 2016-07-21 08:11:46,029 [myid:] - INFO
> [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier,
> so dropping the connection: (5, 1)
> 2016-07-21 08:11:46,029 [myid:] - INFO
> [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message
> format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,032 [myid:] - INFO
> [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message
> format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,032 [myid:] - INFO
> [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,029 [myid:] - WARN
> [SendWorker:2:QuorumCnxManager$SendWorker@732] - Exception when using
> channel: for id 2 my id = 4 error = java.net.SocketException: Socket closed
> 2016-07-21 08:11:46,033 [myid:] - WARN
> [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:11:46,029 [myid:] - INFO
> [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier,
> so dropping the connection: (5, 2)
> 2016-07-21 08:11:46,028 [myid:] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message
> format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,033 [myid:] - INFO
> [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier,
> so dropping the connection: (5, 1)
> 2016-07-21 08:11:46,032 [myid:] - INFO
> [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message
> format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,031 [myid:] - INFO
> [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:52088
> 2016-07-21 08:11:46,031 [myid:] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,031 [myid:] - INFO
> [/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:46117
> 2016-07-21 08:11:46,030 [myid:] - WARN
> [SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:11:46,030 [myid:] - WARN
> [RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:11:46,030 [myid:] - INFO
> [WorkerSender[myid=3]:QuorumCnxManager@199] - Have smaller server identifier,
> so dropping the connection: (5, 3)
> 2016-07-21 08:11:46,030 [myid:] - INFO
> [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:FastLeaderElection@818] - New
> election. My id = 5, proposed zxid=0x0
> 2016-07-21 08:11:46,030 [myid:] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,034 [myid:] - INFO
> [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:38947
> 2016-07-21 08:11:46,034 [myid:] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message
> format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,034 [myid:] - INFO
> [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message
> format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,033 [myid:] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,035 [myid:] - INFO
> [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:52089
> 2016-07-21 08:11:46,035 [myid:] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,036 [myid:] - INFO
> [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,036 [myid:] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message
> format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,037 [myid:] - INFO
> [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:43442
> 2016-07-21 08:11:46,036 [myid:] - INFO
> [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,036 [myid:] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,037 [myid:] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,037 [myid:] - INFO
> [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,038 [myid:] - INFO
> [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:52091
> 2016-07-21 08:11:46,037 [myid:] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message
> format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,036 [myid:] - INFO
> [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier,
> so dropping the connection: (5, 1)
> 2016-07-21 08:11:46,038 [myid:] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,038 [myid:] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,038 [myid:] - INFO
> [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,038 [myid:] - INFO
> [/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:51573
> 2016-07-21 08:11:46,038 [myid:] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,037 [myid:] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,039 [myid:] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,039 [myid:] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,039 [myid:] - INFO
> [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,040 [myid:] - INFO
> [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,039 [myid:] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,039 [myid:] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,039 [myid:] - INFO
> [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:52093
> 2016-07-21 08:11:46,040 [myid:] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,040 [myid:] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,040 [myid:] - INFO
> [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,040 [myid:] - INFO
> [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,041 [myid:] - INFO
> [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,041 [myid:] - INFO
> [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:43444
> 2016-07-21 08:11:46,041 [myid:] - INFO
> [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,041 [myid:] - WARN
> [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1,
> my id = 5, error =
> java.net.SocketException: Socket closed
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.io.DataInputStream.readFully(DataInputStream.java:195)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:805)
> 2016-07-21 08:11:46,042 [myid:] - WARN
> [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:11:46,041 [myid:] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,041 [myid:] - WARN
> [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:11:46,041 [myid:] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11222:NIOServerCnxn@827] - Processing
> stat command from /127.0.0.1:52363
> 2016-07-21 08:11:46,041 [myid:] - WARN
> [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5,
> my id = 1, error =
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:11:46,043 [myid:] - WARN
> [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:11:46,041 [myid:] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,041 [myid:] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,043 [myid:] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,043 [myid:] - WARN
> [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:11:46,042 [myid:] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,042 [myid:] - WARN
> [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:11:46,044 [myid:] - WARN
> [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:11:46,042 [myid:] - INFO
> [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,042 [myid:] - INFO
> [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:52096
> 2016-07-21 08:11:46,041 [myid:] - INFO
> [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message
> format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,045 [myid:] - WARN
> [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:11:46,045 [myid:] - WARN
> [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:11:46,045 [myid:] - INFO [Thread-2:NIOServerCnxn@1008] -
> Closed socket connection for client /127.0.0.1:52363 (no session established
> for client)
> 2016-07-21 08:11:46,044 [myid:] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,044 [myid:] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,044 [myid:] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,046 [myid:] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,045 [myid:] - INFO
> [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection
> request /127.0.0.1:43446
> 2016-07-21 08:11:46,045 [myid:] - INFO
> [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,045 [myid:] - WARN
> [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5,
> my id = 1, error =
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:11:46,047 [myid:] - WARN
> [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:11:46,045 [myid:] - WARN
> [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1,
> my id = 5, error =
> java.net.SocketException: Socket closed
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.net.SocketInputStream.read(SocketInputStream.java:210)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:11:46,047 [myid:] - WARN
> [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:11:46,047 [myid:] - INFO
> [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,047 [myid:] - WARN
> [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:11:46,048 [myid:] - WARN
> [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:11:46,046 [myid:] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,048 [myid:] - INFO
> [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,048 [myid:] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,049 [myid:] - INFO
> [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,049 [myid:] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,049 [myid:] - INFO
> [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message
> format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,245 [myid:] - INFO
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:QuorumPeer@844] - FOLLOWING
> 2016-07-21 08:11:46,246 [myid:] - INFO
> [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11223:QuorumPeer@844] - FOLLOWING
> 2016-07-21 08:11:46,247 [myid:] - INFO
> [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11224:QuorumPeer@844] - FOLLOWING
> 2016-07-21 08:11:46,249 [myid:] - INFO
> [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11222:QuorumPeer@844] - FOLLOWING
> 2016-07-21 08:11:46,249 [myid:] - INFO
> [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:QuorumPeer@856] - LEADING
> 2016-07-21 08:11:46,251 [myid:] - INFO
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Learner@86] - TCP NoDelay set to:
> true
> 2016-07-21 08:11:46,255 [myid:] - INFO
> [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:Leader@59] - TCP NoDelay set to:
> true
> 2016-07-21 08:11:46,259 [myid:] - INFO
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server
> environment:zookeeper.version=3.4.9-SNAPSHOT-1753645, built on 07/21/2016
> 07:46 GMT
> 2016-07-21 08:11:46,259 [myid:] - INFO
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server
> environment:host.name=asf907.gq1.ygridcore.net
> 2016-07-21 08:11:46,260 [myid:] - INFO
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server
> environment:java.version=1.7.0_80
> 2016-07-21 08:11:46,260 [myid:] - INFO
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server
> environment:java.vendor=Oracle Corporation
> 2016-07-21 08:11:46,260 [myid:] - INFO
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server
> environment:java.home=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/jre
> 2016-07-21 08:11:46,260 [myid:] - INFO
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server
> environment:java.class.path=/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/antlr-2.7.6.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/checkstyle-5.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-beanutils-core-1.7.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-cli-1.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-lang-1.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-logging-1.0.3.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/google-collections-0.9.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/junit-4.8.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/jline-0.9.94.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/log4j-1.2.16.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/netty-3.10.5.Final.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/slf4j-api-1.6.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/slf4j-log4j12-1.6.1.jar:/home/jenkins/tools/ant/apache-ant-1.9.4/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar
> 2016-07-21 08:11:46,260 [myid:] - INFO
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 2016-07-21 08:11:46,260 [myid:] - INFO
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server
> environment:java.io.tmpdir=/tmp
> 2016-07-21 08:11:46,260 [myid:] - INFO
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server
> environment:java.compiler=<NA>
> 2016-07-21 08:11:46,260 [myid:] - INFO
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server
> environment:os.name=Linux
> 2016-07-21 08:11:46,260 [myid:] - INFO
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server
> environment:os.arch=amd64
> 2016-07-21 08:11:46,260 [myid:] - INFO
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server
> environment:os.version=3.13.0-36-lowlatency
> 2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:
> ...[truncated 1420555 chars]...
> ager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:21,678 [myid:] - WARN
> [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:21,672 [myid:] - WARN
> [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1,
> my id = 5, error =
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:21,679 [myid:] - WARN
> [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:21,672 [myid:] - WARN
> [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:21,679 [myid:] - WARN
> [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:21,679 [myid:] - WARN
> [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:21,679 [myid:] - WARN
> [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:21,678 [myid:] - WARN
> [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:21,679 [myid:] - WARN
> [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:21,677 [myid:] - WARN
> [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:21,680 [myid:] - WARN
> [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:21,675 [myid:] - WARN
> [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:21,680 [myid:] - WARN
> [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:22,000 [myid:] - INFO
> [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
> 2016-07-21 08:14:22,317 [myid:] - INFO
> [/127.0.0.1:12241:QuorumCnxManager$Listener@560] - Leaving listener
> 2016-07-21 08:14:22,320 [myid:] - INFO
> [/127.0.0.1:12242:QuorumCnxManager$Listener@560] - Leaving listener
> 2016-07-21 08:14:22,579 [myid:] - INFO
> [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:Follower@166] - shutdown called
> java.lang.Exception: shutdown Follower
> at
> org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
> 2016-07-21 08:14:22,579 [myid:] - INFO
> [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:FollowerZooKeeperServer@140] -
> Shutting down
> 2016-07-21 08:14:22,579 [myid:] - INFO
> [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:SyncRequestProcessor@209] -
> Shutting down
> 2016-07-21 08:14:22,579 [myid:] - WARN
> [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:QuorumPeer@874] - QuorumPeer main
> thread exited
> 2016-07-21 08:14:22,582 [myid:] - INFO [main:QuorumBase@306] - Shutting down
> quorum peer QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234
> 2016-07-21 08:14:22,582 [myid:] - INFO [main:Follower@166] - shutdown called
> java.lang.Exception: shutdown Follower
> at
> org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891)
> at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307)
> at
> org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:298)
> at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285)
> at
> org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
> at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
> at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
> at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
> 2016-07-21 08:14:22,582 [myid:] - INFO [main:FollowerZooKeeperServer@140] -
> Shutting down
> 2016-07-21 08:14:22,582 [myid:] - INFO [main:ZooKeeperServer@469] - shutting
> down
> 2016-07-21 08:14:22,582 [myid:] - INFO [main:FollowerRequestProcessor@107] -
> Shutting down
> 2016-07-21 08:14:22,582 [myid:] - INFO [main:CommitProcessor@184] - Shutting
> down
> 2016-07-21 08:14:22,582 [myid:] - INFO
> [FollowerRequestProcessor:2:FollowerRequestProcessor@97] -
> FollowerRequestProcessor exited loop!
> 2016-07-21 08:14:22,582 [myid:] - INFO
> [CommitProcessor:2:CommitProcessor@153] - CommitProcessor exited loop!
> 2016-07-21 08:14:22,582 [myid:] - INFO [main:FinalRequestProcessor@402] -
> shutdown of request processor complete
> 2016-07-21 08:14:22,583 [myid:] - INFO [main:SyncRequestProcessor@209] -
> Shutting down
> 2016-07-21 08:14:22,583 [myid:] - INFO
> [SyncThread:2:SyncRequestProcessor@187] - SyncRequestProcessor exited!
> 2016-07-21 08:14:22,584 [myid:] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11234:NIOServerCnxnFactory@219] -
> NIOServerCnxn factory exited run method
> 2016-07-21 08:14:22,585 [myid:] - WARN
> [SendWorker:3:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:22,585 [myid:] - WARN
> [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2,
> my id = 4, error =
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:22,585 [myid:] - WARN
> [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:22,585 [myid:] - WARN
> [RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id 3,
> my id = 2, error =
> java.net.SocketException: Socket closed
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.net.SocketInputStream.read(SocketInputStream.java:210)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:22,586 [myid:] - WARN
> [RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:22,585 [myid:] - WARN
> [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2,
> my id = 5, error =
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:22,586 [myid:] - WARN
> [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:22,585 [myid:] - WARN
> [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5,
> my id = 2, error =
> java.net.SocketException: Socket closed
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.net.SocketInputStream.read(SocketInputStream.java:210)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:22,587 [myid:] - WARN
> [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:22,585 [myid:] - WARN
> [SendWorker:3:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:22,585 [myid:] - WARN
> [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2,
> my id = 3, error =
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:22,587 [myid:] - WARN
> [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:22,585 [myid:] - WARN
> [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:22,587 [myid:] - WARN
> [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:22,585 [myid:] - ERROR
> [/127.0.0.1:12239:QuorumCnxManager$Listener@547] - Exception while listening
> java.net.SocketException: Socket closed
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
> 2016-07-21 08:14:22,587 [myid:] - WARN
> [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:22,588 [myid:] - WARN
> [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:22,587 [myid:] - WARN
> [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:22,589 [myid:] - WARN
> [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:22,586 [myid:] - WARN
> [SendWorker:4:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:22,589 [myid:] - WARN
> [SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:22,586 [myid:] - WARN
> [RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4,
> my id = 2, error =
> java.net.SocketException: Socket closed
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.net.SocketInputStream.read(SocketInputStream.java:210)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:22,589 [myid:] - WARN
> [RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:22,586 [myid:] - INFO [main:QuorumBase@310] - Shutting down
> leader election QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234
> 2016-07-21 08:14:22,589 [myid:] - INFO [main:QuorumBase@315] - Waiting for
> QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234 to exit thread
> 2016-07-21 08:14:22,585 [myid:] - WARN
> [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:22,590 [myid:] - WARN
> [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:22,673 [myid:] - INFO
> [/127.0.0.1:12238:QuorumCnxManager$Listener@560] - Leaving listener
> 2016-07-21 08:14:23,079 [myid:] - INFO
> [WorkerSender[myid=3]:FastLeaderElection$Messenger$WorkerSender@438] -
> WorkerSender is down
> 2016-07-21 08:14:23,080 [myid:] - INFO
> [WorkerSender[myid=4]:FastLeaderElection$Messenger$WorkerSender@438] -
> WorkerSender is down
> 2016-07-21 08:14:23,084 [myid:] - INFO
> [WorkerSender[myid=5]:FastLeaderElection$Messenger$WorkerSender@438] -
> WorkerSender is down
> 2016-07-21 08:14:23,088 [myid:] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@407] -
> WorkerReceiver is down
> 2016-07-21 08:14:23,088 [myid:] - INFO
> [WorkerReceiver[myid=4]:FastLeaderElection$Messenger$WorkerReceiver@407] -
> WorkerReceiver is down
> 2016-07-21 08:14:23,089 [myid:] - INFO
> [WorkerReceiver[myid=5]:FastLeaderElection$Messenger$WorkerReceiver@407] -
> WorkerReceiver is down
> 2016-07-21 08:14:23,578 [myid:] - INFO
> [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:Follower@166] - shutdown called
> java.lang.Exception: shutdown Follower
> at
> org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
> 2016-07-21 08:14:23,578 [myid:] - INFO
> [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:FollowerZooKeeperServer@140] -
> Shutting down
> 2016-07-21 08:14:23,578 [myid:] - INFO
> [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:SyncRequestProcessor@209] -
> Shutting down
> 2016-07-21 08:14:23,578 [myid:] - WARN
> [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:QuorumPeer@874] - QuorumPeer main
> thread exited
> 2016-07-21 08:14:23,579 [myid:] - INFO [main:QuorumBase@306] - Shutting down
> quorum peer QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235
> 2016-07-21 08:14:23,579 [myid:] - INFO [main:Follower@166] - shutdown called
> java.lang.Exception: shutdown Follower
> at
> org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891)
> at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307)
> at
> org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:299)
> at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285)
> at
> org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
> at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
> at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
> at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
> 2016-07-21 08:14:23,579 [myid:] - INFO [main:FollowerZooKeeperServer@140] -
> Shutting down
> 2016-07-21 08:14:23,579 [myid:] - INFO [main:ZooKeeperServer@469] - shutting
> down
> 2016-07-21 08:14:23,580 [myid:] - INFO [main:FollowerRequestProcessor@107] -
> Shutting down
> 2016-07-21 08:14:23,580 [myid:] - INFO [main:CommitProcessor@184] - Shutting
> down
> 2016-07-21 08:14:23,580 [myid:] - INFO
> [FollowerRequestProcessor:3:FollowerRequestProcessor@97] -
> FollowerRequestProcessor exited loop!
> 2016-07-21 08:14:23,580 [myid:] - INFO
> [CommitProcessor:3:CommitProcessor@153] - CommitProcessor exited loop!
> 2016-07-21 08:14:23,580 [myid:] - INFO [main:FinalRequestProcessor@402] -
> shutdown of request processor complete
> 2016-07-21 08:14:23,581 [myid:] - INFO [main:SyncRequestProcessor@209] -
> Shutting down
> 2016-07-21 08:14:23,581 [myid:] - INFO
> [SyncThread:3:SyncRequestProcessor@187] - SyncRequestProcessor exited!
> 2016-07-21 08:14:23,582 [myid:] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11235:NIOServerCnxnFactory@219] -
> NIOServerCnxn factory exited run method
> 2016-07-21 08:14:23,583 [myid:] - ERROR
> [/127.0.0.1:12240:QuorumCnxManager$Listener@547] - Exception while listening
> java.net.SocketException: Socket closed
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
> 2016-07-21 08:14:23,583 [myid:] - WARN
> [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:23,585 [myid:] - WARN
> [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:23,584 [myid:] - WARN
> [RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4,
> my id = 3, error =
> java.net.SocketException: Socket closed
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.net.SocketInputStream.read(SocketInputStream.java:210)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:23,585 [myid:] - WARN
> [RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:23,584 [myid:] - WARN
> [SendWorker:4:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:23,585 [myid:] - WARN
> [SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:23,584 [myid:] - WARN
> [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5,
> my id = 3, error =
> java.net.SocketException: Socket closed
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.net.SocketInputStream.read(SocketInputStream.java:210)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:23,585 [myid:] - WARN
> [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:23,584 [myid:] - WARN
> [RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id 3,
> my id = 5, error =
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:23,586 [myid:] - WARN
> [RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:23,584 [myid:] - WARN
> [RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id 3,
> my id = 4, error =
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:23,586 [myid:] - WARN
> [RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:23,584 [myid:] - INFO [main:QuorumBase@310] - Shutting down
> leader election QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235
> 2016-07-21 08:14:23,586 [myid:] - INFO [main:QuorumBase@315] - Waiting for
> QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235 to exit thread
> 2016-07-21 08:14:23,586 [myid:] - WARN
> [SendWorker:3:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:23,586 [myid:] - WARN
> [SendWorker:3:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:23,586 [myid:] - WARN
> [SendWorker:3:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:23,587 [myid:] - WARN
> [SendWorker:3:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:23,588 [myid:] - INFO
> [/127.0.0.1:12239:QuorumCnxManager$Listener@560] - Leaving listener
> 2016-07-21 08:14:24,355 [myid:] - INFO
> [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@438] -
> WorkerSender is down
> 2016-07-21 08:14:24,358 [myid:] - INFO
> [WorkerSender[myid=1]:FastLeaderElection$Messenger$WorkerSender@438] -
> WorkerSender is down
> 2016-07-21 08:14:24,360 [myid:] - INFO
> [WorkerSender[myid=3]:FastLeaderElection$Messenger$WorkerSender@438] -
> WorkerSender is down
> 2016-07-21 08:14:24,361 [myid:] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@407] -
> WorkerReceiver is down
> 2016-07-21 08:14:24,361 [myid:] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@407] -
> WorkerReceiver is down
> 2016-07-21 08:14:24,362 [myid:] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@407] -
> WorkerReceiver is down
> 2016-07-21 08:14:24,578 [myid:] - INFO
> [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:Follower@166] - shutdown called
> java.lang.Exception: shutdown Follower
> at
> org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
> 2016-07-21 08:14:24,578 [myid:] - INFO
> [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:FollowerZooKeeperServer@140] -
> Shutting down
> 2016-07-21 08:14:24,578 [myid:] - INFO
> [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:SyncRequestProcessor@209] -
> Shutting down
> 2016-07-21 08:14:24,578 [myid:] - WARN
> [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:QuorumPeer@874] - QuorumPeer main
> thread exited
> 2016-07-21 08:14:24,579 [myid:] - INFO [main:QuorumBase@306] - Shutting down
> quorum peer QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236
> 2016-07-21 08:14:24,579 [myid:] - INFO [main:Follower@166] - shutdown called
> java.lang.Exception: shutdown Follower
> at
> org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891)
> at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307)
> at
> org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:300)
> at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285)
> at
> org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
> at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
> at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
> at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
> 2016-07-21 08:14:24,580 [myid:] - INFO [main:FollowerZooKeeperServer@140] -
> Shutting down
> 2016-07-21 08:14:24,581 [myid:] - INFO [main:ZooKeeperServer@469] - shutting
> down
> 2016-07-21 08:14:24,581 [myid:] - INFO [main:FollowerRequestProcessor@107] -
> Shutting down
> 2016-07-21 08:14:24,581 [myid:] - INFO [main:CommitProcessor@184] - Shutting
> down
> 2016-07-21 08:14:24,581 [myid:] - INFO [main:FinalRequestProcessor@402] -
> shutdown of request processor complete
> 2016-07-21 08:14:24,581 [myid:] - INFO
> [FollowerRequestProcessor:4:FollowerRequestProcessor@97] -
> FollowerRequestProcessor exited loop!
> 2016-07-21 08:14:24,581 [myid:] - INFO [main:SyncRequestProcessor@209] -
> Shutting down
> 2016-07-21 08:14:24,581 [myid:] - INFO
> [CommitProcessor:4:CommitProcessor@153] - CommitProcessor exited loop!
> 2016-07-21 08:14:24,582 [myid:] - INFO
> [SyncThread:4:SyncRequestProcessor@187] - SyncRequestProcessor exited!
> 2016-07-21 08:14:24,583 [myid:] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11236:NIOServerCnxnFactory@219] -
> NIOServerCnxn factory exited run method
> 2016-07-21 08:14:24,585 [myid:] - INFO [main:QuorumBase@310] - Shutting down
> leader election QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236
> 2016-07-21 08:14:24,586 [myid:] - ERROR
> [/127.0.0.1:12241:QuorumCnxManager$Listener@547] - Exception while listening
> java.net.SocketException: Socket closed
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
> 2016-07-21 08:14:24,586 [myid:] - WARN
> [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:24,586 [myid:] - WARN
> [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:24,586 [myid:] - INFO [main:QuorumBase@315] - Waiting for
> QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236 to exit thread
> 2016-07-21 08:14:24,586 [myid:] - WARN
> [RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4,
> my id = 5, error =
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:24,587 [myid:] - WARN
> [RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:24,585 [myid:] - INFO
> [/127.0.0.1:12240:QuorumCnxManager$Listener@560] - Leaving listener
> 2016-07-21 08:14:24,585 [myid:] - WARN
> [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5,
> my id = 4, error =
> java.net.SocketException: Socket closed
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.net.SocketInputStream.read(SocketInputStream.java:210)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:24,588 [myid:] - WARN
> [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:24,588 [myid:] - WARN
> [SendWorker:4:QuorumCnxManager$SendWorker@727] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:24,588 [myid:] - WARN
> [SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:25,579 [myid:] - INFO
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:Follower@166] - shutdown called
> java.lang.Exception: shutdown Follower
> at
> org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
> 2016-07-21 08:14:25,579 [myid:] - INFO
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:FollowerZooKeeperServer@140] -
> Shutting down
> 2016-07-21 08:14:25,579 [myid:] - INFO
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:SyncRequestProcessor@209] -
> Shutting down
> 2016-07-21 08:14:25,579 [myid:] - WARN
> [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:QuorumPeer@874] - QuorumPeer main
> thread exited
> 2016-07-21 08:14:25,579 [myid:] - INFO [main:QuorumBase@306] - Shutting down
> quorum peer QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237
> 2016-07-21 08:14:25,579 [myid:] - INFO [main:Leader@496] - Shutting down
> 2016-07-21 08:14:25,580 [myid:] - INFO [main:Leader@502] - Shutdown called
> java.lang.Exception: shutdown Leader! reason: quorum Peer shutdown
> at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:502)
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:888)
> at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307)
> at
> org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:301)
> at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285)
> at
> org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
> at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
> at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
> at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
> 2016-07-21 08:14:25,581 [myid:] - INFO [main:ZooKeeperServer@469] - shutting
> down
> 2016-07-21 08:14:25,581 [myid:] - INFO
> [LearnerCnxAcceptor-/127.0.0.1:12237:Leader$LearnerCnxAcceptor@325] -
> exception while shutting down acceptor: java.net.SocketException: Socket
> closed
> 2016-07-21 08:14:25,582 [myid:] - INFO [main:SessionTrackerImpl@225] -
> Shutting down
> 2016-07-21 08:14:25,582 [myid:] - INFO [main:PrepRequestProcessor@765] -
> Shutting down
> 2016-07-21 08:14:25,582 [myid:] - INFO [main:ProposalRequestProcessor@88] -
> Shutting down
> 2016-07-21 08:14:25,582 [myid:] - INFO [main:CommitProcessor@184] - Shutting
> down
> 2016-07-21 08:14:25,582 [myid:] - INFO [ProcessThread(sid:5
> cport:-1)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
> 2016-07-21 08:14:25,583 [myid:] - INFO
> [CommitProcessor:5:CommitProcessor@153] - CommitProcessor exited loop!
> 2016-07-21 08:14:25,582 [myid:] - INFO
> [main:Leader$ToBeAppliedRequestProcessor@661] - Shutting down
> 2016-07-21 08:14:25,583 [myid:] - INFO [main:FinalRequestProcessor@402] -
> shutdown of request processor complete
> 2016-07-21 08:14:25,583 [myid:] - INFO [main:SyncRequestProcessor@209] -
> Shutting down
> 2016-07-21 08:14:25,583 [myid:] - INFO
> [SyncThread:5:SyncRequestProcessor@187] - SyncRequestProcessor exited!
> 2016-07-21 08:14:25,585 [myid:] - WARN
> [LearnerHandler-/127.0.0.1:43485:LearnerHandler@644] - ******* GOODBYE
> /127.0.0.1:43485 ********
> 2016-07-21 08:14:25,585 [myid:] - WARN
> [LearnerHandler-/127.0.0.1:43486:LearnerHandler@644] - ******* GOODBYE
> /127.0.0.1:43486 ********
> 2016-07-21 08:14:25,585 [myid:] - WARN
> [LearnerHandler-/127.0.0.1:43486:LearnerHandler@656] - Ignoring unexpected
> exception
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
> at
> org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
> at
> org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
> 2016-07-21 08:14:25,585 [myid:] - WARN
> [LearnerHandler-/127.0.0.1:43485:LearnerHandler@656] - Ignoring unexpected
> exception
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
> at
> org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
> at
> org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
> 2016-07-21 08:14:25,585 [myid:] - WARN
> [LearnerHandler-/127.0.0.1:43488:LearnerHandler@644] - ******* GOODBYE
> /127.0.0.1:43488 ********
> 2016-07-21 08:14:25,586 [myid:] - INFO
> [/127.0.0.1:12241:QuorumCnxManager$Listener@560] - Leaving listener
> 2016-07-21 08:14:25,586 [myid:] - WARN
> [LearnerHandler-/127.0.0.1:43487:LearnerHandler@644] - ******* GOODBYE
> /127.0.0.1:43487 ********
> 2016-07-21 08:14:25,586 [myid:] - WARN
> [LearnerHandler-/127.0.0.1:43487:LearnerHandler@656] - Ignoring unexpected
> exception
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
> at
> org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
> at
> org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
> 2016-07-21 08:14:25,586 [myid:] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11237:NIOServerCnxnFactory@219] -
> NIOServerCnxn factory exited run method
> 2016-07-21 08:14:25,586 [myid:] - WARN
> [LearnerHandler-/127.0.0.1:43488:LearnerHandler@656] - Ignoring unexpected
> exception
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
> at
> org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
> at
> org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
> 2016-07-21 08:14:25,587 [myid:] - WARN
> [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237:QuorumPeer@862] - Unexpected
> exception
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:456)
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:859)
> 2016-07-21 08:14:25,588 [myid:] - INFO
> [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237:Leader@496] - Shutting down
> 2016-07-21 08:14:25,588 [myid:] - INFO [main:QuorumBase@310] - Shutting down
> leader election QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237
> 2016-07-21 08:14:25,588 [myid:] - ERROR
> [/127.0.0.1:12242:QuorumCnxManager$Listener@547] - Exception while listening
> java.net.SocketException: Socket closed
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
> 2016-07-21 08:14:25,588 [myid:] - INFO [main:QuorumBase@315] - Waiting for
> QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237 to exit thread
> 2016-07-21 08:14:25,588 [myid:] - WARN
> [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237:QuorumPeer@874] - QuorumPeer main
> thread exited
> 2016-07-21 08:14:25,589 [myid:] - INFO [main:FourLetterWordMain@62] -
> connecting to 127.0.0.1 11233
> 2016-07-21 08:14:25,589 [myid:] - INFO [main:QuorumBase@291] -
> 127.0.0.1:11233 is no longer accepting client connections
> 2016-07-21 08:14:25,589 [myid:] - INFO [main:FourLetterWordMain@62] -
> connecting to 127.0.0.1 11234
> 2016-07-21 08:14:25,589 [myid:] - INFO [main:QuorumBase@291] -
> 127.0.0.1:11234 is no longer accepting client connections
> 2016-07-21 08:14:25,590 [myid:] - INFO [main:FourLetterWordMain@62] -
> connecting to 127.0.0.1 11235
> 2016-07-21 08:14:25,590 [myid:] - INFO [main:QuorumBase@291] -
> 127.0.0.1:11235 is no longer accepting client connections
> 2016-07-21 08:14:25,590 [myid:] - INFO [main:FourLetterWordMain@62] -
> connecting to 127.0.0.1 11236
> 2016-07-21 08:14:25,590 [myid:] - INFO [main:QuorumBase@291] -
> 127.0.0.1:11236 is no longer accepting client connections
> 2016-07-21 08:14:25,590 [myid:] - INFO [main:FourLetterWordMain@62] -
> connecting to 127.0.0.1 11237
> 2016-07-21 08:14:25,590 [myid:] - INFO [main:QuorumBase@291] -
> 127.0.0.1:11237 is no longer accepting client connections
> 2016-07-21 08:14:25,592 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED
> testLateLogs
> 2016-07-21 08:14:25,592 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED
> testLateLogs
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
