Hello,
I'm having problems using ZK in read-only mode.
I am using a fresh install of ZK 3.5.1-alpha where I configure the
installation to be in a 3 nodes ensemble.
I start this node only (QA-E8WIN11) with -Dreadonlymode.enabled=true.
I see that the node QA-E8WIN11 starts the RO server properly:
2016-07-12 16:10:10,955 [myid:1] - INFO
[Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started
Then I make a read-only enabled client:
ZooKeeper zk = new ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,
*QA-E8WIN11*:2181", 45000, this, true);
System.out.println("Successfully established the connection with
ZooKeeper");
zk.getData("/", false, null);
System.out.println("Done.");
Note that only *QA-E8WIN11* is started, the other 2 are not.
When the ZK client picks *QA-E8WIN11* first, it works.
But when it picks a different node, it fails with:
16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating client
connection, connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
sessionTimeout=45000 watcher=TestZkReadOnly@68de145
Successfully established the connection with ZooKeeper
16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
connection to server QA-E8WIN13/10.11.8.230:2181. Will not attempt to
authenticate using SASL (unknown error)
16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client session timed
out, have not heard from server in 15001ms for sessionid 0x0, closing
socket connection and attempting reconnect
16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
connection to server QA-E8WIN12/10.11.8.232:2181. Will not attempt to
authenticate using SASL (unknown error)
Exception in thread "main"
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
at TestZkReadOnly.run(TestZkReadOnly.java:26)
at TestZkReadOnly.main(TestZkReadOnly.java:16)
I'm attaching all the logs. The server and the client are on the same
machine
*QA-E8WIN11.*
Any help would be greatly appreciated!
Thanks,
Benjamin Jaton
16:10:13.320 org.apache.zookeeper.ZooKeeper:109 - Client environment:zookeeper.version=3.5.1-alpha-1693007, built on 07/28/2015 07:19 GMT
16:10:13.324 org.apache.zookeeper.ZooKeeper:109 - Client environment:host.name=QA-E8WIN11
16:10:13.324 org.apache.zookeeper.ZooKeeper:109 - Client environment:java.version=1.8.0_91
16:10:13.324 org.apache.zookeeper.ZooKeeper:109 - Client environment:java.vendor=Oracle Corporation
16:10:13.327 org.apache.zookeeper.ZooKeeper:109 - Client environment:java.home=C:\radiantone\vds\jdk\jre
16:10:13.328 org.apache.zookeeper.ZooKeeper:109 - Client environment:java.class.path=C:\wksp\test-clean-zk\bin;C:\radiantone\zookeeper-3.5.1-alpha.jar;C:\wksp\test-clean-zk\resources;C:\zookeeper-3.5.1-alpha\lib\commons-cli-1.2.jar;C:\zookeeper-3.5.1-alpha\lib\jackson-core-asl-1.9.11.jar;C:\zookeeper-3.5.1-alpha\lib\jackson-mapper-asl-1.9.11.jar;C:\zookeeper-3.5.1-alpha\lib\javacc.jar;C:\zookeeper-3.5.1-alpha\lib\jetty-6.1.26.jar;C:\zookeeper-3.5.1-alpha\lib\jetty-util-6.1.26.jar;C:\zookeeper-3.5.1-alpha\lib\jline-2.11.jar;C:\zookeeper-3.5.1-alpha\lib\log4j-1.2.16.jar;C:\zookeeper-3.5.1-alpha\lib\netty-3.7.0.Final.jar;C:\zookeeper-3.5.1-alpha\lib\servlet-api-2.5-20081211.jar;C:\zookeeper-3.5.1-alpha\lib\slf4j-api-1.7.5.jar;C:\zookeeper-3.5.1-alpha\lib\slf4j-log4j12-1.7.5.jar
16:10:13.328 org.apache.zookeeper.ZooKeeper:109 - Client environment:java.library.path=C:\radiantone\vds\jdk\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Notepad++;C:\unboundid-ldapsdk-2.3.7-se\tools;C:\Program Files\Java\jdk1.7.0_71\bin;C:\Program Files (x86)\GnuWin32\bin;.
16:10:13.328 org.apache.zookeeper.ZooKeeper:109 - Client environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\2\
16:10:13.329 org.apache.zookeeper.ZooKeeper:109 - Client environment:java.compiler=<NA>
16:10:13.329 org.apache.zookeeper.ZooKeeper:109 - Client environment:os.name=Windows Server 2008 R2
16:10:13.329 org.apache.zookeeper.ZooKeeper:109 - Client environment:os.arch=amd64
16:10:13.329 org.apache.zookeeper.ZooKeeper:109 - Client environment:os.version=6.1
16:10:13.329 org.apache.zookeeper.ZooKeeper:109 - Client environment:user.name=Administrator
16:10:13.329 org.apache.zookeeper.ZooKeeper:109 - Client environment:user.home=C:\Users\Administrator
16:10:13.329 org.apache.zookeeper.ZooKeeper:109 - Client environment:user.dir=C:\wksp\test-clean-zk
16:10:13.329 org.apache.zookeeper.ZooKeeper:109 - Client environment:os.memory.free=119MB
16:10:13.332 org.apache.zookeeper.ZooKeeper:109 - Client environment:os.memory.max=1820MB
16:10:13.332 org.apache.zookeeper.ZooKeeper:109 - Client environment:os.memory.total=123MB
16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating client connection, connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181 sessionTimeout=45000 watcher=TestZkReadOnly@68de145
Successfully established the connection with ZooKeeper
16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening socket connection to server QA-E8WIN13/10.11.8.230:2181. Will not attempt to authenticate using SASL (unknown error)
16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client session timed out, have not heard from server in 15001ms for sessionid 0x0, closing socket connection and attempting reconnect
16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening socket connection to server QA-E8WIN12/10.11.8.232:2181. Will not attempt to authenticate using SASL (unknown error)
Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
at TestZkReadOnly.run(TestZkReadOnly.java:26)
at TestZkReadOnly.main(TestZkReadOnly.java:16)
C:\zookeeper-3.5.1-alpha\bin>call C:\jdk\bin\java "-Dzookeeper.log.dir=C:\zookeeper-3.5.1-alpha\bin\..\logs" "-Dreadonlymode.enabled=true" "-Dzookeeper.root.logger=INFO,CONSOLE" "-Dzookeeper.log.file=zookeeper-Administrator-server-QA-E8WIN11.log" "-XX:+HeapDumpOnOutOfMemoryError" "-XX:OnOutOfMemoryError=cmd /c taskkill /pid %%p /t /f" -cp "C:\zookeeper-3.5.1-alpha\bin\..\build\classes;C:\zookeeper-3.5.1-alpha\bin\..\build\lib\*;C:\zookeeper-3.5.1-alpha\bin\..\*;C:\zookeeper-3.5.1-alpha\bin\..\lib\*;C:\zookeeper-3.5.1-alpha\bin\..\conf" org.apache.zookeeper.server.quorum.QuorumPeerMain "C:\zookeeper-3.5.1-alpha\bin\..\conf\zoo.cfg"
2016-07-12 16:10:05,404 [myid:] - INFO [main:QuorumPeerConfig@114] - Reading configuration from: C:\zookeeper-3.5.1-alpha\bin\..\conf\zoo.cfg
2016-07-12 16:10:05,404 [myid:] - INFO [main:QuorumPeerConfig@316] - clientPortAddress is 0.0.0.0/0.0.0.0:2181
2016-07-12 16:10:05,404 [myid:] - INFO [main:QuorumPeerConfig@320] - secureClientPort is not set
2016-07-12 16:10:05,435 [myid:1] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2016-07-12 16:10:05,451 [myid:1] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2016-07-12 16:10:05,451 [myid:1] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2016-07-12 16:10:05,560 [myid:1] - INFO [main:QuorumPeerMain@136] - Starting quorum peer
2016-07-12 16:10:05,685 [myid:1] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-07-12 16:10:05,685 [myid:1] - INFO [main:NIOServerCnxnFactory@686] - binding to port /0.0.0.0:2181
2016-07-12 16:10:05,747 [myid:1] - INFO [main:Slf4jLog@67] - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2016-07-12 16:10:05,794 [myid:1] - INFO [main:QuorumPeer@1280] - Local sessions disabled
2016-07-12 16:10:05,794 [myid:1] - INFO [main:QuorumPeer@1291] - Local session upgrading disabled
2016-07-12 16:10:05,794 [myid:1] - INFO [main:QuorumPeer@1258] - tickTime set to 2000
2016-07-12 16:10:05,794 [myid:1] - INFO [main:QuorumPeer@1302] - minSessionTimeout set to 4000
2016-07-12 16:10:05,794 [myid:1] - INFO [main:QuorumPeer@1313] - maxSessionTimeout set to 40000
2016-07-12 16:10:05,794 [myid:1] - INFO [main:QuorumPeer@1328] - initLimit set to 10
2016-07-12 16:10:05,825 [myid:1] - INFO [main:Slf4jLog@67] - jetty-6.1.26
2016-07-12 16:10:05,875 [myid:1] - INFO [main:Slf4jLog@67] - Started [email protected]:8080
2016-07-12 16:10:05,885 [myid:1] - INFO [main:JettyAdminServer@105] - Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands
2016-07-12 16:10:05,904 [myid:1] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@631] - My election bind port: QA-E8WIN11/10.11.8.236:3888
2016-07-12 16:10:05,912 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@986] - LOOKING
2016-07-12 16:10:05,912 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@989] - Attempting to start ReadOnlyZooKeeperServer
2016-07-12 16:10:05,923 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:zookeeper.version=3.5.1-alpha-1693007, built on 07/28/2015 07:19 GMT
2016-07-12 16:10:05,923 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:host.name=QA-E8WIN11
2016-07-12 16:10:05,923 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.version=1.8.0_91
2016-07-12 16:10:05,923 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.vendor=Oracle Corporation
2016-07-12 16:10:05,924 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.home=C:\jdk\jre
2016-07-12 16:10:05,924 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.class.path=C:\zookeeper-3.5.1-alpha\bin\..\build\classes;C:\zookeeper-3.5.1-alpha\bin\..\build\lib\*;C:\zookeeper-3.5.1-alpha\bin\..\zookeeper-3.5.1-alpha.jar;C:\zookeeper-3.5.1-alpha\bin\..\lib\commons-cli-1.2.jar;C:\zookeeper-3.5.1-alpha\bin\..\lib\jackson-core-asl-1.9.11.jar;C:\zookeeper-3.5.1-alpha\bin\..\lib\jackson-mapper-asl-1.9.11.jar;C:\zookeeper-3.5.1-alpha\bin\..\lib\javacc.jar;C:\zookeeper-3.5.1-alpha\bin\..\lib\jetty-6.1.26.jar;C:\zookeeper-3.5.1-alpha\bin\..\lib\jetty-util-6.1.26.jar;C:\zookeeper-3.5.1-alpha\bin\..\lib\jline-2.11.jar;C:\zookeeper-3.5.1-alpha\bin\..\lib\log4j-1.2.16.jar;C:\zookeeper-3.5.1-alpha\bin\..\lib\netty-3.7.0.Final.jar;C:\zookeeper-3.5.1-alpha\bin\..\lib\servlet-api-2.5-20081211.jar;C:\zookeeper-3.5.1-alpha\bin\..\lib\slf4j-api-1.7.5.jar;C:\zookeeper-3.5.1-alpha\bin\..\lib\slf4j-log4j12-1.7.5.jar;C:\zookeeper-3.5.1-alpha\bin\..\conf
2016-07-12 16:10:05,924 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.library.path=C:\jdk\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Notepad++;C:\unboundid-ldapsdk-2.3.7-se\tools;C:\Program Files\Java\jdk1.7.0_71\bin;C:\Program Files (x86)\GnuWin32\bin;.
2016-07-12 16:10:05,924 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\2\
2016-07-12 16:10:05,924 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.compiler=<NA>
2016-07-12 16:10:05,925 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.name=Windows Server 2008 R2
2016-07-12 16:10:05,925 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.arch=amd64
2016-07-12 16:10:05,925 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.version=6.1
2016-07-12 16:10:05,925 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:user.name=Administrator
2016-07-12 16:10:05,925 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:user.home=C:\Users\Administrator
2016-07-12 16:10:05,925 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:user.dir=C:\zookeeper-3.5.1-alpha\bin
2016-07-12 16:10:05,925 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.free=106MB
2016-07-12 16:10:05,926 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.max=1820MB
2016-07-12 16:10:05,926 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.total=123MB
2016-07-12 16:10:05,928 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@858] - minSessionTimeout set to 4000
2016-07-12 16:10:05,928 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@867] - maxSessionTimeout set to 40000
2016-07-12 16:10:05,928 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@156] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir C:\zookeeper-3.5.1-alpha\data\version-2 snapdir C:\zookeeper-3.5.1-alpha\data\version-2
2016-07-12 16:10:05,930 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@894] - New election. My id = 1, proposed zxid=0x0
2016-07-12 16:10:08,022 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.11.12.4:49175
2016-07-12 16:10:08,033 [myid:1] - WARN [NIOWorkerThread-1:NIOServerCnxn@369] - Exception causing close of session 0x0: ZooKeeperServer not running
2016-07-12 16:10:08,033 [myid:1] - INFO [NIOWorkerThread-1:NIOServerCnxn@606] - Closed socket connection for client /10.11.12.4:49175 (no session established for client)
2016-07-12 16:10:09,501 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.11.12.4:49178
2016-07-12 16:10:09,504 [myid:1] - WARN [NIOWorkerThread-2:NIOServerCnxn@369] - Exception causing close of session 0x0: ZooKeeperServer not running
2016-07-12 16:10:09,504 [myid:1] - INFO [NIOWorkerThread-2:NIOServerCnxn@606] - Closed socket connection for client /10.11.12.4:49178 (no session established for client)
2016-07-12 16:10:10,873 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.11.12.4:49182
2016-07-12 16:10:10,875 [myid:1] - WARN [NIOWorkerThread-3:NIOServerCnxn@369] - Exception causing close of session 0x0: ZooKeeperServer not running
2016-07-12 16:10:10,875 [myid:1] - INFO [NIOWorkerThread-3:NIOServerCnxn@606] - Closed socket connection for client /10.11.12.4:49182 (no session established for client)
2016-07-12 16:10:10,935 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@459] - Cannot open channel to 2 at election address QA-E8WIN12/10.11.8.232:3888
java.net.SocketTimeoutException: connect timed out
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:444)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:485)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:421)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-12 16:10:10,955 [myid:1] - INFO [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started
2016-07-12 16:10:12,725 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.11.12.4:49191
2016-07-12 16:10:12,729 [myid:1] - INFO [NIOWorkerThread-4:ZooKeeperServer@945] - Refusing session request for client /10.11.12.4:49191 as it has seen zxid 0x3 our last zxid is 0x0 client must try another server
2016-07-12 16:10:12,729 [myid:1] - INFO [NIOWorkerThread-4:NIOServerCnxn@606] - Closed socket connection for client /10.11.12.4:49191 (no session established for client)
2016-07-12 16:10:14,689 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.11.12.4:49194
2016-07-12 16:10:14,692 [myid:1] - INFO [NIOWorkerThread-5:ZooKeeperServer@945] - Refusing session request for client /10.11.12.4:49194 as it has seen zxid 0x3 our last zxid is 0x0 client must try another server
2016-07-12 16:10:14,692 [myid:1] - INFO [NIOWorkerThread-5:NIOServerCnxn@606] - Closed socket connection for client /10.11.12.4:49194 (no session established for client)
2016-07-12 16:10:15,945 [myid:1] - WARN [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumCnxManager@459] - Cannot open channel to 2 at election address QA-E8WIN12/10.11.8.232:3888
java.net.SocketTimeoutException: connect timed out
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:444)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:485)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:513)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:919)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1023)
2016-07-12 16:10:15,946 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@928] - Notification time out: 400
2016-07-12 16:10:16,508 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.11.12.4:49198
2016-07-12 16:10:16,508 [myid:1] - INFO [NIOWorkerThread-6:ZooKeeperServer@945] - Refusing session request for client /10.11.12.4:49198 as it has seen zxid 0x3 our last zxid is 0x0 client must try another server
2016-07-12 16:10:16,508 [myid:1] - INFO [NIOWorkerThread-6:NIOServerCnxn@606] - Closed socket connection for client /10.11.12.4:49198 (no session established for client)
2016-07-12 16:10:17,645 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.11.12.4:49199
2016-07-12 16:10:17,647 [myid:1] - INFO [NIOWorkerThread-7:ZooKeeperServer@945] - Refusing session request for client /10.11.12.4:49199 as it has seen zxid 0x3 our last zxid is 0x0 client must try another server
2016-07-12 16:10:17,648 [myid:1] - INFO [NIOWorkerThread-7:NIOServerCnxn@606] - Closed socket connection for client /10.11.12.4:49199 (no session established for client)
2016-07-12 16:10:19,048 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.11.12.4:49200
2016-07-12 16:10:19,051 [myid:1] - INFO [NIOWorkerThread-8:ZooKeeperServer@945] - Refusing session request for client /10.11.12.4:49200 as it has seen zxid 0x3 our last zxid is 0x0 client must try another server
2016-07-12 16:10:19,051 [myid:1] - INFO [NIOWorkerThread-8:NIOServerCnxn@606] - Closed socket connection for client /10.11.12.4:49200 (no session established for client)
2016-07-12 16:10:20,728 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.11.12.4:49213
2016-07-12 16:10:20,730 [myid:1] - INFO [NIOWorkerThread-1:ZooKeeperServer@945] - Refusing session request for client /10.11.12.4:49213 as it has seen zxid 0x3 our last zxid is 0x0 client must try another server
2016-07-12 16:10:20,730 [myid:1] - INFO [NIOWorkerThread-1:NIOServerCnxn@606] - Closed socket connection for client /10.11.12.4:49213 (no session established for client)
2016-07-12 16:10:20,946 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@459] - Cannot open channel to 3 at election address QA-E8WIN13/10.11.8.230:3888
java.net.SocketTimeoutException: connect timed out
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:444)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:485)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:421)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-12 16:10:20,947 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-12 16:10:22,084 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.11.12.4:49217
2016-07-12 16:10:22,086 [myid:1] - INFO [NIOWorkerThread-2:ZooKeeperServer@945] - Refusing session request for client /10.11.12.4:49217 as it has seen zxid 0x3 our last zxid is 0x0 client must try another server
2016-07-12 16:10:22,086 [myid:1] - INFO [NIOWorkerThread-2:NIOServerCnxn@606] - Closed socket connection for client /10.11.12.4:49217 (no session established for client)
2016-07-12 16:10:24,178 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.11.12.4:49227
2016-07-12 16:10:24,178 [myid:1] - INFO [NIOWorkerThread-3:ZooKeeperServer@945] - Refusing session request for client /10.11.12.4:49227 as it has seen zxid 0x3 our last zxid is 0x0 client must try another server
2016-07-12 16:10:24,178 [myid:1] - INFO [NIOWorkerThread-3:NIOServerCnxn@606] - Closed socket connection for client /10.11.12.4:49227 (no session established for client)
2016-07-12 16:10:25,948 [myid:1] - WARN [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumCnxManager@459] - Cannot open channel to 2 at election address QA-E8WIN12/10.11.8.232:3888
java.net.SocketTimeoutException: connect timed out
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:444)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:485)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:513)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:919)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1023)
2016-07-12 16:10:26,174 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.11.12.4:49233
2016-07-12 16:10:26,175 [myid:1] - INFO [NIOWorkerThread-4:ZooKeeperServer@945] - Refusing session request for client /10.11.12.4:49233 as it has seen zxid 0x3 our last zxid is 0x0 client must try another server
2016-07-12 16:10:26,175 [myid:1] - INFO [NIOWorkerThread-4:NIOServerCnxn@606] - Closed socket connection for client /10.11.12.4:49233 (no session established for client)
2016-07-12 16:10:27,582 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.11.12.4:49234
2016-07-12 16:10:27,584 [myid:1] - INFO [NIOWorkerThread-5:ZooKeeperServer@945] - Refusing session request for client /10.11.12.4:49234 as it has seen zxid 0x3 our last zxid is 0x0 client must try another server
2016-07-12 16:10:27,584 [myid:1] - INFO [NIOWorkerThread-5:NIOServerCnxn@606] - Closed socket connection for client /10.11.12.4:49234 (no session established for client)
2016-07-12 16:10:29,498 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.11.12.4:49236
2016-07-12 16:10:29,500 [myid:1] - INFO [NIOWorkerThread-6:ZooKeeperServer@945] - Refusing session request for client /10.11.12.4:49236 as it has seen zxid 0x3 our last zxid is 0x0 client must try another server
2016-07-12 16:10:29,500 [myid:1] - INFO [NIOWorkerThread-6:NIOServerCnxn@606] - Closed socket connection for client /10.11.12.4:49236 (no session established for client)
2016-07-12 16:10:30,754 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.11.12.4:49237
2016-07-12 16:10:30,756 [myid:1] - INFO [NIOWorkerThread-7:ZooKeeperServer@945] - Refusing session request for client /10.11.12.4:49237 as it has seen zxid 0x3 our last zxid is 0x0 client must try another server
2016-07-12 16:10:30,756 [myid:1] - INFO [NIOWorkerThread-7:NIOServerCnxn@606] - Closed socket connection for client /10.11.12.4:49237 (no session established for client)
2016-07-12 16:10:30,950 [myid:1] - WARN [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumCnxManager@459] - Cannot open channel to 3 at election address QA-E8WIN13/10.11.8.230:3888
java.net.SocketTimeoutException: connect timed out
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:444)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:485)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:513)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:919)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1023)
2016-07-12 16:10:30,951 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@928] - Notification time out: 800
2016-07-12 16:10:32,128 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.11.12.4:49244
2016-07-12 16:10:32,128 [myid:1] - INFO [NIOWorkerThread-8:ZooKeeperServer@945] - Refusing session request for client /10.11.12.4:49244 as it has seen zxid 0x3 our last zxid is 0x0 client must try another server
2016-07-12 16:10:32,129 [myid:1] - INFO [NIOWorkerThread-8:NIOServerCnxn@606] - Closed socket connection for client /10.11.12.4:49244 (no session established for client)
