Hello,
I am looking into this issue:
https://issues.apache.org/jira/browse/CURATOR-125
I have 3 ZK servers, only 1 is started.
I get this log and it might be causing my issue down the road:
java.io.IOException: Xid out of order. Got Xid 14 with err -119 expected
Xid 13 for a packet with details: clientPath:/ serverPath:/ finished:false
header:: 13,3 replyHeader:: 0,0,-4 request:: '/,T response::
at
org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:795)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
Client and servers are running the official 3.4.5.
I am attaching the log file.
The code I am running is in the bug:
https://issues.apache.org/jira/browse/CURATOR-125 ( Test.java ).
I can open a bug if necessary.
Thanks,
Ben
0 [main] INFO org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2 [main] DEBUG org.apache.curator.CuratorZookeeperClient - Starting
2 [main] DEBUG org.apache.curator.ConnectionState - Starting
2 [main] DEBUG org.apache.curator.ConnectionState - reset
12 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
12 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=w-rli09-ben
12 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_05
12 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
12 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/local/apps/jdk1.8.0_05/jre
13 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=(...)
13 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
13 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
13 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
13 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
13 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
13 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.8.0-42-generic
13 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=benji
13 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/benji
13 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=(...)
14 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=w-rli09-ben:2181,ubuntu:2181,s-rli10-devtest:2181 sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@150c158
33 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server w-rli09-ben/127.0.1.1:2181. Will not attempt to authenticate using SASL (unknown error)
37 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to w-rli09-ben/127.0.1.1:2181, initiating session
43 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server w-rli09-ben/127.0.1.1:2181, sessionid = 0x147658c3b6c0000, negotiated timeout = 30000 (READ-ONLY mode)
50 [main-EventThread] INFO org.apache.curator.framework.state.ConnectionStateManager - State change: READ_ONLY
CURATOR STATE: READ_ONLY
130 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Checking server /10.11.12.210:2181 for being r/w. Timeout 200
335 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Checking server /10.11.0.80:2181 for being r/w. Timeout 400
736 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Checking server w-rli09-ben/127.0.1.1:2181 for being r/w. Timeout 800
1537 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Checking server s-rli10-devtest/10.11.12.210:2181 for being r/w. Timeout 1600
3139 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Checking server ubuntu/10.11.0.80:2181 for being r/w. Timeout 3200
6344 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Checking server w-rli09-ben/127.0.1.1:2181 for being r/w. Timeout 6400
7696 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Unable to read additional data from server sessionid 0x147658c3b6c0000, likely server has closed socket, closing socket connection and attempting reconnect
7797 [main-EventThread] INFO org.apache.curator.framework.state.ConnectionStateManager - State change: SUSPENDED
CURATOR STATE: SUSPENDED
8374 [main-SendThread(s-rli10-devtest:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server s-rli10-devtest/10.11.12.210:2181. Will not attempt to authenticate using SASL (unknown error)
8376 [main-SendThread(s-rli10-devtest:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x147658c3b6c0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
9414 [main-SendThread(ubuntu:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server ubuntu/10.11.0.80:2181. Will not attempt to authenticate using SASL (unknown error)
9414 [main-SendThread(ubuntu:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to ubuntu/10.11.0.80:2181, initiating session
9438 [main-SendThread(ubuntu:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server ubuntu/10.11.0.80:2181, sessionid = 0x247658809da0000, negotiated timeout = 30000
9439 [main-EventThread] INFO org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
CURATOR STATE: CONNECTED
10044 [main-SendThread(ubuntu:2181)] INFO org.apache.zookeeper.ClientCnxn - Unable to read additional data from server sessionid 0x247658809da0000, likely server has closed socket, closing socket connection and attempting reconnect
10145 [main-EventThread] INFO org.apache.curator.framework.state.ConnectionStateManager - State change: SUSPENDED
CURATOR STATE: SUSPENDED
10268 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server w-rli09-ben/127.0.1.1:2181. Will not attempt to authenticate using SASL (unknown error)
10268 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to w-rli09-ben/127.0.1.1:2181, initiating session
10269 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Unable to read additional data from server sessionid 0x247658809da0000, likely server has closed socket, closing socket connection and attempting reconnect
10938 [main-SendThread(s-rli10-devtest:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server s-rli10-devtest/10.11.12.210:2181. Will not attempt to authenticate using SASL (unknown error)
10939 [main-SendThread(s-rli10-devtest:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x247658809da0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
12792 [main-SendThread(ubuntu:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server ubuntu/10.11.0.80:2181. Will not attempt to authenticate using SASL (unknown error)
12793 [main-SendThread(ubuntu:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x247658809da0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
13197 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server w-rli09-ben/127.0.1.1:2181. Will not attempt to authenticate using SASL (unknown error)
13197 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to w-rli09-ben/127.0.1.1:2181, initiating session
13198 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server w-rli09-ben/127.0.1.1:2181, sessionid = 0x247658809da0000, negotiated timeout = 30000 (READ-ONLY mode)
13198 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Checking server s-rli10-devtest/10.11.12.210:2181 for being r/w. Timeout 12800
13199 [main-EventThread] INFO org.apache.curator.framework.state.ConnectionStateManager - State change: READ_ONLY
CURATOR STATE: READ_ONLY
20375 [main-SendThread(w-rli09-ben:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x247658809da0000 for server w-rli09-ben/127.0.1.1:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Xid out of order. Got Xid 14 with err -119 expected Xid 13 for a packet with details: clientPath:/ serverPath:/ finished:false header:: 13,3 replyHeader:: 0,0,-4 request:: '/,T response::
at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:795)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
20476 [main-EventThread] INFO org.apache.curator.framework.state.ConnectionStateManager - State change: SUSPENDED
CURATOR STATE: SUSPENDED
21016 [main-SendThread(ubuntu:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server ubuntu/10.11.0.80:2181. Will not attempt to authenticate using SASL (unknown error)
21017 [main-SendThread(ubuntu:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x247658809da0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
22425 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server w-rli09-ben/127.0.1.1:2181. Will not attempt to authenticate using SASL (unknown error)
22426 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to w-rli09-ben/127.0.1.1:2181, initiating session
22427 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server w-rli09-ben/127.0.1.1:2181, sessionid = 0x247658809da0000, negotiated timeout = 30000 (READ-ONLY mode)
22427 [main-EventThread] INFO org.apache.curator.framework.state.ConnectionStateManager - State change: READ_ONLY
CURATOR STATE: READ_ONLY
26001 [main-SendThread(w-rli09-ben:2181)] INFO org.apache.zookeeper.ClientCnxn - Checking server s-rli10-devtest/10.11.12.210:2181 for being r/w. Timeout 25600
