> On Feb 25, 2015, at 10:16 AM, Gour Saha <[email protected]> wrote:
>
> Can you check the zk logs at /var/log/zookeeper/zookeeper.out and see if
> you find something?
I see a bunch of these but I’m assuming these are normal for a disconnected
client connection:
2015-02-23 19:40:21,320 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket
connection for client /192.168.64.105:34018 (no session established for client)
2015-02-23 19:41:21,311 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted
socket connection from /192.168.64.105:34031
2015-02-23 19:41:21,319 - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of
stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x0,
likely client has closed socket
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-02-23 19:41:21,319 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket
connection for client /192.168.64.105:34031 (no session established for client)
2015-02-23 19:41:52,896 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted
socket connection from /192.168.64.104:46949
2015-02-23 19:41:52,896 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client
attempting to establish new session at /192.168.64.104:46949
2015-02-23 19:41:52,900 - INFO [CommitProcessor:4:ZooKeeperServer@617] -
Established session 0x44bb7e82d730002 with negotiated timeout 10000 for client
/192.168.64.104:46949
2015-02-23 19:41:52,916 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket
connection for client /192.168.64.104:46949 which had sessionid
0x44bb7e82d730002
2015-02-23 19:42:21,313 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted
socket connection from /192.168.64.105:34054
2015-02-23 19:42:21,314 - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of
stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x0,
likely client has closed socket
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-02-23 19:42:21,314 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket
connection for client /192.168.64.105:34054 (no session established for client)
2015-02-23 19:42:38,263 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted
socket connection from /192.168.64.1:52286
2015-02-23 19:42:38,265 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client
attempting to establish new session at /192.168.64.1:52286
2015-02-23 19:42:38,269 - INFO [CommitProcessor:4:ZooKeeperServer@617] -
Established session 0x44bb7e82d730003 with negotiated timeout 40000 for client
/192.168.64.1:52286
2015-02-23 19:42:39,316 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket
connection for client /192.168.64.1:52286 which had sessionid 0x44bb7e82d730003
2015-02-23 19:43:14,665 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted
socket connection from /192.168.64.105:34129
2015-02-23 19:43:14,667 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client
attempting to establish new session at /192.168.64.105:34129
2015-02-23 19:43:14,672 - INFO [CommitProcessor:4:ZooKeeperServer@617] -
Established session 0x44bb7e82d730004 with negotiated timeout 10000 for client
/192.168.64.105:34129
2015-02-23 19:43:14,681 - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of
stream exception
EndOfStreamException: Unable to read additional data from client sessionid
0x44bb7e82d730004, likely client has closed socket
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
>
> Also, see if you can use the zkCli.sh client to query in a loop for few
> minutes (with few secs of interval between queries) and see if you get
> similar intermittent connection issues?
Tried “watch -n 5 ./zkCli.sh ls /services”. Didn’t see any issue on the client
side, though interestingly at times the connection seemed to be using the IPv6
address?
>
> -Gour
>
> On 2/25/15, 6:53 AM, "Jon Maron" <[email protected]> wrote:
>
>> I¹ve noticed that I¹m having intermittent issues accessing the zookeeper
>> quorum during ³destroy² attempts:
>>
>> 2015-02-25 09:48:02,345 [main] WARN client.SliderClient
>> (SliderClient.java:getZkClient(523)) - Unable to connect to zookeeper
>> quorum
>> c6402.ambari.apache.org:2181,c6404.ambari.apache.org:2181,c6403.ambari.apa
>> che.org:2181,c6405.ambari.apache.org:2181
>> java.net.ConnectException: Unable to connect to ZK quorum
>> at
>> org.apache.slider.core.zk.BlockingZKWatcher.waitForZKConnection(BlockingZK
>> Watcher.java:63)
>> at
>> org.apache.slider.client.SliderClient.getZkClient(SliderClient.java:518)
>> at
>> org.apache.slider.client.SliderClient.deleteZookeeperNode(SliderClient.jav
>> a:458)
>> at
>> org.apache.slider.client.SliderClient.actionDestroy(SliderClient.java:550)
>> at org.apache.slider.client.SliderClient.exec(SliderClient.java:383)
>> at
>> org.apache.slider.client.SliderClient.runService(SliderClient.java:348)
>> at
>> org.apache.slider.core.main.ServiceLauncher.launchService(ServiceLauncher.
>> java:188)
>> at
>> org.apache.slider.core.main.ServiceLauncher.launchServiceRobustly(ServiceL
>> auncher.java:475)
>> at
>> org.apache.slider.core.main.ServiceLauncher.launchServiceAndExit(ServiceLa
>> uncher.java:403)
>> at
>> org.apache.slider.core.main.ServiceLauncher.serviceMain(ServiceLauncher.ja
>> va:630)
>> at org.apache.slider.Slider.main(Slider.java:49)
>> 2015-02-25 09:48:02,656 [main] DEBUG client.SliderClient
>> (SliderClient.java:deleteZookeeperNode(474)) - Unable to recursively
>> delete zk node /services/slider/users/jmaron/hbase-test
>> 2015-02-25 09:48:02,656 [main] DEBUG client.SliderClient
>> (SliderClient.java:deleteZookeeperNode(475)) - Reason:
>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss for
>> /services/slider/users/jmaron/hbase-test
>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1073)
>> at org.apache.slider.core.zk.ZKIntegration.stat(ZKIntegration.java:164)
>> at
>> org.apache.slider.core.zk.ZKIntegration.exists(ZKIntegration.java:160)
>> at
>> org.apache.slider.client.SliderClient.deleteZookeeperNode(SliderClient.jav
>> a:460)
>> at
>> org.apache.slider.client.SliderClient.actionDestroy(SliderClient.java:550)
>> at org.apache.slider.client.SliderClient.exec(SliderClient.java:383)
>> at
>> org.apache.slider.client.SliderClient.runService(SliderClient.java:348)
>> at
>> org.apache.slider.core.main.ServiceLauncher.launchService(ServiceLauncher.
>> java:188)
>> at
>> org.apache.slider.core.main.ServiceLauncher.launchServiceRobustly(ServiceL
>> auncher.java:475)
>> at
>> org.apache.slider.core.main.ServiceLauncher.launchServiceAndExit(ServiceLa
>> uncher.java:403)
>> at
>> org.apache.slider.core.main.ServiceLauncher.serviceMain(ServiceLauncher.ja
>> va:630)
>> at org.apache.slider.Slider.main(Slider.java:49)
>>
>> Any ideas on why that may occur? My cluster is running on a set of VMs
>> on my development box. These failed ZK interactions will subsequently
>> yield issues in trying to recreate the given application (in this case
>> HBase)
>>
>> ‹ Jon
>
