Yup, those logs are normal. Are all your 3 nodes healthy? Not sure what could be causing it.
On 2/25/15, 7:28 AM, "Jon Maron" <[email protected]> wrote: > >> On Feb 25, 2015, at 10:16 AM, Gour Saha <[email protected]> wrote: >> >> Can you check the zk logs at /var/log/zookeeper/zookeeper.out and see if >> you find something? > >I see a bunch of these but I’m assuming these are normal for a >disconnected client connection: > >2015-02-23 19:40:21,320 - INFO >[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed >socket connection for client /192.168.64.105:34018 (no session >established for client) >2015-02-23 19:41:21,311 - INFO >[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - >Accepted socket connection from /192.168.64.105:34031 >2015-02-23 19:41:21,319 - WARN >[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught >end of stream exception >EndOfStreamException: Unable to read additional data from client >sessionid 0x0, likely client has closed socket > at >org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) > at >org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory. >java:208) > at java.lang.Thread.run(Thread.java:745) >2015-02-23 19:41:21,319 - INFO >[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed >socket connection for client /192.168.64.105:34031 (no session >established for client) >2015-02-23 19:41:52,896 - INFO >[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - >Accepted socket connection from /192.168.64.104:46949 >2015-02-23 19:41:52,896 - INFO >[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client >attempting to establish new session at /192.168.64.104:46949 >2015-02-23 19:41:52,900 - INFO [CommitProcessor:4:ZooKeeperServer@617] - >Established session 0x44bb7e82d730002 with negotiated timeout 10000 for >client /192.168.64.104:46949 >2015-02-23 19:41:52,916 - INFO >[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed >socket connection for client /192.168.64.104:46949 which had sessionid >0x44bb7e82d730002 >2015-02-23 19:42:21,313 - INFO >[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - >Accepted socket connection from /192.168.64.105:34054 >2015-02-23 19:42:21,314 - WARN >[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught >end of stream exception >EndOfStreamException: Unable to read additional data from client >sessionid 0x0, likely client has closed socket > at >org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) > at >org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory. >java:208) > at java.lang.Thread.run(Thread.java:745) >2015-02-23 19:42:21,314 - INFO >[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed >socket connection for client /192.168.64.105:34054 (no session >established for client) >2015-02-23 19:42:38,263 - INFO >[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - >Accepted socket connection from /192.168.64.1:52286 >2015-02-23 19:42:38,265 - INFO >[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client >attempting to establish new session at /192.168.64.1:52286 >2015-02-23 19:42:38,269 - INFO [CommitProcessor:4:ZooKeeperServer@617] - >Established session 0x44bb7e82d730003 with negotiated timeout 40000 for >client /192.168.64.1:52286 >2015-02-23 19:42:39,316 - INFO >[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed >socket connection for client /192.168.64.1:52286 which had sessionid >0x44bb7e82d730003 >2015-02-23 19:43:14,665 - INFO >[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - >Accepted socket connection from /192.168.64.105:34129 >2015-02-23 19:43:14,667 - INFO >[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client >attempting to establish new session at /192.168.64.105:34129 >2015-02-23 19:43:14,672 - INFO [CommitProcessor:4:ZooKeeperServer@617] - >Established session 0x44bb7e82d730004 with negotiated timeout 10000 for >client /192.168.64.105:34129 >2015-02-23 19:43:14,681 - WARN >[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught >end of stream exception >EndOfStreamException: Unable to read additional data from client >sessionid 0x44bb7e82d730004, likely client has closed socket > at >org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) > at >org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory. >java:208) > at java.lang.Thread.run(Thread.java:745) > > >> >> Also, see if you can use the zkCli.sh client to query in a loop for few >> minutes (with few secs of interval between queries) and see if you get >> similar intermittent connection issues? > >Tried “watch -n 5 ./zkCli.sh ls /services”. Didn’t see any issue on the >client side, though interestingly at times the connection seemed to be >using the IPv6 address? > >> >> -Gour >> >> On 2/25/15, 6:53 AM, "Jon Maron" <[email protected]> wrote: >> >>> I¹ve noticed that I¹m having intermittent issues accessing the >>>zookeeper >>> quorum during ³destroy² attempts: >>> >>> 2015-02-25 09:48:02,345 [main] WARN client.SliderClient >>> (SliderClient.java:getZkClient(523)) - Unable to connect to zookeeper >>> quorum >>> >>>c6402.ambari.apache.org:2181,c6404.ambari.apache.org:2181,c6403.ambari.a >>>pa >>> che.org:2181,c6405.ambari.apache.org:2181 >>> java.net.ConnectException: Unable to connect to ZK quorum >>> at >>> >>>org.apache.slider.core.zk.BlockingZKWatcher.waitForZKConnection(Blocking >>>ZK >>> Watcher.java:63) >>> at >>> >>>org.apache.slider.client.SliderClient.getZkClient(SliderClient.java:518) >>> at >>> >>>org.apache.slider.client.SliderClient.deleteZookeeperNode(SliderClient.j >>>av >>> a:458) >>> at >>> >>>org.apache.slider.client.SliderClient.actionDestroy(SliderClient.java:55 >>>0) >>> at org.apache.slider.client.SliderClient.exec(SliderClient.java:383) >>> at >>> org.apache.slider.client.SliderClient.runService(SliderClient.java:348) >>> at >>> >>>org.apache.slider.core.main.ServiceLauncher.launchService(ServiceLaunche >>>r. >>> java:188) >>> at >>> >>>org.apache.slider.core.main.ServiceLauncher.launchServiceRobustly(Servic >>>eL >>> auncher.java:475) >>> at >>> >>>org.apache.slider.core.main.ServiceLauncher.launchServiceAndExit(Service >>>La >>> uncher.java:403) >>> at >>> >>>org.apache.slider.core.main.ServiceLauncher.serviceMain(ServiceLauncher. >>>ja >>> va:630) >>> at org.apache.slider.Slider.main(Slider.java:49) >>> 2015-02-25 09:48:02,656 [main] DEBUG client.SliderClient >>> (SliderClient.java:deleteZookeeperNode(474)) - Unable to recursively >>> delete zk node /services/slider/users/jmaron/hbase-test >>> 2015-02-25 09:48:02,656 [main] DEBUG client.SliderClient >>> (SliderClient.java:deleteZookeeperNode(475)) - Reason: >>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> KeeperErrorCode = ConnectionLoss for >>> /services/slider/users/jmaron/hbase-test >>> at >>>org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >>> at >>>org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) >>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1073) >>> at >>>org.apache.slider.core.zk.ZKIntegration.stat(ZKIntegration.java:164) >>> at >>>org.apache.slider.core.zk.ZKIntegration.exists(ZKIntegration.java:160) >>> at >>> >>>org.apache.slider.client.SliderClient.deleteZookeeperNode(SliderClient.j >>>av >>> a:460) >>> at >>> >>>org.apache.slider.client.SliderClient.actionDestroy(SliderClient.java:55 >>>0) >>> at org.apache.slider.client.SliderClient.exec(SliderClient.java:383) >>> at >>> org.apache.slider.client.SliderClient.runService(SliderClient.java:348) >>> at >>> >>>org.apache.slider.core.main.ServiceLauncher.launchService(ServiceLaunche >>>r. >>> java:188) >>> at >>> >>>org.apache.slider.core.main.ServiceLauncher.launchServiceRobustly(Servic >>>eL >>> auncher.java:475) >>> at >>> >>>org.apache.slider.core.main.ServiceLauncher.launchServiceAndExit(Service >>>La >>> uncher.java:403) >>> at >>> >>>org.apache.slider.core.main.ServiceLauncher.serviceMain(ServiceLauncher. >>>ja >>> va:630) >>> at org.apache.slider.Slider.main(Slider.java:49) >>> >>> Any ideas on why that may occur? My cluster is running on a set of VMs >>> on my development box. These failed ZK interactions will subsequently >>> yield issues in trying to recreate the given application (in this case >>> HBase) >>> >>> ‹ Jon >> >
