Hi Vladi,
Maybe It's a longshot, but I saw someone referencing the following zookeeper bug in the elasticsearch-zookeeper mailing list. https://issues.apache.org/jira/browse/ZOOKEEPER-885#comment-13749751 It is worth to dive into the JIRA issues logs and compare to your ZK logs. Itai ________________________________ From: Vladi Feigin <[email protected]> Sent: Wednesday, November 12, 2014 10:53 PM To: [email protected] Subject: Worker moved to other node. The possible reason ZK capacity? Hi All, Nimbus for some reason moved the worker to other node. In the supervisor log we've found these warnings below. Does it mean we run out of ZK capacity? Or some other reason? Thank you in advance, Vladi 2014-11-12 11:18:04 -3389175 [ConnectionStateManager-0] WARN com.netflix.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered. 2014-11-12 11:18:04 -3389172 [main-EventThread] WARN backtype.storm.cluster - Received event :disconnected::none: with disconnected Zookeeper. 2014-11-12 11:18:04 -3388713 [main-SendThread(xxx:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server xxx:2181 2014-11-12 11:18:04 -3388712 [main-SendThread(xxx:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to xxx:2181, initiating session 2014-11-12 11:18:04 -3388708 [main-EventThread] WARN com.netflix.curator.ConnectionState - Session expired event received 2014-11-12 11:18:04 -3388707 [main-SendThread(xxx:2181)] INFO org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x349390bb7cb85f7 has expired, closing socket connection 2014-11-12 11:18:04 -3388707 [main-EventThread] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=xxx:2181,xxxw:2181,xxxz:2181/storm sessionTimeout=20000 watcher=com.netflix.curator.ConnectionState@d13998e 2014-11-12 11:18:04 -3388706 [main-EventThread] INFO com.netflix.curator.framework.state.ConnectionStateManager - State change: LOST 2014-11-12 11:18:04 -3388706 [main-SendThread()] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server xxx:2181 2014-11-12 11:18:04 -3388706 [ConnectionStateManager-0] WARN com.netflix.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered. 2014-11-12 11:18:04 -3388705 [main-EventThread] WARN backtype.storm.cluster - Received event :expired::none: with disconnected Zookeeper. 2014-11-12 11:18:04 -3388705 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down 2014-11-12 11:18:04 -3388705 [main-SendThread(xxx:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to xxx:2181, initiating session 2014-11-12 11:18:04 -3388702 [main-SendThread(xxx:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server xxx:2181, sessionid = 0x14939127de28166, negotiated timeout = 20000 2014-11-12 11:18:04 -3388701 [main-EventThread] INFO com.netflix.curator.framework.state.ConnectionStateManager - State change: RECONNECTED 2014-11-12 11:18:04 -3388701 [ConnectionStateManager-0] WARN com.netflix.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
