Hello Yash,
Please note that Devang and Bobby described two different root causes from the same problem you are facing. FYI, the problem Bobby described in zookeeper is discussed in https://issues.apache.org/jira/browse/ZOOKEEPER-885<https://issues.apache.org/jira/browse/ZOOKEEPER-885#comment-13749751> ? Itai ________________________________ From: Devang Shah <[email protected]> Sent: Tuesday, November 18, 2014 1:16 AM To: [email protected] Subject: Re: Why would Zookeeper Timeouts lead to worker restarts? We had faced similar issues and identified the cause to be heavy processing of the bolts. GC was hogging the CPU cycles and workers didn't get time to communicate with Zookeeper which eventually ended in worker restarts. Try using conc-mark-sweep algo. Thanks and Regards, Devang On 15 Nov 2014 03:47, "Bobby Evans" <[email protected]<mailto:[email protected]>> wrote: Workers communicate to nimbus though ZooKeeper. If ZK is under a lot of load and is not able to handle all of the write requests, nimbus can think that the workers are not heart-beating, even when they are. If this happens it will reschedule them, possibly on a different node/location. If that happens the Supervisors will dutifully obey and change things around to match the new scheduling. If you rebalance your topology, nimbus lies to the scheduler telling it that none of the workers for your topology are running, and lets it pick all new places for them to run. Then the supervisors shoot/launch workers to make things match the new scheduling. - Bobby On Friday, November 14, 2014 12:11 PM, Yashwant Ganti <[email protected]<mailto:[email protected]>> wrote: Hello All, We had a problem with Zookeeper infrastructure recently and the components in our topology would face timeouts when trying to establish a connection. After a prolonged period of attempting to connect and throwing exceptions, I noticed that some of the workers were restarted. One of the other devs mentioned that this was because the topology had to be 'rebalanced'. I did not quite understand this and was wondering if someone could shed light on why storm decided to restart the workers. Thanks, Yash
