We had faced similar issues and identified the cause to be heavy processing of the bolts. GC was hogging the CPU cycles and workers didn't get time to communicate with Zookeeper which eventually ended in worker restarts. Try using conc-mark-sweep algo.
Thanks and Regards, Devang On 15 Nov 2014 03:47, "Bobby Evans" <[email protected]> wrote: > Workers communicate to nimbus though ZooKeeper. If ZK is under a lot of > load and is not able to handle all of the write requests, nimbus can think > that the workers are not heart-beating, even when they are. If this > happens it will reschedule them, possibly on a different node/location. If > that happens the Supervisors will dutifully obey and change things around > to match the new scheduling. If you rebalance your topology, nimbus lies > to the scheduler telling it that none of the workers for your topology are > running, and lets it pick all new places for them to run. Then the > supervisors shoot/launch workers to make things match the new scheduling. > > - Bobby > > > On Friday, November 14, 2014 12:11 PM, Yashwant Ganti < > [email protected]> wrote: > > > Hello All, > > We had a problem with Zookeeper infrastructure recently and the components > in our topology would face timeouts when trying to establish a connection. > After a prolonged period of attempting to connect and throwing exceptions, > I noticed that some of the workers were restarted. One of the other devs > mentioned that this was because the topology had to be 'rebalanced'. I did > not quite understand this and was wondering if someone could shed light on > why storm decided to restart the workers. > > Thanks, > Yash > > >
