So you are saying that your organization has been doing security scans for a good long while now. Those security scans are called "qualys scan".
After upgrading your storm cluster from 0.8.3 to 0.9.4, you've seen instances of workers dying and nimbus issuing rebalances. You believe that the "qualys scan" is somehow involved. Your reaction then, has been to make changes in storm configuration. Those changes were not successful in stopping the problem. Questions: * what makes you think that the "qualys scan" is causing the problems? * have you turned off the scans and then seen the worker deaths / problems go away? * have you asked your contacts at the company, qualys, what they think of this problem? * how frequent are these qualys scans performed? How long do they "touch" or "focus" on your cluster / one of its nodes? * do you have an idea as to the mechanism of the interference? Thank you for your time! +++++++++++++++++++++ Jeff Maass <[email protected]> linkedin.com/in/jeffmaass stackoverflow.com/users/373418/maassql +++++++++++++++++++++ On Fri, May 22, 2015 at 1:44 PM, Amit Singh <[email protected]> wrote: > We have recently upgraded to storm 0.9.4 from storm 0.8.3 > > We have seen a peculiar behavior when we run a security scan in our > environment. [This is a qualys scan] > > 1. The workers on a particular host is killed by the nimbus or the > supervisor. > a. supervisor kills it because of a worker heartbeat timeout. > b. Nimbus sees the tasks as not alive and issues a rebalance. > > 2. I have tried playing around the following parameters > a. storm.messaging.netty.server_worker_threads:24 [so the tcp scan are > stiffled on the threads available for processing] > b. supervisor.worker.timeout.secs: 120 [because i saw some worker > heartbeat timeout] > c. nimbus.task.timeout.secs: 30 [nimbus seemed to have some tasks > marked as dead.] > > 3. This is reproducible almost all the time. > > > Is there anybody else out there who has faced this issue and if so whats > the workaround ? > > Let me know if any other details are required. > > Thanks, > Amit. > >
