Thanks Rui for the answer. The link mentioned that we can use Storm Web UI to re-balance the topology, but can I increase the number of parallelism of particular bolt only? I guess not.
Btw, we are using streamparse, https://streamparse.readthedocs.io/en/stable/topologies.html#parallelism-and-workers From: [email protected] At: 01/15/20 09:58:08To: Zainal Arifin (BLOOMBERG/ 731 LEX ) , [email protected] Subject: Re: question on deactivate and kill Deactivating a topology means the spouts stop consuming, but everything else continues to work as expected, meaning your bolts will process all in flight tuples. You can change the number of workers and executors in runtime. In order to scale up the latter, in you need to have previously submitted the topology with a larger number of tasks than executors. https://storm.apache.org/releases/1.2.3/Understanding-the-parallelism-of-a-Storm-topology.html On Wed, 15 Jan 2020 at 16:41, Zainal Arifin (BLOOMBERG/ 731 LEX) <[email protected]> wrote: Hi, Right now we have a bolt task that is in back log, i.e. ~10 hours behind processing the data. There was a bug and we have the fix which will speed up drastically. I think if I kill the topology (to deploy the fix), I will definitely lose those events been processed by the bolt, is that correct? How about if I just de-activate the topology, will the bolt still processing the events, and only spouts stop receiving the events? I wish there is a way to increase the parallelism for the bolt at run time, because even we don't deploy the fix, but if there are multiple instances of that bolt running, it'll still speed up. Is there a way to do so (we run v1.1.1)? Or perhaps there is a better solution for our case? Thanks, zainal
