Re: question on deactivate and kill
To change the number of executors to a particular bolt, you can execute this on a Nimbus: storm rebalance exclamation-topology -e even-digit-bolt=3 But again, this will have a practical effect if you have spare tasks that those executors might work on. On Wed, 15 Jan 2020 at 17:32, Zainal Arifin (BLOOMBERG/ 731 LEX) < zari...@bloomberg.net> wrote: > Thanks Rui for the answer. > > The link mentioned that we can use Storm Web UI to re-balance the > topology, but can I increase the number of parallelism of particular bolt > only? I guess not. > > Btw, we are using streamparse, > https://streamparse.readthedocs.io/en/stable/topologies.html#parallelism-and-workers > > > From: user@storm.apache.org At: 01/15/20 09:58:08 > To: Zainal Arifin (BLOOMBERG/ 731 LEX ) , > user@storm.apache.org > Subject: Re: question on deactivate and kill > > Deactivating a topology means the spouts stop consuming, but everything > else continues to work as expected, meaning your bolts will process all in > flight tuples. > You can change the number of workers and executors in runtime. In order to > scale up the latter, in you need to have previously submitted the topology > with a larger number of tasks than executors. > > > https://storm.apache.org/releases/1.2.3/Understanding-the-parallelism-of-a-Storm-topology.html > > On Wed, 15 Jan 2020 at 16:41, Zainal Arifin (BLOOMBERG/ 731 LEX) < > zari...@bloomberg.net> wrote: > >> Hi, >> Right now we have a bolt task that is in back log, i.e. ~10 hours behind >> processing the data. >> There was a bug and we have the fix which will speed up drastically. >> I think if I kill the topology (to deploy the fix), I will definitely >> lose those events been processed by the bolt, is that correct? >> How about if I just de-activate the topology, will the bolt still >> processing the events, and only spouts stop receiving the events? >> I wish there is a way to increase the parallelism for the bolt at run >> time, because even we don't deploy the fix, but if there are multiple >> instances of that bolt running, it'll still speed up. Is there a way to do >> so (we run v1.1.1)? Or perhaps there is a better solution for our case? >> >> Thanks, >> zainal >> > >
Re: question on deactivate and kill
Thanks Rui for the answer. The link mentioned that we can use Storm Web UI to re-balance the topology, but can I increase the number of parallelism of particular bolt only? I guess not. Btw, we are using streamparse, https://streamparse.readthedocs.io/en/stable/topologies.html#parallelism-and-workers From: user@storm.apache.org At: 01/15/20 09:58:08To: Zainal Arifin (BLOOMBERG/ 731 LEX ) , user@storm.apache.org Subject: Re: question on deactivate and kill Deactivating a topology means the spouts stop consuming, but everything else continues to work as expected, meaning your bolts will process all in flight tuples. You can change the number of workers and executors in runtime. In order to scale up the latter, in you need to have previously submitted the topology with a larger number of tasks than executors. https://storm.apache.org/releases/1.2.3/Understanding-the-parallelism-of-a-Storm-topology.html On Wed, 15 Jan 2020 at 16:41, Zainal Arifin (BLOOMBERG/ 731 LEX) wrote: Hi, Right now we have a bolt task that is in back log, i.e. ~10 hours behind processing the data. There was a bug and we have the fix which will speed up drastically. I think if I kill the topology (to deploy the fix), I will definitely lose those events been processed by the bolt, is that correct? How about if I just de-activate the topology, will the bolt still processing the events, and only spouts stop receiving the events? I wish there is a way to increase the parallelism for the bolt at run time, because even we don't deploy the fix, but if there are multiple instances of that bolt running, it'll still speed up. Is there a way to do so (we run v1.1.1)? Or perhaps there is a better solution for our case? Thanks, zainal
Re: question on deactivate and kill
Deactivating a topology means the spouts stop consuming, but everything else continues to work as expected, meaning your bolts will process all in flight tuples. You can change the number of workers and executors in runtime. In order to scale up the latter, in you need to have previously submitted the topology with a larger number of tasks than executors. https://storm.apache.org/releases/1.2.3/Understanding-the-parallelism-of-a-Storm-topology.html On Wed, 15 Jan 2020 at 16:41, Zainal Arifin (BLOOMBERG/ 731 LEX) < zari...@bloomberg.net> wrote: > Hi, > Right now we have a bolt task that is in back log, i.e. ~10 hours behind > processing the data. > There was a bug and we have the fix which will speed up drastically. > I think if I kill the topology (to deploy the fix), I will definitely lose > those events been processed by the bolt, is that correct? > How about if I just de-activate the topology, will the bolt still > processing the events, and only spouts stop receiving the events? > I wish there is a way to increase the parallelism for the bolt at run > time, because even we don't deploy the fix, but if there are multiple > instances of that bolt running, it'll still speed up. Is there a way to do > so (we run v1.1.1)? Or perhaps there is a better solution for our case? > > Thanks, > zainal >
