Thanks Rui! It looks like I am having a bigger problem as Storm seems having issue now, maybe someone can give me a recommendation here. So right now I have 2 machines, and both of them have a scheduled maintenance, which the infra team has procedure which it'll call our stop script before machine reboot, and call our start script after reboot. In our stop script, it kills following components: nimbus, supervisor and ui, then in start script, it launches: nimbus, supervisor and ui. Right now I don't have script to check if topology is running yet, as I am still experimenting this machine reboot behavior. Note that those machines won't be brought down at the same time, so it'd be one down, and it's back up, and then bring another one down. So I'd imagine:
T1: Storm (and topology) runs on both machine A and machine B T2: Storm is killed on machine A, and machine is rebooted, assuming all spout/bolt will now run on machine B T3: machine A is back online, and Storm is launched at start up, assuming it'll pick up the topology T4: Storm is killed on machine B, and machine is rebooted T5: machine B is back online, and Storm is launched, and assuming everything works fine (somehow it's not) I guess some assumptions above about Storm continues to work is wrong, what is the best practice or recommended way for our situation? Thanks, zainal From: [email protected] At: 02/21/20 18:01:56To: Zainal Arifin (BLOOMBERG/ 731 LEX ) Cc: [email protected] Subject: Re: machine reboot The rebalance command through Storm UI does not guarantee you that tasks will go to another worker / machine. There is a command line rebalance command but it's used to increase the number of workers or executors: $ storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10 Without trying it, I'm not sure you can call it without parameters. And if even if it works, I'm not sure it will have the desired effect. Please have a look here for further information about the rebalance command: https://storm.apache.org/releases/2.1.0/Understanding-the-parallelism-of-a-Storm-topology.html On Fri, Feb 21, 2020, 23:22 Zainal Arifin (BLOOMBERG/ 731 LEX) <[email protected]> wrote: Thanks Rui! Can I force the rebalance from a script? I am thinking to add in my current script, after it starts Storm, and checks if the topology is running. If yes, what would be the command or API? From: [email protected] At: 02/21/20 16:48:53To: Zainal Arifin (BLOOMBERG/ 731 LEX ) , [email protected] Subject: Re: machine reboot As long the workers and tasks in machine A are healthy and sending hearbeats to Nimbus, they will keep running there. A redeployment of the topologies or a a rebalance command (you can use Storm UI for this), may send tasks to be executed in machine B. On Fri, Feb 21, 2020, 22:14 Zainal Arifin (BLOOMBERG/ 731 LEX) <[email protected]> wrote: Hi, We run Storm on 2 machines (let's call it machine A and B), and everything works fine. Then I want to test the machine being reboot, so basically when the machine being brought down, it'll call my script to stop Storm, and when the machine back up, it called my script to start Storm. From my testing to reboot 1 machine (let say B), after the machine is back online, I notice Storm runs fine there, but I see all spout/bolt tasks are running on machine A. I waited for a few minutes (now it's already 1 hour), and all tasks are still running on machine A. I'd expect some of the tasks will be automatically distributed (rebalance) to machine B, is that not the case? Or is there something I need to configure? Thanks!
