Anyone have any updates on automatically re-balancing storm workers when using an autoscaling group?
On Mon, Jun 22, 2015 at 2:50 AM, Andrew Wu <[email protected]> wrote: > I agree with you. I think there are 2 levels of scaling issue > 1) Topology level, mainly on number of workers. we are setting 2 twice > amount of workers than number of supervisors so that even we double number > of supervisors there will be at least 1 vm on each supervisor. it works on > sharing cpu load but does not work on memory. > 2) Cluster level, storm will not automatically use new supervisor until > topology is "rebalanced”. > > I have investigated this a bit and here’s something may help > This can be a solution for 2) > https://github.com/apache/storm/blob/master/STORM-UI-REST-API.md. you can > have a saperate monitor program to detect cluster changes and rebalance > topologies programatically. > 1) is a bit hard to solve, I*d like to know if there is any solution out > there as well. > > > > *Andrew Ge Wu* > Backend Developer & Integration Tech Lead > Mobile: +46 76 88 99 970 > > Website: www.innometrics.com > > > > On 02 Jun 2015, at 17:31, Dmitry Goldenberg <[email protected]> > wrote: > > Hi, > > What's the latest perspective on cluster autoscaling in Storm? We want > to be able to have machines provisioned/added or decommissioned via AWS > based on the system load. > > E.g. this post: > https://groups.google.com/forum/#!searchin/storm-user/autoscaling/storm-user/HLpFAZvbwgU/14RTS3ltiwsJ > didn't seem to get an answer. > > Is there a way to pre-provision a pool of machines ready to go but not > fully in use, then use them as needed? > > I see > https://issues.apache.org/jira/browse/STORM-594 > marked as Minor - ? > > This writeup seems to indicate there's autoscaling in Storm: > > https://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%[email protected]%3E > > Any thoughts/recommendations? Thanks. > > >
