I agree with you. I think there are 2 levels of scaling issue 1) Topology level, mainly on number of workers. we are setting 2 twice amount of workers than number of supervisors so that even we double number of supervisors there will be at least 1 vm on each supervisor. it works on sharing cpu load but does not work on memory. 2) Cluster level, storm will not automatically use new supervisor until topology is "rebalanced”.
I have investigated this a bit and here’s something may help This can be a solution for 2) https://github.com/apache/storm/blob/master/STORM-UI-REST-API.md. you can have a saperate monitor program to detect cluster changes and rebalance topologies programatically. 1) is a bit hard to solve, I*d like to know if there is any solution out there as well. Andrew Ge Wu Backend Developer & Integration Tech Lead Mobile: +46 76 88 99 970 Website: www.innometrics.com<http://www.innometrics.com/> [cid:A29501B8-EA20-463B-A4A4-203BB3DA4788] On 02 Jun 2015, at 17:31, Dmitry Goldenberg <[email protected]<mailto:[email protected]>> wrote: Hi, What's the latest perspective on cluster autoscaling in Storm? We want to be able to have machines provisioned/added or decommissioned via AWS based on the system load. E.g. this post: https://groups.google.com/forum/#!searchin/storm-user/autoscaling/storm-user/HLpFAZvbwgU/14RTS3ltiwsJ didn't seem to get an answer. Is there a way to pre-provision a pool of machines ready to go but not fully in use, then use them as needed? I see https://issues.apache.org/jira/browse/STORM-594 marked as Minor - ? This writeup seems to indicate there's autoscaling in Storm: https://mail-archives.apache.org/mod_mbox/storm-user/201501.mbox/%[email protected]%3E Any thoughts/recommendations? Thanks.
