Hi all,
I have investigated ways of auto scaling Storm topologies recently and
it is perfectly feasible.
- For monitoring, you can use either the Thrift interface or the new
MetricsConsumer bolt, it's probably possible using the REST API as well,
but I haven't tried that out yet.
- For scaling, Storm automatically discovers the addition of new nodes
(VMs) to the cluster and allows you to change the number of executors
(threads) per component on the fly.
- You can as well plug in a custom scheduler, which is called upon each
rebalancing request, to place the executors of your topology as you see
fits.
I hope this could help.
Ahmed
On 01/28/2015 09:55 PM, Parth Brahmbhatt wrote:
Not sure how much progress has been made on
https://issues.apache.org/jira/browse/STORM-594 but that is the only
initiative that I am aware of.
Thanks
Parth
On 1/28/15, 12:42 PM, "Margus Roo" <[email protected]> wrote:
I have not heard or read about it from the documentation.
Margus (margusja) Roo
http://margus.roo.ee
skype: margusja
+372 51 480
On 28/01/15 20:59, Manpreet Singh wrote:
Hi,
I am a newbie to Apache Storm and analyzing it for a product. I am
comparing it with Gearman from parallel processing aspect. So far
Apache Storm seems to be winning on most of the aspects. But one of
the important aspect which I got stuck is 'auto-scale'. Auto-scale (up
as well as down) of storm resources doesn't look possible. This is
critical for my product as stream data could vary a lot through out
the day. Having auto-scale will save me cost as well as monitoring the
usage every now and then.
Are there any workarounds to this problem with Apache Storm?
Thanks in advance.
-Manpreet.
