[
https://issues.apache.org/jira/browse/STORM-594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14634297#comment-14634297
]
Jon Weygandt commented on STORM-594:
------------------------------------
I have some interest in solving this problem. We have a Hack Week coming up,
and there are several folks interested. Having read the proposals, as well as
the Dynamic Storm paper, I believe my current ideas line up a bit closer to the
original proposal.
I will attempt to tackle it from what I'm going to call a "Balanced Storm
Topology" (don't know if there is another name): This is where the parallelism
will be auto-scaled to a multiple of the number of auto-scaled workers, and
each worker has the same number and type of bolts and spouts. I have observed
that the FAQs indicate this is a good thing, and Nimbus seems to gravitate to
this configuration if the parallelism is right. I believe this will be the most
stable layout of the topology, short of one instance per worker, and one
worker per host/container (sounds like a description for Heron).
My plan is to use latency*events/sec normalized to 0-100% as the primary metric
for determine these values.
One of the goals will be to limit the number of iterations as rebalancing a
Storm topology is not without its costs. Another feature of our use cases is
that there are daily and weekly fluctuations in our load, but the max is
somewhat stable (it grows slowly over time). And since we need to be guaranteed
to have enough hardware to handle our peak loads, the sizing parameters and
scaling algorithms will reflect this scenario. Generally we know the max input
rate (or predicted input rate) at the spouts, the system will be sized to that
at a minimum, and there won't be any scale-down below this limit.
Because of this I'd like to consider is the "replay on startup" issue, where if
Storm is down for a bit, and you have messages queued up, that on startup, you
system will typically burst to 100% till the backlog is drained. I wish to put
some controls in place to handle this, as I don't think the system should try
to scale up briefly, and then scale down. This will be a bit more difficult, as
I believe that I need to know from each spout if it is playing a backlog, or
current time, as well as the backlog size, and the rate at which it is
changing. Sounds like perhaps a new interface that auto-scaling spouts might
support?
Then as inputs:
*) user can elect certain headroom on the latency*events number (or we may have
good defaults)
*) planned number of events for each spout, as the current load may be less, or
may have daily variations, or you may want to plan ahead for holiday time or
some such.
*) estimated maximum down time and desired backlog catch-up time.
And given time, various events will need to come out of Nimbus to external
systems to add more hardware, as well as various alarms/notifications.
> Auto-Scaling Resources in a Topology
> ------------------------------------
>
> Key: STORM-594
> URL: https://issues.apache.org/jira/browse/STORM-594
> Project: Apache Storm
> Issue Type: New Feature
> Reporter: HARSHA BALASUBRAMANIAN
> Assignee: Pooyan Jamshidi
> Priority: Minor
> Attachments: Algorithm for Auto-Scaling.pdf, Project Plan and
> Scope.pdf
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> A useful feature missing in Storm topologies is the ability to auto-scale
> resources, based on a pre-configured metric. The feature proposed here aims
> to build such a auto-scaling mechanism using a feedback system. A brief
> overview of the feature is provided here. The finer details of the required
> components and the scaling algorithm (uses a Feedback System) are provided in
> the PDFs attached.
> Brief Overview:
> Topologies may get created with or (ideally) without parallelism hints and
> tasks in their bolts and spouts, before submitting them, If auto-scaling is
> set in the topology (using a Boolean flag), the topology will also get
> submitted to the auto-scale module.
> The auto-scale module will read a pre-configured metric (threshold/min) from
> a configuration file. Using this value, the topology's resources will be
> modified till the threshold is reached. At each stage in the auto-scale
> module's execution, feedback from the previous execution will be used to tune
> the resources.
> The systems that need to be in place to achieve this are:
> 1. Metrics which provide the current threshold (no: of acks per minute) for a
> topology's spouts and bolts.
> 2. Access to Storm's CLI tool which can change a topology's resources are
> runtime.
> 3. A new java or clojure module which runs within the Nimbus daemon or in
> parallel to it. This will be the auto-scale module.
> Limitations: (This is not an exhaustive list. More will be added as the
> design matures. Also, some of the points here may get resolved)
> To test the feature there will be a number of limitations in the first
> release. As the feature matures, it will be allowed to scale more
> 1. The auto-scale module will be limited to a few topologies (maybe 4 or 5 at
> maximum)
> 2. New bolts will not be added to scale a topology. This feature will be
> limited to increasing the resources within the existing topology.
> 3. Topology resources will not be decreased when it is running at more than
> the required number (except for a few cases)
> 4. This feature will work only for long-running topologies where the input
> threshold can become equal to or greater than the required threshold
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
