[
https://issues.apache.org/jira/browse/STORM-2823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16258774#comment-16258774
]
Jungtaek Lim commented on STORM-2823:
-------------------------------------
I'm afraid that the origin usage of Storm is in fact opposite, as you can see
"Why use Storm?" in the index page: Apache Storm is a free and open source
"distributed" realtime computation system. So we are dealing with multiple
workers in topology, though some topologies may run a worker per each, but that
may be not a thing which only streaming framework could handle.
The relevant technology of Streaming framework would be elasticity (which is
also fit to cloud env.), which is also in fact opposite, cause worker would be
terminated when topology can keep up processing the input without the worker,
and another worker would be launched when topology can't keep up processing
current input.
Btw, RAS enables users to assign required resources to component/topology
level, and assign workers to machine accordingly, so "launching homogeneous
worker" is no longer a limitation of Storm.
Please refer p21 ~ p24 of the slide to see how much RAS helps Yahoo! to utilize
their resources:
https://www.slideshare.net/HadoopSummit/resource-aware-scheduling-in-apache-spark
(Sadly the link is not properly titled. Its title is "Resource Aware Scheduling
in Apache Storm".)
> Ability to have an option to combine topologies at run time in a single
> process space
> -------------------------------------------------------------------------------------
>
> Key: STORM-2823
> URL: https://issues.apache.org/jira/browse/STORM-2823
> Project: Apache Storm
> Issue Type: New Feature
> Components: storm-core
> Affects Versions: 1.0.1
> Environment: CentOS 7, Docker
> Reporter: Anton Alfred
>
> Unlike an API server which service multiple APIs within the same process
> space, the Topologies needs to run in separate processes.
> Lets say we have Topology TP-1 which use 1 GB of memory.
> Now we create the same for n Topologies of TP-1..... TP-n
> As the topologies increase the memory allocation is now multiplied by the
> number of topologies.
> This design though scalable is not similar to the API route we have before
> which was within the same process space.
> So in a micros services world, each topology would be responsible for a
> similar set of objects, like employee, customer, product, order, order
> details etc.
> As the number of topologies increase the worker allocation is not sufficient.
> Most topologies are not utilized fully but since these are in different
> process space the memory allocated can't be used.
> If we have an ability to say that TP-1 --- TP 10 Can run within the same
> process space but behave like individual topologies we could conserve the
> resource usage.
> Now user are forced to combine topologies to the hardware provided with "if"
> logics to route the correct object that needs to be processed.
> This way one can still configure topologies as API in the same API server and
> reuse resources collectively for related group of topologies acting as micro
> services.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
