[
https://issues.apache.org/jira/browse/STORM-2823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16258761#comment-16258761
]
Anton Alfred commented on STORM-2823:
-------------------------------------
What I am saying is Topology boundary should not be a worker process. It should
be virtual. So 1 Topology should not be equal to 1 worker process.
We should have an option to say N Topologies in 1 Worker process. If one choose
1 topology in 1 worker process which is the current design this should address
it.
Beyond this all the features of a topology should be exactly the same within
the process.
With regard to the fail fast option this too should be optional depending on
where the error occurred.
A topology can fail due to the following constructs
1. Technology (Storm itself)
2. Topology
* Configuration inside Topology
* Data inside (Tuples) Topology
Technology
While the Technology failures are beyond the control and when the worker
process dies due to out of memory, node restart etc. In this case the N
Topologies are in 1 worker process, all topologies die, which is the case even
if we kept it in multiple worker process it would have died when node is
restarted. The out of memory is similar instead of 1 topology, N topologies
die. This occurrence is rare.
Topology
The topology can fail due to
Configuration
This is accepted that as soon as some configuration on any topology fails, that
particular topology should fail so all topologies would fail in the combined
option. This occurrence is development time and is not wildly found at run time.
Data Inside (Tuples)
While failing the entire topology could be an option, this is not feasible in
most data processing. The variation in data being processed is so much a schema
bound approach is not always possible. So there is always those data that the
bolts code don't understand and either can choose to ignore the message and log
an error or error out die.
Since a topology dying has it own consequence of not able to know which bolt
processed last to replay, one should have an option not to die. Incase of a
data issue even dying and coming back alive does not solve the problem as the
same data would be replayed leading to the death of the topology again and
again. So a data issue that stops processing unless corrected or waits till
correction could be an approach that was taken. The other which is ignore the
error and continue is also the option which could also be given a choice. In
this case the bolt do not die but do a try catch and never allow the exception
to go outside itself and acknowledge the incorrect data so that the next
message can be processed.
So in practice it could be have an analogy to an API Server.
If 1 API Server Process hosts 1 API this is 1 Topology to 1 Worker Process
If 1 API Server Process hosts N APIs this is N Topologies in 1 Worker Process
So all aspects of failure of API to serve a request applies to the Worker
Process and should be independent of the other APIs hosted in the API Server.
> Ability to have an option to combine topologies at run time in a single
> process space
> -------------------------------------------------------------------------------------
>
> Key: STORM-2823
> URL: https://issues.apache.org/jira/browse/STORM-2823
> Project: Apache Storm
> Issue Type: New Feature
> Components: storm-core
> Affects Versions: 1.0.1
> Environment: CentOS 7, Docker
> Reporter: Anton Alfred
>
> Unlike an API server which service multiple APIs within the same process
> space, the Topologies needs to run in separate processes.
> Lets say we have Topology TP-1 which use 1 GB of memory.
> Now we create the same for n Topologies of TP-1..... TP-n
> As the topologies increase the memory allocation is now multiplied by the
> number of topologies.
> This design though scalable is not similar to the API route we have before
> which was within the same process space.
> So in a micros services world, each topology would be responsible for a
> similar set of objects, like employee, customer, product, order, order
> details etc.
> As the number of topologies increase the worker allocation is not sufficient.
> Most topologies are not utilized fully but since these are in different
> process space the memory allocated can't be used.
> If we have an ability to say that TP-1 --- TP 10 Can run within the same
> process space but behave like individual topologies we could conserve the
> resource usage.
> Now user are forced to combine topologies to the hardware provided with "if"
> logics to route the correct object that needs to be processed.
> This way one can still configure topologies as API in the same API server and
> reuse resources collectively for related group of topologies acting as micro
> services.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
