[
https://issues.apache.org/jira/browse/STORM-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425334#comment-17425334
]
Roshan Naik commented on STORM-3685:
------------------------------------
[~bipinprasad] & [~ethanli]
a very very late +1 on this approach of warning but not banning cycles.
Sometimes there is a need for cycles if downstream operators need to report
progress or send metadata to upstream operators. This can be used for flow
control where sources can be suspended/resumed based on progress notifications
from downstream operators. For example:
* For (event-time based) windowed joins, can be useful to synchronize streams
involved around event time boundaries. Allowing both streams to progress
without any flow controls can cause one stream to move far too ahead compared
to the other stream and that will lead to having too open windows opens for too
long... and thus lead to excessive memory consumption.
* The [Kappa+ Architecture|https://www.youtube.com/watch?v=4qSlsYogALo] uses
flow control on sources for enabling usecases like efficient backfills using
native streaming mode execution.
* As you know, technically, any topology with ACKing enabled has a cycle. And
{{topology.max.spout.pending}} is a form of flow control.
> Detect Loops in Topology at Submit
> ----------------------------------
>
> Key: STORM-3685
> URL: https://issues.apache.org/jira/browse/STORM-3685
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-client
> Reporter: Bipin Prasad
> Assignee: Bipin Prasad
> Priority: Major
> Fix For: 2.3.0
>
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> Topology graph is expected to be a Directed Acyclic Graph (DAG). Detect
> cycles in DAG when Topology is submitted and prevent cycles in Topology.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
