[
https://issues.apache.org/jira/browse/FLINK-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14636523#comment-14636523
]
Gyula Fora commented on FLINK-2390:
-----------------------------------
That's the whole point. The head operator receives inputs as either feedback or
normal input. We can detect when the normal inputs finish as the input
operators terminate at one point. The tricky thing is knowing when the loop has
terminated, and that's what the proposed algorithm does by sending events
around the loop like we do with checkpoint barriers.
> Replace iteration timeout with algorithm for detecting termination
> ------------------------------------------------------------------
>
> Key: FLINK-2390
> URL: https://issues.apache.org/jira/browse/FLINK-2390
> Project: Flink
> Issue Type: New Feature
> Components: Streaming
> Reporter: Gyula Fora
> Fix For: 0.10
>
>
> Currently the user can set a timeout which will shut down the iteration
> source/sink nodes if no new data is received during that time to allow
> program termination in iterative streaming jobs.
> This method is used due to the non-trivial nature of termination in iterative
> streaming jobs. While termination is not a main concern in long running
> streaming jobs, this behaviour makes iterative tests non-deterministic and
> they often fail on travis due to the timeout. Also setting a timeout can
> cause jobs to terminate prematurely.
> I propose to remove iteration timeouts and replace it with the following
> algorithm for detecting termination:
> -We first identify loop edges in the jobgraph (the channels from the
> iteration sources to the head operators)
> -Once the head operators (the ones with loop input) finish with all their
> non-loop inputs they broadcast a marker to their outputs.
> -Each operator will broadcast a marker once it received a marker from all its
> non-finished inputs
> -Iteration sources are terminated when they receive 2 consecutive markers
> without receiving any record in-between
> The idea behind the algorithm is to find out when no more outputs are
> generated from the operators inside an iteration after their normal inputs
> are finished.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
