[
https://issues.apache.org/jira/browse/FLINK-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14680209#comment-14680209
]
Stephan Ewen commented on FLINK-2497:
-------------------------------------
I like this improvement.
I would make this dependent on the upcoming interface changes for dynamic
memory management, so the back channel can react to availability/pressure for
memory.
> Rework streaming iteration flow control
> ---------------------------------------
>
> Key: FLINK-2497
> URL: https://issues.apache.org/jira/browse/FLINK-2497
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
> Reporter: Gyula Fora
>
> Currently streaming iterations use a simple Blocking queue to pass data from
> the iteration tails to the iteration heads.
> While this works most of the time, during increased load (or even short
> bursts) the increased amount of records in transit inside the loop can cause
> the limited number of input/output buffer-pools to fill up and deadlock the
> system. (caused by the back-pressure mechanism)
> One solution to this problem could be to remove back-pressure on the
> loop-edge by replacing the Blocking queue with some soft spilling queue that
> will write the records automatically to disk letting the iteration tail to
> progress with its input.
> For a more robust solution we need to implement more elaborate flow control
> in streaming programs containing iterations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
