[
https://issues.apache.org/jira/browse/FLINK-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gyula Fora closed FLINK-2497.
-----------------------------
Resolution: Won't Fix
> Rework streaming iteration flow control
> ---------------------------------------
>
> Key: FLINK-2497
> URL: https://issues.apache.org/jira/browse/FLINK-2497
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
> Reporter: Gyula Fora
> Priority: Minor
> Labels: auto-deprioritized-major
>
> Currently streaming iterations use a simple Blocking queue to pass data from
> the iteration tails to the iteration heads.
> While this works most of the time, during increased load (or even short
> bursts) the increased amount of records in transit inside the loop can cause
> the limited number of input/output buffer-pools to fill up and deadlock the
> system. (caused by the back-pressure mechanism)
> One solution to this problem could be to remove back-pressure on the
> loop-edge by replacing the Blocking queue with some soft spilling queue that
> will write the records automatically to disk letting the iteration tail to
> progress with its input.
> For a more robust solution we need to implement more elaborate flow control
> in streaming programs containing iterations.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
