[
https://issues.apache.org/jira/browse/FLINK-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427167#comment-17427167
]
LI Zhennan commented on FLINK-2497:
-----------------------------------
This issue may related with FLIP-15:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66853132
While the fix/refactor is unclear, there is a walkaround:
https://gist.github.com/nanmu42/53fe23c3f15c777291108a7f4df62b9a
Hope that helps.
> Rework streaming iteration flow control
> ---------------------------------------
>
> Key: FLINK-2497
> URL: https://issues.apache.org/jira/browse/FLINK-2497
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
> Reporter: Gyula Fora
> Priority: Minor
> Labels: auto-deprioritized-major
>
> Currently streaming iterations use a simple Blocking queue to pass data from
> the iteration tails to the iteration heads.
> While this works most of the time, during increased load (or even short
> bursts) the increased amount of records in transit inside the loop can cause
> the limited number of input/output buffer-pools to fill up and deadlock the
> system. (caused by the back-pressure mechanism)
> One solution to this problem could be to remove back-pressure on the
> loop-edge by replacing the Blocking queue with some soft spilling queue that
> will write the records automatically to disk letting the iteration tail to
> progress with its input.
> For a more robust solution we need to implement more elaborate flow control
> in streaming programs containing iterations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
