[
https://issues.apache.org/jira/browse/FLINK-32598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhipeng Zhang updated FLINK-32598:
----------------------------------
Description:
In Flink ML, we use feedback edge to implement the iteration module. Suppose
the job topology is like `OpA -> HeadOperator -> OpB -> TailOperator`, then the
basic process of each iteration is as follows:
* At the first iteration, HeadOperator takes the input from OpA and forward it
to OpB.
* Later, OpB consumes the input from HeadOperator and forward the output to
TailOperator.
* Finally, TailOperator puts the records into a memory message queue and
HeadOperator consumes the message queue.
When the output from OpB contains many records and these records cannot be
consumed soon, the message queue would grow big and finally lead to OOM.
> Spill data from feedback edge to disk to avoid possible OOM
> -----------------------------------------------------------
>
> Key: FLINK-32598
> URL: https://issues.apache.org/jira/browse/FLINK-32598
> Project: Flink
> Issue Type: Improvement
> Components: Library / Machine Learning
> Reporter: Zhipeng Zhang
> Priority: Major
> Fix For: ml-2.4.0
>
>
> In Flink ML, we use feedback edge to implement the iteration module. Suppose
> the job topology is like `OpA -> HeadOperator -> OpB -> TailOperator`, then
> the basic process of each iteration is as follows:
> * At the first iteration, HeadOperator takes the input from OpA and forward
> it to OpB.
> * Later, OpB consumes the input from HeadOperator and forward the output to
> TailOperator.
> * Finally, TailOperator puts the records into a memory message queue and
> HeadOperator consumes the message queue.
> When the output from OpB contains many records and these records cannot be
> consumed soon, the message queue would grow big and finally lead to OOM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
