Akshay Iyangar created BEAM-7442:
------------------------------------
Summary: Bounded Reads for Flink Runner fails with OOM
Key: BEAM-7442
URL: https://issues.apache.org/jira/browse/BEAM-7442
Project: Beam
Issue Type: Bug
Components: runner-flink
Reporter: Akshay Iyangar
When Flink runner is reading from a bounded source and if the total number of
files are huge and the count is more. FlinkRunner throws an OOM error. This is
happening because the current implementation doesn't read them sequentially but
simultaneously thus causing all of the files to be in memory which quickly
breaks the cluster.
Solution : To wrap `UnboundedReadFromBoundedSource` class by a wrapper to see
that when the stream is a bounded source we make it read it sequentially using
a queue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
