[ https://issues.apache.org/jira/browse/BEAM-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851183#comment-16851183 ]
Akshay Iyangar commented on BEAM-7442: -------------------------------------- Yes .. I have taken those comments into consideration and should have the PR out . shortly.. like i was not able to assign the ticket to myself.. will appreciate if someone can help me with that.. > Bounded Reads for Flink Runner fails with OOM > --------------------------------------------- > > Key: BEAM-7442 > URL: https://issues.apache.org/jira/browse/BEAM-7442 > Project: Beam > Issue Type: Bug > Components: runner-flink > Reporter: Akshay Iyangar > Assignee: Maximilian Michels > Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > When Flink runner is reading from a bounded source and if the total number of > files are huge and the count is more. FlinkRunner throws an OOM error. This > is happening because the current implementation doesn't read them > sequentially but simultaneously thus causing all of the files to be in memory > which quickly breaks the cluster. > Solution : To wrap `UnboundedReadFromBoundedSource` class by a wrapper to see > that when the stream is a bounded source we make it read it sequentially > using a queue. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)