[
https://issues.apache.org/jira/browse/BEAM-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851183#comment-16851183
]
Akshay Iyangar edited comment on BEAM-7442 at 5/29/19 6:47 PM:
---------------------------------------------------------------
Yes .. I have taken those comments into consideration and should have the PR
out . shortly.. like i was not able to assign the ticket to myself.. will
appreciate if someone can help me with that.
was (Author: aiyangar):
Yes .. I have taken those comments into consideration and should have the PR
out . shortly.. like i was not able to assign the ticket to myself.. will
appreciate if someone can help me with that..
> Bounded Reads for Flink Runner fails with OOM
> ---------------------------------------------
>
> Key: BEAM-7442
> URL: https://issues.apache.org/jira/browse/BEAM-7442
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Reporter: Akshay Iyangar
> Assignee: Maximilian Michels
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When Flink runner is reading from a bounded source and if the total number of
> files are huge and the count is more. FlinkRunner throws an OOM error. This
> is happening because the current implementation doesn't read them
> sequentially but simultaneously thus causing all of the files to be in memory
> which quickly breaks the cluster.
> Solution : To wrap `UnboundedReadFromBoundedSource` class by a wrapper to see
> that when the stream is a bounded source we make it read it sequentially
> using a queue.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
