[
https://issues.apache.org/jira/browse/BEAM-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reuven Lax closed BEAM-2302.
----------------------------
> WriteFiles with runner-determined sharding and large numbers of windows
> causes OOM errors
> -----------------------------------------------------------------------------------------
>
> Key: BEAM-2302
> URL: https://issues.apache.org/jira/browse/BEAM-2302
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
> Reporter: Reuven Lax
> Assignee: Reuven Lax
> Fix For: 2.1.0
>
>
> This is because the WriteWindowedBundles transform will create many file
> writers, and the sheer number of file buffers (which defaults to 64mb per
> writer) uses up all memory. The fix is the same as was done in BigQueryIO -
> if too many writers are opened, spill into a shuffle, and write the files
> after the shuffle
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
