Dylan Hercher created BEAM-12685:
------------------------------------
Summary: Allow managed thread count in AvroIO
Key: BEAM-12685
URL: https://issues.apache.org/jira/browse/BEAM-12685
Project: Beam
Issue Type: Improvement
Components: io-java-files
Reporter: Dylan Hercher
During execution, the `ReadAllViaFileBasedSource` runs ReShuffle and creates an
un-grouped set of file range readers. This can easily cause OOM issues when
the number of groups changes as there is no limit to the number of concurrent
file reads.
Using Reshuffle.viaRandomKeys.withNumBuckets instead will allow the same
default behavior, but lets the user configure the number of readers as and when
needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
