[
https://issues.apache.org/jira/browse/BEAM-12685?focusedWorklogId=633859&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633859
]
ASF GitHub Bot logged work on BEAM-12685:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Aug/21 22:07
Start Date: 04/Aug/21 22:07
Worklog Time Spent: 10m
Work Description: dhercher commented on pull request #15246:
URL: https://github.com/apache/beam/pull/15246#issuecomment-893006636
My issue is that the reshuffle is having the inverse effect, since we force
full parallelism at this stage its very easy to cause OOM crash loops when
reading many files at once.
At least in the Dataflow runner, it does not appear it knows how to properly
scale down the number of threads when this sort of issue occurs to avoid the
issue. Removing the reshuffle at least allows for someone to force their
desired behavior when they know what they want rather than force a single
aggressive strategy.
I suppose the feature flag could control if the reshuffle occurs at all to
maintain the current behavior by default and allow the user to manage the
parallelism where needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 633859)
Remaining Estimate: 50m (was: 1h)
Time Spent: 1h 10m (was: 1h)
> Allow managed thread count in AvroIO
> ------------------------------------
>
> Key: BEAM-12685
> URL: https://issues.apache.org/jira/browse/BEAM-12685
> Project: Beam
> Issue Type: Improvement
> Components: io-java-files
> Reporter: Dylan Hercher
> Priority: P2
> Original Estimate: 2h
> Time Spent: 1h 10m
> Remaining Estimate: 50m
>
> During execution, the `ReadAllViaFileBasedSource` runs ReShuffle and creates
> an un-grouped set of file range readers. This can easily cause OOM issues
> when the number of groups changes as there is no limit to the number of
> concurrent file reads.
>
> Using Reshuffle.viaRandomKeys.withNumBuckets instead will allow the same
> default behavior, but lets the user configure the number of readers as and
> when needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
