[
https://issues.apache.org/jira/browse/BEAM-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Hulette updated BEAM-9049:
--------------------------------
Description: In the streaming dataflow worker we've implemented push-back
so that we will reduce parallelism when there is memory pressure. Since we
cannot do this on the batch worker, it doesn't make sense for our thrashing
detection to be so aggressive. We should increase the thresholds used for
thrashing detection when running on Batch workloads. (was: MemoryMonitor has
many knobs that are just hard-coded for now. We should make some (or all) of
these knobs available via pipeline options, so that users have control over
when workers will be killed due to thrashing.
https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/MemoryMonitor.java#L123)
> MemoryMonitor thrashing detection is too aggressive for batch workers
> ---------------------------------------------------------------------
>
> Key: BEAM-9049
> URL: https://issues.apache.org/jira/browse/BEAM-9049
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Reporter: Brian Hulette
> Assignee: Brian Hulette
> Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> In the streaming dataflow worker we've implemented push-back so that we will
> reduce parallelism when there is memory pressure. Since we cannot do this on
> the batch worker, it doesn't make sense for our thrashing detection to be so
> aggressive. We should increase the thresholds used for thrashing detection
> when running on Batch workloads.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
