[jira] [Updated] (BEAM-9049) MemoryMonitor thrashing detection is too aggressive for batch workers
[ https://issues.apache.org/jira/browse/BEAM-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9049: -- This Jira ticket has a pull request attached to it, but is still open. Did the pull request resolve the issue? If so, could you please mark it resolved? This will help the project have a clear view of its open issues. > MemoryMonitor thrashing detection is too aggressive for batch workers > - > > Key: BEAM-9049 > URL: https://issues.apache.org/jira/browse/BEAM-9049 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Brian Hulette >Priority: P3 > Time Spent: 1h 50m > Remaining Estimate: 0h > > In the streaming dataflow worker we've implemented push-back so that we will > reduce parallelism when there is memory pressure. Since we cannot do this on > the batch worker, it doesn't make sense for our thrashing detection to be so > aggressive. We should increase the thresholds used for thrashing detection > when running on Batch workloads. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (BEAM-9049) MemoryMonitor thrashing detection is too aggressive for batch workers
[ https://issues.apache.org/jira/browse/BEAM-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-9049: Labels: (was: stale-P2) > MemoryMonitor thrashing detection is too aggressive for batch workers > - > > Key: BEAM-9049 > URL: https://issues.apache.org/jira/browse/BEAM-9049 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Brian Hulette >Priority: P3 > Time Spent: 1h 40m > Remaining Estimate: 0h > > In the streaming dataflow worker we've implemented push-back so that we will > reduce parallelism when there is memory pressure. Since we cannot do this on > the batch worker, it doesn't make sense for our thrashing detection to be so > aggressive. We should increase the thresholds used for thrashing detection > when running on Batch workloads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9049) MemoryMonitor thrashing detection is too aggressive for batch workers
[ https://issues.apache.org/jira/browse/BEAM-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-9049: Priority: P3 (was: P2) > MemoryMonitor thrashing detection is too aggressive for batch workers > - > > Key: BEAM-9049 > URL: https://issues.apache.org/jira/browse/BEAM-9049 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Brian Hulette >Priority: P3 > Labels: stale-P2 > Time Spent: 1h 40m > Remaining Estimate: 0h > > In the streaming dataflow worker we've implemented push-back so that we will > reduce parallelism when there is memory pressure. Since we cannot do this on > the batch worker, it doesn't make sense for our thrashing detection to be so > aggressive. We should increase the thresholds used for thrashing detection > when running on Batch workloads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9049) MemoryMonitor thrashing detection is too aggressive for batch workers
[ https://issues.apache.org/jira/browse/BEAM-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-9049: Labels: stale-P2 (was: ) > MemoryMonitor thrashing detection is too aggressive for batch workers > - > > Key: BEAM-9049 > URL: https://issues.apache.org/jira/browse/BEAM-9049 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Brian Hulette >Priority: P2 > Labels: stale-P2 > Time Spent: 1h 40m > Remaining Estimate: 0h > > In the streaming dataflow worker we've implemented push-back so that we will > reduce parallelism when there is memory pressure. Since we cannot do this on > the batch worker, it doesn't make sense for our thrashing detection to be so > aggressive. We should increase the thresholds used for thrashing detection > when running on Batch workloads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9049) MemoryMonitor thrashing detection is too aggressive for batch workers
[ https://issues.apache.org/jira/browse/BEAM-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-9049: Labels: (was: stale-assigned) > MemoryMonitor thrashing detection is too aggressive for batch workers > - > > Key: BEAM-9049 > URL: https://issues.apache.org/jira/browse/BEAM-9049 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Brian Hulette >Priority: P2 > Time Spent: 1h 40m > Remaining Estimate: 0h > > In the streaming dataflow worker we've implemented push-back so that we will > reduce parallelism when there is memory pressure. Since we cannot do this on > the batch worker, it doesn't make sense for our thrashing detection to be so > aggressive. We should increase the thresholds used for thrashing detection > when running on Batch workloads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9049) MemoryMonitor thrashing detection is too aggressive for batch workers
[ https://issues.apache.org/jira/browse/BEAM-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9049: -- Labels: stale-assigned (was: ) > MemoryMonitor thrashing detection is too aggressive for batch workers > - > > Key: BEAM-9049 > URL: https://issues.apache.org/jira/browse/BEAM-9049 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: P2 > Labels: stale-assigned > Time Spent: 1h 40m > Remaining Estimate: 0h > > In the streaming dataflow worker we've implemented push-back so that we will > reduce parallelism when there is memory pressure. Since we cannot do this on > the batch worker, it doesn't make sense for our thrashing detection to be so > aggressive. We should increase the thresholds used for thrashing detection > when running on Batch workloads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9049) MemoryMonitor thrashing detection is too aggressive for batch workers
[ https://issues.apache.org/jira/browse/BEAM-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9049: -- Fix Version/s: (was: 2.19.0) > MemoryMonitor thrashing detection is too aggressive for batch workers > - > > Key: BEAM-9049 > URL: https://issues.apache.org/jira/browse/BEAM-9049 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > In the streaming dataflow worker we've implemented push-back so that we will > reduce parallelism when there is memory pressure. Since we cannot do this on > the batch worker, it doesn't make sense for our thrashing detection to be so > aggressive. We should increase the thresholds used for thrashing detection > when running on Batch workloads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9049) MemoryMonitor thrashing detection is too aggressive for batch workers
[ https://issues.apache.org/jira/browse/BEAM-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9049: --- Status: Open (was: Triage Needed) > MemoryMonitor thrashing detection is too aggressive for batch workers > - > > Key: BEAM-9049 > URL: https://issues.apache.org/jira/browse/BEAM-9049 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Fix For: 2.19.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > In the streaming dataflow worker we've implemented push-back so that we will > reduce parallelism when there is memory pressure. Since we cannot do this on > the batch worker, it doesn't make sense for our thrashing detection to be so > aggressive. We should increase the thresholds used for thrashing detection > when running on Batch workloads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9049) MemoryMonitor thrashing detection is too aggressive for batch workers
[ https://issues.apache.org/jira/browse/BEAM-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Hulette updated BEAM-9049: Description: In the streaming dataflow worker we've implemented push-back so that we will reduce parallelism when there is memory pressure. Since we cannot do this on the batch worker, it doesn't make sense for our thrashing detection to be so aggressive. We should increase the thresholds used for thrashing detection when running on Batch workloads. (was: MemoryMonitor has many knobs that are just hard-coded for now. We should make some (or all) of these knobs available via pipeline options, so that users have control over when workers will be killed due to thrashing. https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/MemoryMonitor.java#L123) > MemoryMonitor thrashing detection is too aggressive for batch workers > - > > Key: BEAM-9049 > URL: https://issues.apache.org/jira/browse/BEAM-9049 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Fix For: 2.19.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > In the streaming dataflow worker we've implemented push-back so that we will > reduce parallelism when there is memory pressure. Since we cannot do this on > the batch worker, it doesn't make sense for our thrashing detection to be so > aggressive. We should increase the thresholds used for thrashing detection > when running on Batch workloads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9049) MemoryMonitor thrashing detection is too aggressive for batch workers
[ https://issues.apache.org/jira/browse/BEAM-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Hulette updated BEAM-9049: Summary: MemoryMonitor thrashing detection is too aggressive for batch workers (was: Add knobs for dataflow worker MemoryMonitor) > MemoryMonitor thrashing detection is too aggressive for batch workers > - > > Key: BEAM-9049 > URL: https://issues.apache.org/jira/browse/BEAM-9049 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Fix For: 2.19.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > MemoryMonitor has many knobs that are just hard-coded for now. We should make > some (or all) of these knobs available via pipeline options, so that users > have control over when workers will be killed due to thrashing. > https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/MemoryMonitor.java#L123 -- This message was sent by Atlassian Jira (v8.3.4#803005)
