[jira] [Work logged] (BEAM-8962) FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction
[ https://issues.apache.org/jira/browse/BEAM-8962?focusedWorklogId=364121=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364121 ] ASF GitHub Bot logged work on BEAM-8962: Author: ASF GitHub Bot Created on: 28/Dec/19 03:08 Start Date: 28/Dec/19 03:08 Worklog Time Spent: 10m Work Description: tweise commented on pull request #10415: [BEAM-8962] Report Flink metric accumulator only when pipeline ends URL: https://github.com/apache/beam/pull/10415 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 364121) Time Spent: 2h 50m (was: 2h 40m) > FlinkMetricContainer causes churn in the JobManager and lets the web frontend > malfunction > - > > Key: BEAM-8962 > URL: https://issues.apache.org/jira/browse/BEAM-8962 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.19.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > The {{FlinkMetricContainer}} wraps the Beam metric container for reporting > metrics, but also stores them as Flink accumulators. With high parallelism > jobs with over a thousand tasks and many built-in Beam metrics for every Beam > step, this can accumulate to over 100MB of serialized data which is stored in > the JobManager's ExecutionGraph. This then fails to even sent over the wire, > due to the akka.framesize limit (10MB by default), and manifests in {{500 > Internal Server Error}}s in the web frontend. > We need to introduce an option to disable the reporting via accumulators. It > is mostly useful for batch workloads where you can retrieve the final > accumulator values at the end of the job. It adds a lot of memory and network > overhead. > Perhaps we could even turn off the accumulators for streaming jobs, or > entirely and make them opt-in. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8962) FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction
[ https://issues.apache.org/jira/browse/BEAM-8962?focusedWorklogId=364082=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364082 ] ASF GitHub Bot logged work on BEAM-8962: Author: ASF GitHub Bot Created on: 28/Dec/19 01:44 Start Date: 28/Dec/19 01:44 Worklog Time Spent: 10m Work Description: tweise commented on issue #10415: [BEAM-8962] Report Flink metric accumulator only when pipeline ends URL: https://github.com/apache/beam/pull/10415#issuecomment-569374451 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 364082) Time Spent: 2h 40m (was: 2.5h) > FlinkMetricContainer causes churn in the JobManager and lets the web frontend > malfunction > - > > Key: BEAM-8962 > URL: https://issues.apache.org/jira/browse/BEAM-8962 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.19.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > The {{FlinkMetricContainer}} wraps the Beam metric container for reporting > metrics, but also stores them as Flink accumulators. With high parallelism > jobs with over a thousand tasks and many built-in Beam metrics for every Beam > step, this can accumulate to over 100MB of serialized data which is stored in > the JobManager's ExecutionGraph. This then fails to even sent over the wire, > due to the akka.framesize limit (10MB by default), and manifests in {{500 > Internal Server Error}}s in the web frontend. > We need to introduce an option to disable the reporting via accumulators. It > is mostly useful for batch workloads where you can retrieve the final > accumulator values at the end of the job. It adds a lot of memory and network > overhead. > Perhaps we could even turn off the accumulators for streaming jobs, or > entirely and make them opt-in. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8962) FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction
[ https://issues.apache.org/jira/browse/BEAM-8962?focusedWorklogId=364078=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364078 ] ASF GitHub Bot logged work on BEAM-8962: Author: ASF GitHub Bot Created on: 28/Dec/19 01:17 Start Date: 28/Dec/19 01:17 Worklog Time Spent: 10m Work Description: tweise commented on issue #10415: [BEAM-8962] Report Flink metric accumulator only when pipeline ends URL: https://github.com/apache/beam/pull/10415#issuecomment-569372564 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 364078) Time Spent: 2.5h (was: 2h 20m) > FlinkMetricContainer causes churn in the JobManager and lets the web frontend > malfunction > - > > Key: BEAM-8962 > URL: https://issues.apache.org/jira/browse/BEAM-8962 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.19.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > The {{FlinkMetricContainer}} wraps the Beam metric container for reporting > metrics, but also stores them as Flink accumulators. With high parallelism > jobs with over a thousand tasks and many built-in Beam metrics for every Beam > step, this can accumulate to over 100MB of serialized data which is stored in > the JobManager's ExecutionGraph. This then fails to even sent over the wire, > due to the akka.framesize limit (10MB by default), and manifests in {{500 > Internal Server Error}}s in the web frontend. > We need to introduce an option to disable the reporting via accumulators. It > is mostly useful for batch workloads where you can retrieve the final > accumulator values at the end of the job. It adds a lot of memory and network > overhead. > Perhaps we could even turn off the accumulators for streaming jobs, or > entirely and make them opt-in. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8962) FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction
[ https://issues.apache.org/jira/browse/BEAM-8962?focusedWorklogId=364003=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364003 ] ASF GitHub Bot logged work on BEAM-8962: Author: ASF GitHub Bot Created on: 27/Dec/19 17:37 Start Date: 27/Dec/19 17:37 Worklog Time Spent: 10m Work Description: mxm commented on issue #10415: [BEAM-8962] Report Flink metric accumulator only when pipeline ends URL: https://github.com/apache/beam/pull/10415#issuecomment-569313477 Rebased and added tests to verify that the registration hook is called on `close()`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 364003) Time Spent: 2h 20m (was: 2h 10m) > FlinkMetricContainer causes churn in the JobManager and lets the web frontend > malfunction > - > > Key: BEAM-8962 > URL: https://issues.apache.org/jira/browse/BEAM-8962 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.19.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > The {{FlinkMetricContainer}} wraps the Beam metric container for reporting > metrics, but also stores them as Flink accumulators. With high parallelism > jobs with over a thousand tasks and many built-in Beam metrics for every Beam > step, this can accumulate to over 100MB of serialized data which is stored in > the JobManager's ExecutionGraph. This then fails to even sent over the wire, > due to the akka.framesize limit (10MB by default), and manifests in {{500 > Internal Server Error}}s in the web frontend. > We need to introduce an option to disable the reporting via accumulators. It > is mostly useful for batch workloads where you can retrieve the final > accumulator values at the end of the job. It adds a lot of memory and network > overhead. > Perhaps we could even turn off the accumulators for streaming jobs, or > entirely and make them opt-in. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8962) FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction
[ https://issues.apache.org/jira/browse/BEAM-8962?focusedWorklogId=363269=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-363269 ] ASF GitHub Bot logged work on BEAM-8962: Author: ASF GitHub Bot Created on: 25/Dec/19 12:33 Start Date: 25/Dec/19 12:33 Worklog Time Spent: 10m Work Description: mxm commented on pull request #10415: [BEAM-8962] Report Flink metric accumulator only when pipeline ends URL: https://github.com/apache/beam/pull/10415#discussion_r361307846 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java ## @@ -227,12 +224,14 @@ public void open(Configuration parameters) { // deserialization method. However, this is a hack, and we want to properly initialize the // options where they are needed. FileSystems.setDefaultPipelineOptions(serializedOptions.get()); +metricContainer = new FlinkMetricContainer(getRuntimeContext()); doFnInvoker = DoFnInvokers.tryInvokeSetupFor(dofn); } @Override public void close() throws Exception { try { + metricContainer.registerMetricsForPipelineResult(); Review comment: Yes, I've checked the Flink code. Have a look at https://github.com/apache/flink/blob/16b635790c77c522a42bac2dbc98011b28f10096/flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java#L1461 When the task is removed, this will send a final accumulator snapshot. The metrics tests which rely on the MetricsPusher would also fail if that did not work correctly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 363269) Time Spent: 2h 10m (was: 2h) > FlinkMetricContainer causes churn in the JobManager and lets the web frontend > malfunction > - > > Key: BEAM-8962 > URL: https://issues.apache.org/jira/browse/BEAM-8962 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.19.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > The {{FlinkMetricContainer}} wraps the Beam metric container for reporting > metrics, but also stores them as Flink accumulators. With high parallelism > jobs with over a thousand tasks and many built-in Beam metrics for every Beam > step, this can accumulate to over 100MB of serialized data which is stored in > the JobManager's ExecutionGraph. This then fails to even sent over the wire, > due to the akka.framesize limit (10MB by default), and manifests in {{500 > Internal Server Error}}s in the web frontend. > We need to introduce an option to disable the reporting via accumulators. It > is mostly useful for batch workloads where you can retrieve the final > accumulator values at the end of the job. It adds a lot of memory and network > overhead. > Perhaps we could even turn off the accumulators for streaming jobs, or > entirely and make them opt-in. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8962) FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction
[ https://issues.apache.org/jira/browse/BEAM-8962?focusedWorklogId=362457=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362457 ] ASF GitHub Bot logged work on BEAM-8962: Author: ASF GitHub Bot Created on: 23/Dec/19 08:16 Start Date: 23/Dec/19 08:16 Worklog Time Spent: 10m Work Description: tweise commented on issue #10415: [BEAM-8962] Report Flink metric accumulator only when pipeline ends URL: https://github.com/apache/beam/pull/10415#issuecomment-568398151 A good improvement to avoid unnecessary overhead, especially for streaming mode. LGTM - see question above. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 362457) Time Spent: 2h (was: 1h 50m) > FlinkMetricContainer causes churn in the JobManager and lets the web frontend > malfunction > - > > Key: BEAM-8962 > URL: https://issues.apache.org/jira/browse/BEAM-8962 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.19.0 > > Time Spent: 2h > Remaining Estimate: 0h > > The {{FlinkMetricContainer}} wraps the Beam metric container for reporting > metrics, but also stores them as Flink accumulators. With high parallelism > jobs with over a thousand tasks and many built-in Beam metrics for every Beam > step, this can accumulate to over 100MB of serialized data which is stored in > the JobManager's ExecutionGraph. This then fails to even sent over the wire, > due to the akka.framesize limit (10MB by default), and manifests in {{500 > Internal Server Error}}s in the web frontend. > We need to introduce an option to disable the reporting via accumulators. It > is mostly useful for batch workloads where you can retrieve the final > accumulator values at the end of the job. It adds a lot of memory and network > overhead. > Perhaps we could even turn off the accumulators for streaming jobs, or > entirely and make them opt-in. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8962) FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction
[ https://issues.apache.org/jira/browse/BEAM-8962?focusedWorklogId=362436=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362436 ] ASF GitHub Bot logged work on BEAM-8962: Author: ASF GitHub Bot Created on: 23/Dec/19 06:51 Start Date: 23/Dec/19 06:51 Worklog Time Spent: 10m Work Description: tweise commented on pull request #10415: [BEAM-8962] Report Flink metric accumulator only when pipeline ends URL: https://github.com/apache/beam/pull/10415#discussion_r360783499 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java ## @@ -227,12 +224,14 @@ public void open(Configuration parameters) { // deserialization method. However, this is a hack, and we want to properly initialize the // options where they are needed. FileSystems.setDefaultPipelineOptions(serializedOptions.get()); +metricContainer = new FlinkMetricContainer(getRuntimeContext()); doFnInvoker = DoFnInvokers.tryInvokeSetupFor(dofn); } @Override public void close() throws Exception { try { + metricContainer.registerMetricsForPipelineResult(); Review comment: Is it guaranteed that the accumulators created in `close` will be captured in the terminated Flink job? Is it covered by existing tests? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 362436) Time Spent: 1h 50m (was: 1h 40m) > FlinkMetricContainer causes churn in the JobManager and lets the web frontend > malfunction > - > > Key: BEAM-8962 > URL: https://issues.apache.org/jira/browse/BEAM-8962 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.19.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > The {{FlinkMetricContainer}} wraps the Beam metric container for reporting > metrics, but also stores them as Flink accumulators. With high parallelism > jobs with over a thousand tasks and many built-in Beam metrics for every Beam > step, this can accumulate to over 100MB of serialized data which is stored in > the JobManager's ExecutionGraph. This then fails to even sent over the wire, > due to the akka.framesize limit (10MB by default), and manifests in {{500 > Internal Server Error}}s in the web frontend. > We need to introduce an option to disable the reporting via accumulators. It > is mostly useful for batch workloads where you can retrieve the final > accumulator values at the end of the job. It adds a lot of memory and network > overhead. > Perhaps we could even turn off the accumulators for streaming jobs, or > entirely and make them opt-in. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8962) FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction
[ https://issues.apache.org/jira/browse/BEAM-8962?focusedWorklogId=361381=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361381 ] ASF GitHub Bot logged work on BEAM-8962: Author: ASF GitHub Bot Created on: 18/Dec/19 16:24 Start Date: 18/Dec/19 16:24 Worklog Time Spent: 10m Work Description: mxm commented on pull request #10415: [BEAM-8962] Report Flink metric accumulator only when pipeline ends URL: https://github.com/apache/beam/pull/10415 To avoid the runtime overhead of continuously reporting accumulators from the TaskManagers to the JobManager, we can defer the reporting of the metrics accumulator until the pipeline shuts down. A final snapshot of the accumulators will be reported then. This means the metrics will still be available in the PipelineResult. Of course, the metrics reporting via Flink is not affected by this change. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [![Build
[jira] [Work logged] (BEAM-8962) FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction
[ https://issues.apache.org/jira/browse/BEAM-8962?focusedWorklogId=360358=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360358 ] ASF GitHub Bot logged work on BEAM-8962: Author: ASF GitHub Bot Created on: 16/Dec/19 16:35 Start Date: 16/Dec/19 16:35 Worklog Time Spent: 10m Work Description: mxm commented on issue #10381: [BEAM-8962] Add option to disable the metric container accumulator URL: https://github.com/apache/beam/pull/10381#issuecomment-566138675 I think it is worth to think about disabling the metric accumulator by default and only enable it via a `--enable_metric_accumulator`. The reason is that it provides very little value. The accumulator is used to aggregate the final metric values to write them to the configured Beam MetricSink. However, this is only done on job completion which makes this feature useless for streaming applications. Even for batch, you probably want to be able to see metrics during job execution which the accumulator does not provide. I'm inclined to remove the feature entirely. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360358) Time Spent: 1.5h (was: 1h 20m) > FlinkMetricContainer causes churn in the JobManager and lets the web frontend > malfunction > - > > Key: BEAM-8962 > URL: https://issues.apache.org/jira/browse/BEAM-8962 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.19.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > The {{FlinkMetricContainer}} wraps the Beam metric container for reporting > metrics, but also stores them as Flink accumulators. With high parallelism > jobs with over a thousand tasks and many built-in Beam metrics for every Beam > step, this can accumulate to over 100MB of serialized data which is stored in > the JobManager's ExecutionGraph. This then fails to even sent over the wire, > due to the akka.framesize limit (10MB by default), and manifests in {{500 > Internal Server Error}}s in the web frontend. > We need to introduce an option to disable the reporting via accumulators. It > is mostly useful for batch workloads where you can retrieve the final > accumulator values at the end of the job. It adds a lot of memory and network > overhead. > Perhaps we could even turn off the accumulators for streaming jobs, or > entirely and make them opt-in. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8962) FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction
[ https://issues.apache.org/jira/browse/BEAM-8962?focusedWorklogId=360353=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360353 ] ASF GitHub Bot logged work on BEAM-8962: Author: ASF GitHub Bot Created on: 16/Dec/19 16:29 Start Date: 16/Dec/19 16:29 Worklog Time Spent: 10m Work Description: mxm commented on pull request #10381: [BEAM-8962] Add option to disable the metric container accumulator URL: https://github.com/apache/beam/pull/10381 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360353) Time Spent: 1h 20m (was: 1h 10m) > FlinkMetricContainer causes churn in the JobManager and lets the web frontend > malfunction > - > > Key: BEAM-8962 > URL: https://issues.apache.org/jira/browse/BEAM-8962 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.19.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The {{FlinkMetricContainer}} wraps the Beam metric container for reporting > metrics, but also stores them as Flink accumulators. With high parallelism > jobs with over a thousand tasks and many built-in Beam metrics for every Beam > step, this can accumulate to over 100MB of serialized data which is stored in > the JobManager's ExecutionGraph. This then fails to even sent over the wire, > due to the akka.framesize limit (10MB by default), and manifests in {{500 > Internal Server Error}}s in the web frontend. > We need to introduce an option to disable the reporting via accumulators. It > is mostly useful for batch workloads where you can retrieve the final > accumulator values at the end of the job. It adds a lot of memory and network > overhead. > Perhaps we could even turn off the accumulators for streaming jobs, or > entirely and make them opt-in. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8962) FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction
[ https://issues.apache.org/jira/browse/BEAM-8962?focusedWorklogId=360298=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360298 ] ASF GitHub Bot logged work on BEAM-8962: Author: ASF GitHub Bot Created on: 16/Dec/19 14:31 Start Date: 16/Dec/19 14:31 Worklog Time Spent: 10m Work Description: tweise commented on pull request #10381: [BEAM-8962] Add option to disable the metric container accumulator URL: https://github.com/apache/beam/pull/10381#discussion_r358263908 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java ## @@ -159,6 +159,15 @@ void setEnableMetrics(Boolean enableMetrics); + @Description( + "By default, uses Flink accumulators to store the metrics which allows to query metrics from the PipelineResult. " + + "If set to true, metrics will still be reported but can't be queried via PipelineResult. " + + "This saves network and memory.") + @Default.Boolean(false) Review comment: I find it important that flags are named to express their semantics. In this case, the flag's purpose is to turn something off, and there are many similar examples where we would use "skip", "no" etc. There should not be a case where `disabled=false` needs to be specified by the user. But maybe flags should not have default values, since they are fully described by name. That would also avoid the options to be included with their defaults during job submission, even when the user has not specified them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360298) Time Spent: 1h 10m (was: 1h) > FlinkMetricContainer causes churn in the JobManager and lets the web frontend > malfunction > - > > Key: BEAM-8962 > URL: https://issues.apache.org/jira/browse/BEAM-8962 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.19.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The {{FlinkMetricContainer}} wraps the Beam metric container for reporting > metrics, but also stores them as Flink accumulators. With high parallelism > jobs with over a thousand tasks and many built-in Beam metrics for every Beam > step, this can accumulate to over 100MB of serialized data which is stored in > the JobManager's ExecutionGraph. This then fails to even sent over the wire, > due to the akka.framesize limit (10MB by default), and manifests in {{500 > Internal Server Error}}s in the web frontend. > We need to introduce an option to disable the reporting via accumulators. It > is mostly useful for batch workloads where you can retrieve the final > accumulator values at the end of the job. It adds a lot of memory and network > overhead. > Perhaps we could even turn off the accumulators for streaming jobs, or > entirely and make them opt-in. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8962) FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction
[ https://issues.apache.org/jira/browse/BEAM-8962?focusedWorklogId=359972=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-359972 ] ASF GitHub Bot logged work on BEAM-8962: Author: ASF GitHub Bot Created on: 15/Dec/19 15:55 Start Date: 15/Dec/19 15:55 Worklog Time Spent: 10m Work Description: mxm commented on pull request #10381: [BEAM-8962] Add option to disable the metric container accumulator URL: https://github.com/apache/beam/pull/10381#discussion_r357987842 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java ## @@ -62,22 +67,27 @@ private final Map flinkGaugeCache; private final MetricsAccumulator metricsAccumulator; - public FlinkMetricContainer(RuntimeContext runtimeContext) { + public FlinkMetricContainer(RuntimeContext runtimeContext, boolean accumulatorDisabled) { this.runtimeContext = runtimeContext; this.flinkCounterCache = new HashMap<>(); this.flinkDistributionGaugeCache = new HashMap<>(); this.flinkGaugeCache = new HashMap<>(); -Accumulator metricsAccumulator = -runtimeContext.getAccumulator(ACCUMULATOR_NAME); -if (metricsAccumulator == null) { +Accumulator metricsAccumulator; +if (accumulatorDisabled) { + // Do not register the accumulator with Flink metricsAccumulator = new MetricsAccumulator(); Review comment: Yes, it holds all local metrics. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 359972) Time Spent: 1h (was: 50m) > FlinkMetricContainer causes churn in the JobManager and lets the web frontend > malfunction > - > > Key: BEAM-8962 > URL: https://issues.apache.org/jira/browse/BEAM-8962 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.19.0 > > Time Spent: 1h > Remaining Estimate: 0h > > The {{FlinkMetricContainer}} wraps the Beam metric container for reporting > metrics, but also stores them as Flink accumulators. With high parallelism > jobs with over a thousand tasks and many built-in Beam metrics for every Beam > step, this can accumulate to over 100MB of serialized data which is stored in > the JobManager's ExecutionGraph. This then fails to even sent over the wire, > due to the akka.framesize limit (10MB by default), and manifests in {{500 > Internal Server Error}}s in the web frontend. > We need to introduce an option to disable the reporting via accumulators. It > is mostly useful for batch workloads where you can retrieve the final > accumulator values at the end of the job. It adds a lot of memory and network > overhead. > Perhaps we could even turn off the accumulators for streaming jobs, or > entirely and make them opt-in. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8962) FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction
[ https://issues.apache.org/jira/browse/BEAM-8962?focusedWorklogId=359971=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-359971 ] ASF GitHub Bot logged work on BEAM-8962: Author: ASF GitHub Bot Created on: 15/Dec/19 15:54 Start Date: 15/Dec/19 15:54 Worklog Time Spent: 10m Work Description: mxm commented on pull request #10381: [BEAM-8962] Add option to disable the metric container accumulator URL: https://github.com/apache/beam/pull/10381#discussion_r357987764 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java ## @@ -159,6 +159,15 @@ void setEnableMetrics(Boolean enableMetrics); + @Description( + "By default, uses Flink accumulators to store the metrics which allows to query metrics from the PipelineResult. " + + "If set to true, metrics will still be reported but can't be queried via PipelineResult. " + + "This saves network and memory.") + @Default.Boolean(false) Review comment: Unfortunately, we can't set true-defaulting options to false. That's why I inverted the flag's name. See https://issues.apache.org/jira/browse/BEAM-8959 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 359971) Time Spent: 50m (was: 40m) > FlinkMetricContainer causes churn in the JobManager and lets the web frontend > malfunction > - > > Key: BEAM-8962 > URL: https://issues.apache.org/jira/browse/BEAM-8962 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.19.0 > > Time Spent: 50m > Remaining Estimate: 0h > > The {{FlinkMetricContainer}} wraps the Beam metric container for reporting > metrics, but also stores them as Flink accumulators. With high parallelism > jobs with over a thousand tasks and many built-in Beam metrics for every Beam > step, this can accumulate to over 100MB of serialized data which is stored in > the JobManager's ExecutionGraph. This then fails to even sent over the wire, > due to the akka.framesize limit (10MB by default), and manifests in {{500 > Internal Server Error}}s in the web frontend. > We need to introduce an option to disable the reporting via accumulators. It > is mostly useful for batch workloads where you can retrieve the final > accumulator values at the end of the job. It adds a lot of memory and network > overhead. > Perhaps we could even turn off the accumulators for streaming jobs, or > entirely and make them opt-in. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8962) FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction
[ https://issues.apache.org/jira/browse/BEAM-8962?focusedWorklogId=359898=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-359898 ] ASF GitHub Bot logged work on BEAM-8962: Author: ASF GitHub Bot Created on: 15/Dec/19 01:21 Start Date: 15/Dec/19 01:21 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #10381: [BEAM-8962] Add option to disable the metric container accumulator URL: https://github.com/apache/beam/pull/10381#discussion_r357947246 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java ## @@ -159,6 +159,15 @@ void setEnableMetrics(Boolean enableMetrics); + @Description( + "By default, uses Flink accumulators to store the metrics which allows to query metrics from the PipelineResult. " + + "If set to true, metrics will still be reported but can't be queried via PipelineResult. " Review comment: ```suggestion + "If set to true, metrics will still be reported to the configured metrics reporter but can't be queried via PipelineResult. " ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 359898) Time Spent: 0.5h (was: 20m) > FlinkMetricContainer causes churn in the JobManager and lets the web frontend > malfunction > - > > Key: BEAM-8962 > URL: https://issues.apache.org/jira/browse/BEAM-8962 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.19.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The {{FlinkMetricContainer}} wraps the Beam metric container for reporting > metrics, but also stores them as Flink accumulators. With high parallelism > jobs with over a thousand tasks and many built-in Beam metrics for every Beam > step, this can accumulate to over 100MB of serialized data which is stored in > the JobManager's ExecutionGraph. This then fails to even sent over the wire, > due to the akka.framesize limit (10MB by default), and manifests in {{500 > Internal Server Error}}s in the web frontend. > We need to introduce an option to disable the reporting via accumulators. It > is mostly useful for batch workloads where you can retrieve the final > accumulator values at the end of the job. It adds a lot of memory and network > overhead. > Perhaps we could even turn off the accumulators for streaming jobs, or > entirely and make them opt-in. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8962) FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction
[ https://issues.apache.org/jira/browse/BEAM-8962?focusedWorklogId=359899=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-359899 ] ASF GitHub Bot logged work on BEAM-8962: Author: ASF GitHub Bot Created on: 15/Dec/19 01:21 Start Date: 15/Dec/19 01:21 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #10381: [BEAM-8962] Add option to disable the metric container accumulator URL: https://github.com/apache/beam/pull/10381#discussion_r357947178 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java ## @@ -62,22 +67,27 @@ private final Map flinkGaugeCache; private final MetricsAccumulator metricsAccumulator; - public FlinkMetricContainer(RuntimeContext runtimeContext) { + public FlinkMetricContainer(RuntimeContext runtimeContext, boolean accumulatorDisabled) { this.runtimeContext = runtimeContext; this.flinkCounterCache = new HashMap<>(); this.flinkDistributionGaugeCache = new HashMap<>(); this.flinkGaugeCache = new HashMap<>(); -Accumulator metricsAccumulator = -runtimeContext.getAccumulator(ACCUMULATOR_NAME); -if (metricsAccumulator == null) { +Accumulator metricsAccumulator; +if (accumulatorDisabled) { + // Do not register the accumulator with Flink metricsAccumulator = new MetricsAccumulator(); Review comment: Is this accumulator still doing anything if it's not registered? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 359899) Time Spent: 0.5h (was: 20m) > FlinkMetricContainer causes churn in the JobManager and lets the web frontend > malfunction > - > > Key: BEAM-8962 > URL: https://issues.apache.org/jira/browse/BEAM-8962 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.19.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The {{FlinkMetricContainer}} wraps the Beam metric container for reporting > metrics, but also stores them as Flink accumulators. With high parallelism > jobs with over a thousand tasks and many built-in Beam metrics for every Beam > step, this can accumulate to over 100MB of serialized data which is stored in > the JobManager's ExecutionGraph. This then fails to even sent over the wire, > due to the akka.framesize limit (10MB by default), and manifests in {{500 > Internal Server Error}}s in the web frontend. > We need to introduce an option to disable the reporting via accumulators. It > is mostly useful for batch workloads where you can retrieve the final > accumulator values at the end of the job. It adds a lot of memory and network > overhead. > Perhaps we could even turn off the accumulators for streaming jobs, or > entirely and make them opt-in. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8962) FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction
[ https://issues.apache.org/jira/browse/BEAM-8962?focusedWorklogId=359493=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-359493 ] ASF GitHub Bot logged work on BEAM-8962: Author: ASF GitHub Bot Created on: 13/Dec/19 16:47 Start Date: 13/Dec/19 16:47 Worklog Time Spent: 10m Work Description: asf-ci commented on issue #10381: [BEAM-8962] Add option to disable the metric container accumulator URL: https://github.com/apache/beam/pull/10381#issuecomment-565513340 SUCCESS --none-- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 359493) Time Spent: 20m (was: 10m) > FlinkMetricContainer causes churn in the JobManager and lets the web frontend > malfunction > - > > Key: BEAM-8962 > URL: https://issues.apache.org/jira/browse/BEAM-8962 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.19.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The {{FlinkMetricContainer}} wraps the Beam metric container for reporting > metrics, but also stores them as Flink accumulators. With high parallelism > jobs with over a thousand tasks and many built-in Beam metrics for every Beam > step, this can accumulate to over 100MB of serialized data which is stored in > the JobManager's ExecutionGraph. This then fails to even sent over the wire, > due to the akka.framesize limit (10MB by default), and manifests in {{500 > Internal Server Error}}s in the web frontend. > We need to introduce an option to disable the reporting via accumulators. It > is mostly useful for batch workloads where you can retrieve the final > accumulator values at the end of the job. It adds a lot of memory and network > overhead. > Perhaps we could even turn off the accumulators for streaming jobs, or > entirely and make them opt-in. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8962) FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction
[ https://issues.apache.org/jira/browse/BEAM-8962?focusedWorklogId=359480=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-359480 ] ASF GitHub Bot logged work on BEAM-8962: Author: ASF GitHub Bot Created on: 13/Dec/19 16:12 Start Date: 13/Dec/19 16:12 Worklog Time Spent: 10m Work Description: mxm commented on pull request #10381: [BEAM-8962] Add option to disable the metric container accumulator URL: https://github.com/apache/beam/pull/10381 The FlinkMetricContainer wraps the Beam metric container for reporting metrics, but also stores them as Flink accumulators. With high parallelism jobs with over a thousand tasks and many built-in Beam metrics for every Beam step, this can accumulate to over 100MB of serialized data which is stored in the JobManager's ExecutionGraph. This then fails to even sent over the wire, due to the akka.framesize limit (10MB by default), and manifests in {{500 Internal Server Error}}s in the web frontend. This introduces an option to disable the reporting via accumulators. The accumulator is mostly useful for batch workloads where you can retrieve the final accumulator values at the end of the job. It adds a lot of memory and network overhead. In the future, we might disable it by default on streaming workloads. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build