@StephanEwen for the context please see the diagram above: basically Flink accumulators are used to store `MetricsContainersStepMaps` that gather beam metrics for the pipeline. When Flink merges these accumulators, Beam metrics get merged. All the point of this discussion is the fact that these flink accumulators are not available to the user in detached mode and thus Beam methods that request Beam metrics do not work.
That being said, Beam Flink runner also updates internal Flink metrics via the DoFnRunner: see https://github.com/apache/beam/blob/0299676889dbc2db7ebb40db4acd27189735b75e/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/DoFnRunnerWithMetricsUpdate.java#L93 and https://github.com/apache/beam/blob/0299676889dbc2db7ebb40db4acd27189735b75e/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java#L101 So I guess we can have access to Beam metrics from the native flink metrics system in the Flink UI (not Beam `PipelineResult.metrics()` or `MetricsPusher`) even in detached mode. Alternatively @aljoscha suggests to set up a Metrics reporter (https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/metrics.html) and Beam metrics will then be exported like Flink metrics to that reporter. @JozoVilcek As I don't know native Flink well I cannot provide more details but it may be redundant with the runner updating flink metrics (see links above). [ Full content available at: https://github.com/apache/beam/pull/4548 ] This message was relayed via gitbox.apache.org for [email protected]
