@aljoscha Would like to respond to your email here to connect contexts:
> the accumulators not being available is an artifact of how detached mode is > implemented. Basically, we fire of the job and then forget everything about > it. That's why there is no way of getting any data about the job. Regarding Metrics, with Flink you can set up a Metrics reporter (https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/metrics.html) and Beam metrics will then be exported like Flink metrics to that reporter. >Best, Aljoscha I understand that we forget about the job once we fire it in detached mode and also that the Beam metrics are available through a Metrics reporter from Flink. Now my understanding was that these metrics from Beam need to be accumulated by Flink first in order to report from across task managers etc. -> whether through metrics reporter or Flink UI. This is what @StephanEwen assumed too. My question is simple -> if Flink is actually accumulating metrics, why is it not shown in Flink UI for Beam jobs under "Accumulators"? Is this a bug or this support is not there from Flink? If support is not present then how are the metrics reported/accumulated through Metrics reporter? [ Full content available at: https://github.com/apache/beam/pull/4548 ] This message was relayed via gitbox.apache.org for [email protected]
