Beam metrics are available through Flink UI and also being pushed to `MetricReporter`. But Flink and Beam world of metrics are sort of disconnected for the user. In Beam, metric is defined by `(metricType: [Counter, Distribution, Gauge], namespace: String, name: String)` maybe also `stepName: String` but that one is kind of internal to beam metrics system.
In flink `MetricReporter` world, user gets notification about added metrics he can react to. They have form of `(metricType: Metric, metricName: String, group: MetricGroup)` . This form us much richer and contains some extra dimensions to beam model within `MetricGroup`. One have to be very much aware to internals, to be able to work with this. Metric pusher (and whole runner agnostic metrics idea) is currently the only easy way how to conveniently retrieve aggregated metrics from Beam definition domain. But, this feature is not always available, not in jobs launched in detached mode. Therefore user have now only the option of implementing custom Flink MetricReporter and extract metrics from there (requires a lot of internal insight). The question is, how can this be improved and have a convenience way of reporting beam metrics also in detached jobs. I hope I do make sense and am not confusing things too much [ Full content available at: https://github.com/apache/beam/pull/4548 ] This message was relayed via gitbox.apache.org for [email protected]
