[
https://issues.apache.org/jira/browse/BEAM-11212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janek Bevendorff updated BEAM-11212:
------------------------------------
Priority: P1 (was: P3)
> Beam metrics should be displayed in Flink UI "Metrics" tab
> ----------------------------------------------------------
>
> Key: BEAM-11212
> URL: https://issues.apache.org/jira/browse/BEAM-11212
> Project: Beam
> Issue Type: Wish
> Components: runner-flink
> Reporter: Kyle Weaver
> Priority: P1
> Labels: portability-flink
> Attachments: Screenshot 2021-01-20 at 12.07.33.png, Screenshot
> 2021-01-20 at 12.08.44.png, flink-metrics.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> All Beam metrics are visible in the Flink UI in a single accumulator value
> (in the "Accumulators" tab), which is a large, hard-to-read blob. Originally,
> this blob was rendered in a bespoke format
> (https://github.com/apache/beam/blob/ead80b469ffeeddcd8e9e5c8dc462eec0b0ffc6b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricQueryResults.java#L63-L72).
> I changed the format to JSON so it could be easily deserialized (BEAM-9600).
> But then an issue was filed (BEAM-10294) reporting that the new JSON format
> was harder to read than the original bespoke format. The temporary fix was to
> revert to the bespoke format in Spark, while allowing Flink to continue to
> use JSON. However, if Beam metrics in Flink are only visible as an
> accumulator, then they are also unreadable because the payloads are in binary
> form (BEAM-10719).
> Having metrics visible in Flink's "Metrics" tab would A) make metrics easier
> to read (even compared to the bespoke accumulator string format), and closer
> to what users of Beamless Flink expect, and B) free us to use the accumulator
> however we wish for Beam internal purposes, without worrying about
> readability.
> One question I'm not sure about is, why can't we see Beam metrics in the
> Flink UI already? I thought we were already translating Beam metrics into
> Flink native metrics
> (https://github.com/apache/beam/blob/ea2a3f6896b66a2852a2ff3d82f4e1b010013d13/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java#L108-L109).
> Is there something else we need to do to display them in the UI?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
