[
https://issues.apache.org/jira/browse/BEAM-11213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300477#comment-17300477
]
Kenneth Knowles commented on BEAM-11213:
----------------------------------------
I see there are some PRs attached to this. Is it resolved? I have removed the
Fix Version because it does not seem like it is a critical regression.
> Beam metrics should be displayed in Spark UI
> --------------------------------------------
>
> Key: BEAM-11213
> URL: https://issues.apache.org/jira/browse/BEAM-11213
> Project: Beam
> Issue Type: Wish
> Components: runner-spark
> Reporter: Kyle Weaver
> Assignee: Tomasz Szerszen
> Priority: P2
> Labels: portability-spark
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> All Beam metrics are visible in the Spark UI in a single accumulator value
> (in the "Accumulators" tab), which is a large, hard-to-read blob. Originally,
> this blob was rendered in a bespoke format
> (https://github.com/apache/beam/blob/ead80b469ffeeddcd8e9e5c8dc462eec0b0ffc6b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricQueryResults.java#L63-L72).
> I changed the format to JSON so it could be easily deserialized (BEAM-9600).
> But then an issue was filed (BEAM-10294) reporting that the new JSON format
> was harder to read than the original bespoke format. The temporary fix was to
> revert to the bespoke format in Spark, while allowing Flink to continue to
> use JSON. However, if Beam metrics are only visible as an accumulator, then
> they are also unreadable because the payloads are in binary form (BEAM-10719).
> Having metrics visible in Spark's "Metrics" tab would A) make metrics easier
> to read (even compared to the bespoke accumulator string format), and closer
> to what users of Beamless Spark expect, and B) free us to use the accumulator
> however we wish for Beam internal purposes, without worrying about
> readability.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
