Parth Chandra created SPARK-47290:
-------------------------------------
Summary: Extend CustomTaskMetric to allow metric values from
multiple sources
Key: SPARK-47290
URL: https://issues.apache.org/jira/browse/SPARK-47290
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.0.0
Reporter: Parth Chandra
Custom task metrics allow a DSV2 to add metrics that can be displayed in the
UI. However, for DSV2 file sources, the FilePartitionReader may have multiple
file readers, and each of these may report their metrics which need to be
aggregated and bubbled up to the Scan. There is no way currently to update a
metric value.
A new interface that extends CustomTaskMetric and defines a way to allow
updates would allow a DSV2 file scan implementation to over come this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
