Parth Chandra created SPARK-47290: ------------------------------------- Summary: Extend CustomTaskMetric to allow metric values from multiple sources Key: SPARK-47290 URL: https://issues.apache.org/jira/browse/SPARK-47290 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Parth Chandra
Custom task metrics allow a DSV2 to add metrics that can be displayed in the UI. However, for DSV2 file sources, the FilePartitionReader may have multiple file readers, and each of these may report their metrics which need to be aggregated and bubbled up to the Scan. There is no way currently to update a metric value. A new interface that extends CustomTaskMetric and defines a way to allow updates would allow a DSV2 file scan implementation to over come this. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org