Gengliang Wang created SPARK-61903:
--------------------------------------
Summary: Support custom metrics for native data sources
Key: SPARK-61903
URL: https://issues.apache.org/jira/browse/SPARK-61903
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Gengliang Wang
A native data source cannot report metrics of its own, such as the number of
files or row groups that it skipped with the pushed domains, or the bytes that
it read. Data Source V2 shows such metrics in the SQL tab of the Spark UI
through {{CustomMetric}} and {{CustomTaskMetric}}, which the scans and the
writes of native data sources do not implement.
Proposal: let a data source declare its metrics, and let the reads and the
writes of its tasks report their values, with functions appended to the structs
of the interface ({{spark_datasource.h}}) within version 1. Then expose them in
the Rust SDK (SPARK-61282).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]