Gengliang Wang created SPARK-61903:
--------------------------------------

             Summary: Support custom metrics for native data sources
                 Key: SPARK-61903
                 URL: https://issues.apache.org/jira/browse/SPARK-61903
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Gengliang Wang


A native data source cannot report metrics of its own, such as the number of 
files or row groups that it skipped with the pushed domains, or the bytes that 
it read. Data Source V2 shows such metrics in the SQL tab of the Spark UI 
through {{CustomMetric}} and {{CustomTaskMetric}}, which the scans and the 
writes of native data sources do not implement.

Proposal: let a data source declare its metrics, and let the reads and the 
writes of its tasks report their values, with functions appended to the structs 
of the interface ({{spark_datasource.h}}) within version 1. Then expose them in 
the Rust SDK (SPARK-61282).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to