Gengliang Wang created SPARK-61898:
--------------------------------------

             Summary: Report the statistics of the scans of native data sources
                 Key: SPARK-61898
                 URL: https://issues.apache.org/jira/browse/SPARK-61898
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Gengliang Wang


Version 1 of the native data source interface ({{spark_datasource.h}}, 
SPARK-60101) has no way for a library to report the size of a scan. Spark 
estimates a native scan with {{spark.sql.defaultSizeInBytes}}, so it does not 
plan one as the broadcast side of a join unless the query has a broadcast hint. 
Adaptive query execution can only broadcast it at run time, from the size of a 
shuffle of the scan.

Proposal: let a reader report the statistics of its scan after the pushdown, at 
least its size in bytes and its number of rows, with a function appended to 
{{SparkDsLibrary}} within version 1. The scan of Spark returns them through 
{{SupportsReportStatistics}}. Then expose it in the Rust SDK (SPARK-61282).

This matters most in queries with joins. For example, the Parquet source of the 
Rust SDK examples (https://github.com/apache/spark/pull/59692) reports no 
statistics, so Spark does not plan to broadcast its tables.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to