Gengliang Wang created SPARK-61898:
--------------------------------------
Summary: Report the statistics of the scans of native data sources
Key: SPARK-61898
URL: https://issues.apache.org/jira/browse/SPARK-61898
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Gengliang Wang
Version 1 of the native data source interface ({{spark_datasource.h}},
SPARK-60101) has no way for a library to report the size of a scan. Spark
estimates a native scan with {{spark.sql.defaultSizeInBytes}}, so it does not
plan one as the broadcast side of a join unless the query has a broadcast hint.
Adaptive query execution can only broadcast it at run time, from the size of a
shuffle of the scan.
Proposal: let a reader report the statistics of its scan after the pushdown, at
least its size in bytes and its number of rows, with a function appended to
{{SparkDsLibrary}} within version 1. The scan of Spark returns them through
{{SupportsReportStatistics}}. Then expose it in the Rust SDK (SPARK-61282).
This matters most in queries with joins. For example, the Parquet source of the
Rust SDK examples (https://github.com/apache/spark/pull/59692) reports no
statistics, so Spark does not plan to broadcast its tables.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]