Gengliang Wang created SPARK-61899:
--------------------------------------
Summary: Support runtime filtering for the scans of native data
sources
Key: SPARK-61899
URL: https://issues.apache.org/jira/browse/SPARK-61899
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Gengliang Wang
Spark applies no runtime filter to the scans of native data sources: the scan
does not implement {{SupportsRuntimeV2Filtering}}, so the filters that Spark
derives from the other side of a join at run time, as in dynamic partition
pruning, do not reach a library, which reads all the data that the filters of
the query itself allow.
Proposal: let a data source declare the columns that it can filter at run time,
and pass the values of a runtime filter to the library as domains
(SPARK-61200), so that it can drop the partitions that cannot match or skip
data within them. This can be added within version 1 of the interface
({{spark_datasource.h}}), as functions or fields appended to its structs. Then
expose it in the Rust SDK (SPARK-61282).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]