Gengliang Wang created SPARK-61899:
--------------------------------------

             Summary: Support runtime filtering for the scans of native data 
sources
                 Key: SPARK-61899
                 URL: https://issues.apache.org/jira/browse/SPARK-61899
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Gengliang Wang


Spark applies no runtime filter to the scans of native data sources: the scan 
does not implement {{SupportsRuntimeV2Filtering}}, so the filters that Spark 
derives from the other side of a join at run time, as in dynamic partition 
pruning, do not reach a library, which reads all the data that the filters of 
the query itself allow.

Proposal: let a data source declare the columns that it can filter at run time, 
and pass the values of a runtime filter to the library as domains 
(SPARK-61200), so that it can drop the partitions that cannot match or skip 
data within them. This can be added within version 1 of the interface 
({{spark_datasource.h}}), as functions or fields appended to its structs. Then 
expose it in the Rust SDK (SPARK-61282).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to