Gengliang Wang created SPARK-61901:
--------------------------------------

             Summary: Let native data sources evaluate filters, so that Spark 
does not evaluate them again
                 Key: SPARK-61901
                 URL: https://issues.apache.org/jira/browse/SPARK-61901
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Gengliang Wang


Spark evaluates all the filters of a native scan again after reading. A library 
only gets the domains of the columns that the filters constrain (SPARK-61200), 
with which it can skip data, but it cannot take a filter over. So:
* Spark does not push a {{LIMIT}} down below a filter;
* a library cannot evaluate the filters from which Spark derives no domain, 
such as {{ENDS_WITH}}, a comparison of two columns, or an {{OR}} that 
constrains a different column on each side;
* the rows that a library already filtered are filtered twice.

Proposal: let a library tell Spark which filters it evaluates exactly, so that 
Spark does not evaluate them again and can push a limit down with them. This 
needs either an answer by which a library declares that it only returns the 
rows within a domain, or a representation of filters in the interface 
({{spark_datasource.h}}) beyond domains. Both can be added within version 1, as 
functions or fields appended to its structs. Then expose it in the Rust SDK 
(SPARK-61282).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to