Gengliang Wang created SPARK-61901:
--------------------------------------
Summary: Let native data sources evaluate filters, so that Spark
does not evaluate them again
Key: SPARK-61901
URL: https://issues.apache.org/jira/browse/SPARK-61901
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Gengliang Wang
Spark evaluates all the filters of a native scan again after reading. A library
only gets the domains of the columns that the filters constrain (SPARK-61200),
with which it can skip data, but it cannot take a filter over. So:
* Spark does not push a {{LIMIT}} down below a filter;
* a library cannot evaluate the filters from which Spark derives no domain,
such as {{ENDS_WITH}}, a comparison of two columns, or an {{OR}} that
constrains a different column on each side;
* the rows that a library already filtered are filtered twice.
Proposal: let a library tell Spark which filters it evaluates exactly, so that
Spark does not evaluate them again and can push a limit down with them. This
needs either an answer by which a library declares that it only returns the
rows within a domain, or a representation of filters in the interface
({{spark_datasource.h}}) beyond domains. Both can be added within version 1, as
functions or fields appended to its structs. Then expose it in the Rust SDK
(SPARK-61282).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]