Szehon Ho created SPARK-53928:
---------------------------------
Summary: Enhance DSV2 partition filtering using catalyst expression
Key: SPARK-53928
URL: https://issues.apache.org/jira/browse/SPARK-53928
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: Szehon Ho
Currently, Spark converts Catalyst Expression to either Filter or Predicate and
pushes it to DSV2 via SupportsPushdownFilters and SupportsPushdownV2Filters
API's.
However, some Spark filters may not convert cleanly. For example,
trim(part_col) = 'a'. There are cases where DSV2 can return the exact
partition value(s) to spark for its InputPartition, and Spark can use the
original catalyst expression for filtering.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
