zzzzming95 created SPARK-43866:
----------------------------------
Summary: Partition filter condition should pushed down to
metastore query if it is equivalence Predicate
Key: SPARK-43866
URL: https://issues.apache.org/jira/browse/SPARK-43866
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.4.0
Reporter: zzzzming95
Typically, hive partition fields are created as string types.
```
CREATE TABLE if not exists test_tb (
id int
)
PARTITIONED BY (dt string)
```
However, cast data conversions are often introduced inadvertently during use.
For example
```
select * from test_tb where dt=20230505;
```
it will prevent the condition `dt=20230505` from being pushed down into the
metastore , because `20230505` is an IntegralType. And resulting in a request
for all partitions. However, in the case of equivalent predicates, partition
field pushdown should be supported.
This can affect execution performance in cases where the data table has very
many partitions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
