[
https://issues.apache.org/jira/browse/SPARK-43866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuming Wang resolved SPARK-43866.
---------------------------------
Resolution: Workaround
> Partition filter condition should pushed down to metastore query if it is
> equivalence Predicate
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-43866
> URL: https://issues.apache.org/jira/browse/SPARK-43866
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: zzzzming95
> Priority: Major
>
> Typically, hive partition fields are created as string types.
> {code:java}
> ```
> CREATE TABLE if not exists test_tb (
> id int
> )
> PARTITIONED BY (dt string)
> ```{code}
> However, cast data conversions are often introduced inadvertently during use.
> For example
> {code:java}
> ```
> select * from test_tb where dt=20230505;
> ```{code}
> it will prevent the condition `dt=20230505` from being pushed down into the
> metastore , because `20230505` is an IntegralType. And resulting in a request
> for all partitions. However, in the case of equivalent predicates, partition
> field pushdown should be supported.
> This can affect execution performance in cases where the data table has very
> many partitions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
