[
https://issues.apache.org/jira/browse/SPARK-43866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zzzzming95 updated SPARK-43866:
-------------------------------
Description:
Typically, hive partition fields are created as string types.
{code:java}
```
CREATE TABLE if not exists test_tb (
id int
)
PARTITIONED BY (dt string)
```{code}
However, cast data conversions are often introduced inadvertently during use.
For example
{code:java}
```
select * from test_tb where dt=20230505;
```{code}
it will prevent the condition `dt=20230505` from being pushed down into the
metastore , because `20230505` is an IntegralType. And resulting in a request
for all partitions. However, in the case of equivalent predicates, partition
field pushdown should be supported.
This can affect execution performance in cases where the data table has very
many partitions.
was:
Typically, hive partition fields are created as string types.
```
CREATE TABLE if not exists test_tb (
id int
)
PARTITIONED BY (dt string)
```
However, cast data conversions are often introduced inadvertently during use.
For example
```
select * from test_tb where dt=20230505;
```
it will prevent the condition `dt=20230505` from being pushed down into the
metastore , because `20230505` is an IntegralType. And resulting in a request
for all partitions. However, in the case of equivalent predicates, partition
field pushdown should be supported.
This can affect execution performance in cases where the data table has very
many partitions.
> Partition filter condition should pushed down to metastore query if it is
> equivalence Predicate
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-43866
> URL: https://issues.apache.org/jira/browse/SPARK-43866
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: zzzzming95
> Priority: Major
>
> Typically, hive partition fields are created as string types.
> {code:java}
> ```
> CREATE TABLE if not exists test_tb (
> id int
> )
> PARTITIONED BY (dt string)
> ```{code}
> However, cast data conversions are often introduced inadvertently during use.
> For example
> {code:java}
> ```
> select * from test_tb where dt=20230505;
> ```{code}
> it will prevent the condition `dt=20230505` from being pushed down into the
> metastore , because `20230505` is an IntegralType. And resulting in a request
> for all partitions. However, in the case of equivalent predicates, partition
> field pushdown should be supported.
> This can affect execution performance in cases where the data table has very
> many partitions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
