[
https://issues.apache.org/jira/browse/SPARK-45199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated SPARK-45199:
---------------------------
Description:
When a {{cast clause}} was used in a column in a filter, it will not be able to
push down. We can remove the cast from attribute side to support predicate push
down if the cast not change the precision or range.
Test query:
{code:sql}
-- dt is string type
explain select * from wechar_tbl where cast(dt as date) =
date_sub(current_date(), 1);
{code}
Before this patch:
{code:bash}
== Physical Plan ==
*(1) ColumnarToRow
+- FileScan parquet default.wechar_tbl[id#5,name#6,dt#7] Batched: true,
DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(0 paths)[],
PartitionFilters: [isnotnull(dt#7), (cast(dt#7 as date) = 2023-09-17)],
PushedFilters: [], ReadSchema: struct<id:int,name:string>
{code}
*cast(dt#7 as date) = 2023-09-17 can not be push down in partition filter*
After this patch:
{code:bash}
== Physical Plan ==
*(1) ColumnarToRow
+- FileScan parquet default.wechar_tbl[id#62,name#63,dt#64] Batched: true,
DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[],
PartitionFilters: [isnotnull(dt#64), (dt#64 = 2023-09-17)], PushedFilters: [],
ReadSchema: struct<id:int,name:string>
{code}
*(dt#64 = 2023-09-17) can be push down in partition filter*
was:When a {{cast clause}} was used in a column in a filter, it will not be
able to push down. We can remove the cast from attribute side to support
predicate push down if the cast not change the precision or range.
> Release cast from attribute in filter to support predicate push down
> --------------------------------------------------------------------
>
> Key: SPARK-45199
> URL: https://issues.apache.org/jira/browse/SPARK-45199
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.5.0
> Reporter: Wechar
> Priority: Major
>
> When a {{cast clause}} was used in a column in a filter, it will not be able
> to push down. We can remove the cast from attribute side to support predicate
> push down if the cast not change the precision or range.
> Test query:
> {code:sql}
> -- dt is string type
> explain select * from wechar_tbl where cast(dt as date) =
> date_sub(current_date(), 1);
> {code}
> Before this patch:
> {code:bash}
> == Physical Plan ==
> *(1) ColumnarToRow
> +- FileScan parquet default.wechar_tbl[id#5,name#6,dt#7] Batched: true,
> DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(0 paths)[],
> PartitionFilters: [isnotnull(dt#7), (cast(dt#7 as date) = 2023-09-17)],
> PushedFilters: [], ReadSchema: struct<id:int,name:string>
> {code}
> *cast(dt#7 as date) = 2023-09-17 can not be push down in partition filter*
> After this patch:
> {code:bash}
> == Physical Plan ==
> *(1) ColumnarToRow
> +- FileScan parquet default.wechar_tbl[id#62,name#63,dt#64] Batched: true,
> DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[],
> PartitionFilters: [isnotnull(dt#64), (dt#64 = 2023-09-17)], PushedFilters:
> [], ReadSchema: struct<id:int,name:string>
> {code}
> *(dt#64 = 2023-09-17) can be push down in partition filter*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
