[
https://issues.apache.org/jira/browse/SPARK-59317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheng Pan updated SPARK-59317:
------------------------------
Description:
Type coercion of the join keys may wrap a DSv2 runtime filter key in a cast,
e.g. {{cast(part_col as bigint) = <scalar subquery>}} when an INT partition
column is compared with a BIGINT scalar subquery.
SPARK-59301 unwraps the cast for the dynamic partition pruning IN filter
({{InSubqueryExec}}). The sibling runtime path,
{{DataSourceV2Strategy.translateScalarSubqueryFilterV2}}, literalizes the
scalar subquery once its result is known and translates the comparison with
{{translateFilterV2}}, which pushes {{CAST(part_col AS BIGINT) = <value>}}
through {{V2ExpressionBuilder}}. The cast is pushed but never unwrapped, so a
source that only prunes on column references ignores the filter. The optimizer
cannot unwrap it either, because the other side is not a literal until runtime.
The fix is to apply {{UnwrapCastInBinaryComparison}} to the literalized
comparison before translation, the same way SPARK-59301 does for {{InSet}}.
Unlike the IN case, the comparison rule rewrites out-of-range or rounded values
into different comparisons or constant results, so it needs its own handling
and tests.
Reported in https://github.com/apache/spark/pull/58580#issuecomment-5579953790
> Unwrap the cast of a DSv2 scalar subquery runtime filter before pushing it
> down
> -------------------------------------------------------------------------------
>
> Key: SPARK-59317
> URL: https://issues.apache.org/jira/browse/SPARK-59317
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Cheng Pan
> Priority: Major
>
> Type coercion of the join keys may wrap a DSv2 runtime filter key in a cast,
> e.g. {{cast(part_col as bigint) = <scalar subquery>}} when an INT partition
> column is compared with a BIGINT scalar subquery.
> SPARK-59301 unwraps the cast for the dynamic partition pruning IN filter
> ({{InSubqueryExec}}). The sibling runtime path,
> {{DataSourceV2Strategy.translateScalarSubqueryFilterV2}}, literalizes the
> scalar subquery once its result is known and translates the comparison with
> {{translateFilterV2}}, which pushes {{CAST(part_col AS BIGINT) = <value>}}
> through {{V2ExpressionBuilder}}. The cast is pushed but never unwrapped, so a
> source that only prunes on column references ignores the filter. The
> optimizer cannot unwrap it either, because the other side is not a literal
> until runtime.
> The fix is to apply {{UnwrapCastInBinaryComparison}} to the literalized
> comparison before translation, the same way SPARK-59301 does for {{InSet}}.
> Unlike the IN case, the comparison rule rewrites out-of-range or rounded
> values into different comparisons or constant results, so it needs its own
> handling and tests.
> Reported in https://github.com/apache/spark/pull/58580#issuecomment-5579953790
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
