[
https://issues.apache.org/jira/browse/SPARK-28375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17049667#comment-17049667
]
Dongjoon Hyun commented on SPARK-28375:
---------------------------------------
Although we don't backport this, I updated this issue type as `Bug` and updated
the affected version due to `postHocOptimizationBatches`.
> Enforce idempotence on the PullupCorrelatedPredicates optimizer rule
> --------------------------------------------------------------------
>
> Key: SPARK-28375
> URL: https://issues.apache.org/jira/browse/SPARK-28375
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0, 2.3.0, 2.4.0, 3.0.0
> Reporter: Yesheng Ma
> Assignee: Dilip Biswal
> Priority: Major
> Labels: correctness
> Fix For: 3.0.0
>
>
> The current PullupCorrelatedPredicates implementation can accidentally remove
> predicates for multiple runs.
> For example, for the following logical plan, one more optimizer run can
> remove the predicate in the SubqueryExpresssion.
> {code:java}
> # Optimized
> Project [a#0]
> +- Filter a#0 IN (list#4 [(b#1 < d#3)])
> : +- Project [c#2, d#3]
> : +- LocalRelation <empty>, [c#2, d#3]
> +- LocalRelation <empty>, [a#0, b#1]
> # Double optimized
> Project [a#0]
> +- Filter a#0 IN (list#4 [])
> : +- Project [c#2, d#3]
> : +- LocalRelation <empty>, [c#2, d#3]
> +- LocalRelation <empty>, [a#0, b#1]
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
