[
https://issues.apache.org/jira/browse/SPARK-54881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Asif updated SPARK-54881:
-------------------------
Affects Version/s: 4.2.0
> BooleanSimplification rule using transformExpressionsUp instead of
> transformExpressionsDown, is inefficient in some cases resulting in delayed
> idempotency
> ----------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-54881
> URL: https://issues.apache.org/jira/browse/SPARK-54881
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.1.0, 4.2.0, 4.1.1
> Reporter: Asif
> Priority: Major
> Labels: pull-request-available
>
> In case of simplification of an expression of type
> Not(A <= B OR A >= B)
> to
> (A > B AND A < B)
> it does not happen in a single pass.
> This is due to use of "transformUp" instead of transformDown.
> Because of transformUp logic, Not (A <= B OR A >=B) is converted to
> Not(A <= B) AND Not(A >=B) in the first pass.
> Only in second pass would it result in final valid form of
> (A > B AND A < B) .
> However, just replacing transformUp with transformDown though fixes this
> issue, but it breaks idempotency in single pass for some cases as seen by
> failure in ConstantFoldingTest.
> so the fix should be such that it satisfies both of the cases.
>
> Will be opening a PR and bugtest in some time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
