[
https://issues.apache.org/jira/browse/SPARK-55072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-55072:
-----------------------------------
Labels: pull-request-available (was: )
> Inferring new Constraint misses IsNotNull on Left Leg, when an Outer Join
> gets converted into inner Join
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-55072
> URL: https://issues.apache.org/jira/browse/SPARK-55072
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.2.0, 4.1.1
> Reporter: Asif
> Priority: Major
> Labels: pull-request-available
>
> Presently, the order of execution of the Optimization rules is :
>
> {quote}{color:#4c9aff}*{{step1}}*{color}
> {{Batch("Operator Optimization before Inferring Filters", fixedPoint,}}
> {{operatorOptimizationRuleSet: _*),}}
>
> {color:#4c9aff}*{{step2}}*{color}
> {{Batch("Infer Filters", Once,}}
> {{InferFiltersFromGenerate,}}
> {{InferFiltersFromConstraints),}}
>
> {color:#4c9aff}*{{step3}}*{color}
> {{Batch("Operator Optimization after Inferring Filters", fixedPoint,}}
> {{operatorOptimizationRuleSet: _*)}}
> {quote}
>
> {{{}In the batch of rules "{}}}{{{}operatorOptimizationRuleSet", the
> conversion of Joins like "Left Outer" to Inner happens.{}}}{{{{}}{}}}
> {{{}After that "{}}}{{{}InferFiltersFromConstraints" is called which is able
> to create new constraints like IsNotNull, to be pushed on either side of the
> Inner Join tables.{}}}{{{{}}{}}}
>
> {{{}Notice that "{}}}{{{}operatorOptimizationRuleSet" is called twice, before
> and after inferring filters.{}}}{{{{}}{}}}
>
> {{It so happens that in TPCDS Q5, atleast, the conversion of LeftOuter to
> Inner for one of the Join cases, happens in {color:#4c9aff}*step3.*{color}}}
>
> {{But since, there is no further call of InferFiltersFromConstraints, the
> IsNotNull constraints generation is missed, for the Left Leg of the Join.}}
>
> {{{}IMHO the batch rule "{}}}{{{}Infer Filters" should be made part of
> "{}}}{{{}operatorOptimizationRuleSet". and there should be
> {color:#4c9aff}*NO step3.*{color}{}}}{{{{}}{}}}
>
> {{Will be opening a PR with possible targeted test.}}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
