Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194600791
@cloud-fan Your concern is valid.
When processing the child Window, it will always enter the case:
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L329-L332
Thus, it will never enter the case:
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L389-L397
You might have a question about the fix. Why not using the existing one
directly by changing the order?
The major issue is that the following two conditions could be always true,
no matter how many times we execute this rule:
```scala
(w.outputSet -- p.references).nonEmpty // the condition of #L329-L332
(w.child.inputSet -- (w.references ++ p.references)).nonEmpty // the
condition of L389-L397
```
Thus, changing the order does not help in this case. We have to combine two
cases into the same one. Sorry, it is little bit hard to explain it.
Also added three test cases for verifying the fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
