agubichev commented on code in PR #42383:
URL: https://github.com/apache/spark/pull/42383#discussion_r1300502131
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/DecorrelateInnerQuery.scala:
##########
@@ -654,6 +654,25 @@ object DecorrelateInnerQuery extends PredicateHelper {
val newProject = Project(newProjectList ++ referencesToAdd,
newChild)
(newProject, joinCond, outerReferenceMap)
+ case w@Window(projectList, partitionSpec, orderSpec, child) =>
+ val outerReferences = collectOuterReferences(w.expressions)
+ assert(outerReferences.isEmpty, s"Correlated column is not allowed
in window " +
+ s"function: $w")
+ val newOuterReferences = parentOuterReferences ++ outerReferences
+ val (newChild, joinCond, outerReferenceMap) =
+ decorrelate(child, newOuterReferences, aggregated = true,
underSetOp)
Review Comment:
window function is not an aggregate, indeed. Moreover, the query that you
want to try, should give a runtime exception (As there are >1 rows returned
from the subquery, unless only 1 row satisfies the b >= d filter).
However, what i want to achieve by setting aggregate = true is to avoid
pulling-up non-equi predicates from under the Window function to the join
above. It is correct to do so for equi-predicates, but for non-equi in general
we need to keep a domain join. Perhaps the `aggregate` input param should be
renamed to reflect that it is about non-equi predicate treatment..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
