Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/11635#discussion_r60847817
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -958,6 +958,28 @@ object PushDownPredicate extends Rule[LogicalPlan]
with PredicateHelper {
project.copy(child = Filter(replaceAlias(condition, aliasMap),
grandChild))
+ // Push [[Filter]] operators through [[Window]] operators. Parts of
the predicate that can be
+ // pushed beneath must satisfy the following two conditions:
+ // 1. All the expressions are part of window partitioning key. The
expressions can be compound.
+ // 2. Deterministic
+ case filter @ Filter(condition, w: Window)
+ if w.partitionSpec.forall(_.isInstanceOf[AttributeReference]) =>
+ val partitionAttrs =
AttributeSet(w.partitionSpec.flatMap(_.references))
+ val (pushDown, stayUp) =
splitConjunctivePredicates(condition).partition { cond =>
+ cond.references.subsetOf(partitionAttrs) && cond.deterministic &&
+ // This is for ensuring all the partitioning expressions have
been converted to alias
+ // in Analyzer. Thus, we do not need to check if the expressions
in conditions are
+ // the same as the expressions used in partitioning columns.
+ partitionAttrs.forall(_.isInstanceOf[Attribute])
+ }
+ if (pushDown.nonEmpty) {
+ val pushDownPredicate = pushDown.reduce(And)
+ val newWindow = w.copy(child = Filter(pushDownPredicate, w.child))
+ if (stayUp.isEmpty) newWindow else Filter(stayUp.reduce(And),
newWindow)
--- End diff --
Math.pow(NIT, 100): What does the style guide say about ternary expression
onliners?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
