cloud-fan commented on code in PR #44975:
URL: https://github.com/apache/spark/pull/44975#discussion_r1474474986
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -1733,8 +1733,8 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
// attributes produced by the aggregate operator's child operator.
val (pushDown, stayUp) = splitConjunctivePredicates(condition).partition
{ cond =>
val replaced = replaceAlias(cond, aliasMap)
- cond.deterministic && !cond.throwable &&
- cond.references.nonEmpty &&
replaced.references.subsetOf(aggregate.child.outputSet)
+ cond.deterministic && cond.references.nonEmpty &&
Review Comment:
I think we need to document these hidden assumptions, otherwise code changes
here are hard to review.
If we can push down a filter through Aggregate, it means the filter only
references the grouping keys. The Aggregate operator can't reduce grouping keys
so the filter won't see any new data after pushing down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
