Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19488#discussion_r145281602
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala
---
@@ -205,14 +205,15 @@ object PhysicalAggregation {
case logical.Aggregate(groupingExpressions, resultExpressions, child)
=>
// A single aggregate expression might appear multiple times in
resultExpressions.
// In order to avoid evaluating an individual aggregate function
multiple times, we'll
- // build a set of the distinct aggregate expressions and build a
function which can
- // be used to re-write expressions so that they reference the single
copy of the
- // aggregate function which actually gets computed.
+ // build a set of semantically distinct aggregate expressions and
re-write expressions so
+ // that they reference the single copy of the aggregate function
which actually gets computed.
+ val equivalentAggregateExpressions = new EquivalentExpressions
val aggregateExpressions = resultExpressions.flatMap { expr =>
expr.collect {
- case agg: AggregateExpression => agg
+ case agg: AggregateExpression
+ if (!equivalentAggregateExpressions.addExpr(agg)) => agg
--- End diff --
Add a comment above this line? `addExpr() returns false when the expression
is non-deterministic`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
