ulysses-you commented on code in PR #57742:
URL: https://github.com/apache/spark/pull/57742#discussion_r3718361895
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala:
##########
@@ -626,13 +792,27 @@ case class HashAggregateExec(
| long $beforeAgg = System.nanoTime();
| $doAggFuncName(partitionIndex);
| $aggTime.add((System.nanoTime() - $beforeAgg) / $NANOS_PER_MILLIS);
+ | $adaptiveStopCheck
|}
- |// output the result
- |$outputFromFastHashMap
- |$outputFromRegularHashMap
+ |$adaptiveResumeBuild
+ |$adaptiveFinalOutput
""".stripMargin
}
+ // Blocking operators normally suppress the child's `shouldStop()` check
because they buffer all
+ // output. With adaptive partial aggregation, pass-through rows are appended
to the output buffer
+ // while consuming child input, so the stop check must be re-enabled to keep
the buffer bounded.
+ override def needStopCheck: Boolean =
+ adaptivePartialAggConfig.isDefined || super.needStopCheck
+
+ // Blocking operators normally do not copy their result because every output
row is drained (via
+ // `shouldStop()`) before the next one is produced. Adaptive pass-through
breaks that assumption:
+ // when an `Expand` sits below, one input row fans out into several
pass-through rows that are all
+ // appended in the same child loop iteration before any drain, and they all
alias the single
+ // result `UnsafeRow`. Copy the result so the buffered rows do not collapse
into the last one.
+ override def needCopyResult: Boolean =
+ adaptivePartialAggConfig.isDefined || super.needCopyResult
Review Comment:
Addressed, now gated on the child:
```scala
override def needCopyResult: Boolean = adaptivePartialAggEnabled &&
child.asInstanceOf[CodegenSupport].needCopyResult
```
`ExpandExec` reports `needCopyResult = true`, so the aliasing fix is
preserved for the plans that need it without adding `row.copy()` to ordinary
output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
