ulysses-you commented on code in PR #57742:
URL: https://github.com/apache/spark/pull/57742#discussion_r3735293623
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala:
##########
@@ -711,6 +995,54 @@ case class HashAggregateExec(
} else {
findOrInsertRegularHashMap
}
+
+ // Every row is either accepted by an aggregation map or streamed
through -- the fast map
+ // serves a row without it ever reaching the regular map, so both
buffers are consulted to
+ // tell the two apart.
+ //
+ // An accepted row counts toward the compaction ratio, so the numerator
matches the
+ // operator-level denominator. Counting inside the regular-map branch
alone would drop the
+ // rows the fast map absorbed from the ratio and bypass an aggregation
that is in fact
+ // reducing. A row no map holds is streamed once pass-through is active:
both probes are
+ // skipped (guarded above), so neither buffer is set, and `rowBypassed`
marks exactly those
+ // rows. The row that fails to insert at the spill boundary lands here
too, while the row
+ // that merely flipped pass-through at the check point is already
aggregated in the map that
+ // took it and must not be re-emitted.
+ val countOrPassThroughRow = if (adaptivePartialAggEnabled) {
+ val heldByAMap = if (isFastHashMapEnabled) {
+ s"($fastRowBuffer != null || $unsafeRowBuffer != null)"
+ } else {
+ s"($unsafeRowBuffer != null)"
+ }
+ // The grouping key was already projected in
`findOrInsertRegularHashMap`
+ // (`unsafeRowKeyCode.code`), so `unsafeRowKeyCode.value` holds this
row's key. Only build
+ // the single-row partial buffer here.
+ s"""
+ |if ($heldByAMap) {
+ | if (!$adaptivePassThroughTerm) {
+ | $processedRowsTerm += 1;
+ | if ($processedRowsTerm == $adaptiveNextCheckRowTerm) {
Review Comment:
@peter-toth there is a history, I the checked the point grow with
adaptiveNextCheckRow *= 2 at first see
e1bbdd717e072d5d5d8ef8d702f4cc40a55657ce, and then simplified it according to
the @cloud-fan 's review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
