ulysses-you commented on code in PR #57742:
URL: https://github.com/apache/spark/pull/57742#discussion_r3720249771
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala:
##########
@@ -154,6 +200,31 @@ case class HashAggregateExec(
private var hashMapTerm: String = _
private var sorterTerm: String = _
+ // Codegen state for adaptive partial aggregation. When the pre-shuffle
reduction ratio of the
+ // regular (second-level) hash map is too low, the operator stops populating
the map and instead
+ // streams each remaining row through as a single-row partial buffer for the
Final aggregate to
+ // merge. Only the regular map is governed: the append-only fast hash map
keeps absorbing hot
Review Comment:
Fixed in d95d6c52655. You were right that this was live: the increment sat
inside the regular-map branch while the denominator counted both maps, so rows
the fast map absorbed were missing from the numerator and a heavily-reducing
aggregate could measure as ineffective.
The increment and the periodic check now run after fast/regular routing,
keyed on either buffer being set:
```java
if (fastAggBuffer != null || unsafeRowAggBuffer != null) {
if (!adaptivePassThrough) {
processedRows += 1;
// periodic check
}
} else if (adaptivePassThrough) {
// build the single-row partial buffer
}
```
The row that fails to insert at the spill boundary holds no buffer, so it
stays out of the completed-row count and is streamed instead.
Added the test you asked for -- a hot-key prefix the fast map retains plus a
short distinct tail reaching the regular map (ratio ~17.5 against the 1.1
threshold), asserting nothing bypasses. It fails before the fix on exactly
`wholeStage=true twoLevelMap=true`, the only combination the bug can reach. The
interpreted path has no fast map and was already consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
