ulysses-you commented on code in PR #57742: URL: https://github.com/apache/spark/pull/57742#discussion_r3718359834
########## sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala: ########## @@ -154,6 +200,31 @@ case class HashAggregateExec( private var hashMapTerm: String = _ private var sorterTerm: String = _ + // Codegen state for adaptive partial aggregation. When the pre-shuffle reduction ratio of the + // regular (second-level) hash map is too low, the operator stops populating the map and instead + // streams each remaining row through as a single-row partial buffer for the Final aggregate to + // merge. Only the regular map is governed: the append-only fast hash map keeps absorbing hot Review Comment: Addressed. The ratio is now operator-level on both sides: the denominator is `fastMap.getNumKeys() + regularMap.getNumKeys()` (falling back to the regular map alone when the fast map is off), and the numerator counts all processed rows. Two-level-map routing no longer changes the decision, so there is a single policy to maintain. `RowBasedAggregateHashMap.getNumKeys()` and a generated `getNumKeys()` in `VectorizedHashMapGenerator` were added to expose the fast map key count. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
