peter-toth commented on code in PR #57742: URL: https://github.com/apache/spark/pull/57742#discussion_r3758014638
########## sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala: ########## @@ -355,7 +420,61 @@ class TungstenAggregationIterator( } /////////////////////////////////////////////////////////////////////////// - // Part 6: Loads input rows and setup aggregationBufferMapIterator if we + // Part 6: Methods and fields used by adaptive partial aggregation pass-through. + /////////////////////////////////////////////////////////////////////////// + + // Indicates that partial aggregation has been bypassed and the remaining input rows should be + // passed through as single-row partial buffers. Set in `processInputs` by either check point. + // It may coexist with earlier spills. The output order matches the generated path: the first + // pass-through row is emitted before the frozen map (or sort-based) output, and the remaining + // rows stream afterwards. + private[this] var passThrough: Boolean = false + + // The row that could not be inserted at the spill check. It is stashed here (as + // a copy) so it becomes the first pass-through row rather than being lost. + private[this] var pendingPassThroughRow: InternalRow = null + + // Whether the first pass-through row has not been emitted yet. It is emitted before the frozen + // map (or sort) output -- mirroring the generated path, where the first bypassed row is appended + // to the output buffer from inside the build loop and the maps only drain afterwards. + private[this] var passThroughTriggerPending: Boolean = false + + // A reused aggregation buffer for building single-row partial buffers during pass-through. It is + // re-initialized from `initialAggregationBuffer` for every passed-through row. + private[this] lazy val passThroughAggregationBuffer: UnsafeRow = createNewAggregationBuffer() Review Comment: **Finding 19.** This is a third caller of `createNewAggregationBuffer()`, whose comment at `:131-132` still reads "This function should be only called at most two times (when we create the hash map, and when we create the re-used buffer for sort-based aggregation)." Worth adding the pass-through buffer there so the invariant is not stale. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
