uros-b commented on code in PR #57059:
URL: https://github.com/apache/spark/pull/57059#discussion_r3577186567
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala:
##########
@@ -897,3 +894,25 @@ case class HashAggregateExec(
override protected def withNewChildInternal(newChild: SparkPlan):
HashAggregateExec =
copy(child = newChild)
}
+
+object HashAggregateExec {
+ /**
+ * Spills the in-memory hash map to disk and returns the sorter holding the
spilled data. Called
+ * by the generated code of [[HashAggregateExec]] when a buffer cannot be
allocated from the hash
+ * map: the first spill destructs the map into a new sorter, and later
spills merge into the
+ * existing one. Returning the sorter lets the generated code keep it in a
single mutable field.
+ *
+ * Extracting this type-independent spill machinery into a shared helper
keeps it compiled once
+ * per JVM instead of being re-emitted into every HashAggregateExec stage's
generated code.
+ */
+ def spillHashMapToSorter(
Review Comment:
Please add a dedicated test that calls the extracted helper directly and
asserts both branches. Since spillHashMapToSorter is now a pure static helper,
it's trivially unit-testable in isolation, so please consider adding a small
SparkFunSuite that calls it directly for both cases.
sorter == null → returns the newly created sorter from
destructAndCreateExternalSorter()
sorter != null → merge(...) is invoked and the same sorter instance is
returned
The controlled-fallback suite does cover these paths indirectly, but a
direct test locks the two-branch contract at the helper boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
