uros-b commented on code in PR #57059:
URL: https://github.com/apache/spark/pull/57059#discussion_r3577192337
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala:
##########
@@ -897,3 +894,25 @@ case class HashAggregateExec(
override protected def withNewChildInternal(newChild: SparkPlan):
HashAggregateExec =
copy(child = newChild)
}
+
+object HashAggregateExec {
+ /**
+ * Spills the in-memory hash map to disk and returns the sorter holding the
spilled data. Called
+ * by the generated code of [[HashAggregateExec]] when a buffer cannot be
allocated from the hash
+ * map: the first spill destructs the map into a new sorter, and later
spills merge into the
+ * existing one. Returning the sorter lets the generated code keep it in a
single mutable field.
+ *
+ * Extracting this type-independent spill machinery into a shared helper
keeps it compiled once
+ * per JVM instead of being re-emitted into every HashAggregateExec stage's
generated code.
+ */
Review Comment:
Nit: SPARK-57909's advanceBatch doc explicitly notes it's "called by the
generated code ... through the static forwarder." Mirroring that one-liner here
would keep the two helpers' docs parallel and hint to future readers why the
fully-qualified HashAggregateExec.spillHashMapToSorter(...) call works from
generated Java.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
