c21 commented on a change in pull request #32242:
URL: https://github.com/apache/spark/pull/32242#discussion_r617125855
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/RowBasedHashMapGenerator.scala
##########
@@ -87,6 +87,18 @@ class RowBasedHashMapGenerator(
|
| buckets = new int[numBuckets];
| java.util.Arrays.fill(buckets, -1);
+ |
+ | // Register a cleanup task with TaskContext to ensure that memory
is guaranteed to be
+ | // freed at the end of the task. This is necessary to avoid memory
leaks in when the
+ | // downstream operator does not fully consume the aggregation
map's output
+ | // (e.g. aggregate followed by limit).
+ | taskContext.addTaskCompletionListener(
Review comment:
@cloud-fan - sounds good, moved out to `HashAggregateExec`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
