[
https://issues.apache.org/jira/browse/SPARK-61647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61647:
-----------------------------------
Labels: pull-request-available (was: )
> Drop a fast hash map's buckets when it closes
> ---------------------------------------------
>
> Key: SPARK-61647
> URL: https://issues.apache.org/jira/browse/SPARK-61647
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Rakesh Kumar
> Priority: Major
> Labels: pull-request-available
>
> A partial hash aggregate with the codegen fast hash map registers a
> task-completion hook that closes the map, in case its output is not fully
> consumed ({{HashAggregateExec.addFastHashMapCloseHook}}). {{close()}} frees
> the map's memory page, but the hook keeps the map, and with it its bucket
> array, reachable until the task ends: 512 KiB at the default
> {{spark.sql.codegen.aggregate.fastHashMap.capacityBit}} of 16. The array is
> on the heap and not tracked by the memory manager.
> A task that coalesces many partitions of such an aggregate creates one map
> per partition, also for an empty one, and holds all of their bucket arrays
> until it ends. Nothing on master coalesces below a partial aggregate today,
> but SPARK-60265 (https://github.com/apache/spark/pull/59522) does: its
> packing node coalesces a scan's partitions below a shuffle, above any partial
> aggregate. With 2,048 input splits in one task, the task held 1 GiB of bucket
> arrays.
> {{close()}} can drop the buckets, in {{RowBasedAggregateHashMap}} and in the
> generated vectorized map. Nothing probes a map after it is closed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]