[ 
https://issues.apache.org/jira/browse/SPARK-61647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61647:
-----------------------------------
    Labels: pull-request-available  (was: )

> Drop a fast hash map's buckets when it closes
> ---------------------------------------------
>
>                 Key: SPARK-61647
>                 URL: https://issues.apache.org/jira/browse/SPARK-61647
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.2.0
>            Reporter: Rakesh Kumar
>            Priority: Major
>              Labels: pull-request-available
>
> A partial hash aggregate with the codegen fast hash map registers a 
> task-completion hook that closes the map, in case its output is not fully 
> consumed ({{HashAggregateExec.addFastHashMapCloseHook}}). {{close()}} frees 
> the map's memory page, but the hook keeps the map, and with it its bucket 
> array, reachable until the task ends: 512 KiB at the default 
> {{spark.sql.codegen.aggregate.fastHashMap.capacityBit}} of 16. The array is 
> on the heap and not tracked by the memory manager.
> A task that coalesces many partitions of such an aggregate creates one map 
> per partition, also for an empty one, and holds all of their bucket arrays 
> until it ends. Nothing on master coalesces below a partial aggregate today, 
> but SPARK-60265 (https://github.com/apache/spark/pull/59522) does: its 
> packing node coalesces a scan's partitions below a shuffle, above any partial 
> aggregate. With 2,048 input splits in one task, the task held 1 GiB of bucket 
> arrays.
> {{close()}} can drop the buckets, in {{RowBasedAggregateHashMap}} and in the 
> generated vectorized map. Nothing probes a map after it is closed.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to