Rakesh Kumar created SPARK-61647:
------------------------------------

             Summary: Drop a fast hash map's buckets when it closes
                 Key: SPARK-61647
                 URL: https://issues.apache.org/jira/browse/SPARK-61647
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 4.2.0
            Reporter: Rakesh Kumar


A partial hash aggregate with the codegen fast hash map registers a 
task-completion hook that closes the map, in case its output is not fully 
consumed ({{HashAggregateExec.addFastHashMapCloseHook}}). {{close()}} frees the 
map's memory page, but the hook keeps the map, and with it its bucket array, 
reachable until the task ends: 512 KiB at the default 
{{spark.sql.codegen.aggregate.fastHashMap.capacityBit}} of 16. The array is on 
the heap and not tracked by the memory manager.

A task that coalesces many partitions of such an aggregate creates one map per 
partition, also for an empty one, and holds all of their bucket arrays until it 
ends. Nothing on master coalesces below a partial aggregate today, but 
SPARK-60265 (https://github.com/apache/spark/pull/59522) does: its packing node 
coalesces a scan's partitions below a shuffle, above any partial aggregate. 
With 2,048 input splits in one task, the task held 1 GiB of bucket arrays.

{{close()}} can drop the buckets, in {{RowBasedAggregateHashMap}} and in the 
generated vectorized map. Nothing probes a map after it is closed.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to