Rakesh Kumar created SPARK-61647:
------------------------------------
Summary: Drop a fast hash map's buckets when it closes
Key: SPARK-61647
URL: https://issues.apache.org/jira/browse/SPARK-61647
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: Rakesh Kumar
A partial hash aggregate with the codegen fast hash map registers a
task-completion hook that closes the map, in case its output is not fully
consumed ({{HashAggregateExec.addFastHashMapCloseHook}}). {{close()}} frees the
map's memory page, but the hook keeps the map, and with it its bucket array,
reachable until the task ends: 512 KiB at the default
{{spark.sql.codegen.aggregate.fastHashMap.capacityBit}} of 16. The array is on
the heap and not tracked by the memory manager.
A task that coalesces many partitions of such an aggregate creates one map per
partition, also for an empty one, and holds all of their bucket arrays until it
ends. Nothing on master coalesces below a partial aggregate today, but
SPARK-60265 (https://github.com/apache/spark/pull/59522) does: its packing node
coalesces a scan's partitions below a shuffle, above any partial aggregate.
With 2,048 input splits in one task, the task held 1 GiB of bucket arrays.
{{close()}} can drop the buckets, in {{RowBasedAggregateHashMap}} and in the
generated vectorized map. Nothing probes a map after it is closed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]