[
https://issues.apache.org/jira/browse/SPARK-61647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh Kumar updated SPARK-61647:
---------------------------------
Description:
A partial hash aggregate with the codegen fast hash map registers a
task-completion hook that closes the map, in case its output is not fully
consumed ({{HashAggregateExec.addFastHashMapCloseHook}}). {{close()}} frees the
map's memory page, but the hook keeps the map reachable until the task ends,
and with it the map's bucket array and its batch: 512 KiB of buckets at the
default {{spark.sql.codegen.aggregate.fastHashMap.capacityBit}} of 16, and with
a variable-length key or value, a {{VariableLengthRowBasedKeyValueBatch}} whose
array of key offsets is the same size. The generated aggregation code also
keeps the batch reachable. These arrays are on the heap and not tracked by the
memory manager.
A task that runs many partitions of such an aggregate creates one map per
partition, also for an empty one, and holds all of these arrays until it ends.
On master, this happens where {{GroupPartitionsExec}} groups a keyed scan's
splits above a partial hash aggregate. {{CombineAdjacentAggregation}} folds a
partial and a final aggregate across {{GroupPartitionsExec}}, but not the first
partial aggregate of a distinct aggregation, which sits below a
{{PartialMerge}} one. In a local test, a DSv2 scan whose splits all report the
same partition key, with {{groupBy(key).agg(countDistinct(j), sum(j))}}, ran
every split's partial aggregate in one task. With 168 splits, the task held 84
MiB of bucket arrays until it ended. With 2,048 splits, it held 1 GiB.
SPARK-60265 (https://github.com/apache/spark/pull/59522) would make this more
common: its packing node coalesces a scan's partitions below a shuffle, above
any partial aggregate. With 2,048 splits packed into one task and a string
grouping key, the task held 1 GiB of buckets and 1 GiB of key offsets.
{{close()}} can drop the buckets, in {{RowBasedAggregateHashMap}} and in the
generated vectorized map, and {{VariableLengthRowBasedKeyValueBatch.close()}}
can drop its key offsets. Nothing reads them after a close.
was:
A partial hash aggregate with the codegen fast hash map registers a
task-completion hook that closes the map, in case its output is not fully
consumed ({{HashAggregateExec.addFastHashMapCloseHook}}). {{close()}} frees the
map's memory page, but the hook keeps the map, and with it its bucket array,
reachable until the task ends: 512 KiB at the default
{{spark.sql.codegen.aggregate.fastHashMap.capacityBit}} of 16. The array is on
the heap and not tracked by the memory manager.
A task that runs many partitions of such an aggregate creates one map per
partition, also for an empty one, and holds all of their bucket arrays until it
ends.
On master, this happens where {{GroupPartitionsExec}} groups a keyed scan's
splits above a partial hash aggregate. {{CombineAdjacentAggregation}} folds a
partial and a final aggregate across {{GroupPartitionsExec}}, but not the first
partial aggregate of a distinct aggregation, which sits below a
{{PartialMerge}} one. In a local test, a DSv2 scan whose splits all report the
same partition key, with {{groupBy(key).agg(countDistinct(j), sum(j))}}, ran
every split's partial aggregate in one task. With 168 splits, the task held 84
MiB of bucket arrays until it ended. With 2,048 splits, it held 1 GiB.
SPARK-60265 (https://github.com/apache/spark/pull/59522) would make this more
common: its packing node coalesces a scan's partitions below a shuffle, above
any partial aggregate.
{{close()}} can drop the buckets, in {{RowBasedAggregateHashMap}} and in the
generated vectorized map. Nothing probes a map after it is closed.
Summary: Drop a closed fast hash map's buckets and key offsets (was:
Drop a fast hash map's buckets when it closes)
> Drop a closed fast hash map's buckets and key offsets
> -----------------------------------------------------
>
> Key: SPARK-61647
> URL: https://issues.apache.org/jira/browse/SPARK-61647
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Rakesh Kumar
> Priority: Major
> Labels: pull-request-available
>
> A partial hash aggregate with the codegen fast hash map registers a
> task-completion hook that closes the map, in case its output is not fully
> consumed ({{HashAggregateExec.addFastHashMapCloseHook}}). {{close()}} frees
> the map's memory page, but the hook keeps the map reachable until the task
> ends, and with it the map's bucket array and its batch: 512 KiB of buckets at
> the default {{spark.sql.codegen.aggregate.fastHashMap.capacityBit}} of 16,
> and with a variable-length key or value, a
> {{VariableLengthRowBasedKeyValueBatch}} whose array of key offsets is the
> same size. The generated aggregation code also keeps the batch reachable.
> These arrays are on the heap and not tracked by the memory manager.
> A task that runs many partitions of such an aggregate creates one map per
> partition, also for an empty one, and holds all of these arrays until it ends.
> On master, this happens where {{GroupPartitionsExec}} groups a keyed scan's
> splits above a partial hash aggregate. {{CombineAdjacentAggregation}} folds a
> partial and a final aggregate across {{GroupPartitionsExec}}, but not the
> first partial aggregate of a distinct aggregation, which sits below a
> {{PartialMerge}} one. In a local test, a DSv2 scan whose splits all report
> the same partition key, with {{groupBy(key).agg(countDistinct(j), sum(j))}},
> ran every split's partial aggregate in one task. With 168 splits, the task
> held 84 MiB of bucket arrays until it ended. With 2,048 splits, it held 1 GiB.
> SPARK-60265 (https://github.com/apache/spark/pull/59522) would make this more
> common: its packing node coalesces a scan's partitions below a shuffle, above
> any partial aggregate. With 2,048 splits packed into one task and a string
> grouping key, the task held 1 GiB of buckets and 1 GiB of key offsets.
> {{close()}} can drop the buckets, in {{RowBasedAggregateHashMap}} and in the
> generated vectorized map, and {{VariableLengthRowBasedKeyValueBatch.close()}}
> can drop its key offsets. Nothing reads them after a close.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]