[
https://issues.apache.org/jira/browse/SPARK-60601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan updated SPARK-60601:
--------------------------------
Affects Version/s: 4.4.0
(was: 5.0.0)
> Detect keys that recur across epochs in the adaptive partial aggregation
> spill check
> ------------------------------------------------------------------------------------
>
> Key: SPARK-60601
> URL: https://issues.apache.org/jira/browse/SPARK-60601
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Wenchen Fan
> Priority: Major
>
> When the aggregation map of adaptive partial aggregation is about to spill,
> the spill check (SPARK-60561) decides whether to keep aggregating through
> spills or pass the rest of the input through. It judges the recent rows of
> the current map epoch, which cannot show whether keys come back after the map
> fills: with keys spread evenly over a task, the recent ratio reflects how
> much of the task's distinct keys the map holds, not how often each key
> repeats. So at the default {{minCompactionToSpill}}, a task whose map holds
> well under half of its keys passes the rest through even when each key
> repeats many times. In the "keys recur, map holds 40% of them" cases of
> {{AdaptivePartialAggregationBenchmark}} that is within about 8% of keeping
> the aggregation on most runners, but up to about 20% slower on one.
> Add a signal for keys that recur across epochs, at no per-row cost beyond a
> bit test. For example, keep a small sample of the first epoch's keys, chosen
> by bits of the hash the map already computes, and count how often later
> epochs hit it. With the two-level map on, the fast map's keys, which outlive
> every spill, can serve as that sample. Since no later epoch exists at the
> first fill, using the signal may mean deferring the decision past the first
> spill.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
