This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new e6fc64f58dc2 [SPARK-60561][SQL] Judge the adaptive partial aggregation
spill check by the recent rows, with a higher threshold
e6fc64f58dc2 is described below
commit e6fc64f58dc26663060af092d3845456f9c3b198
Author: Wenchen Fan <[email protected]>
AuthorDate: Fri Oct 30 07:10:19 2026 +0000
[SPARK-60561][SQL] Judge the adaptive partial aggregation spill check by
the recent rows, with a higher threshold
### What changes were proposed in this pull request?
Adaptive partial aggregation (#57742) evaluates its compaction ratio
periodically and right before its map would spill, against one threshold,
`minCompaction` (default 1.05). This PR changes the evaluation before a spill
in two ways, on both the codegen and the interpreted path:
- It measures the recent rows, those processed since the periodic
evaluation before last, against the keys they added, instead of the whole map
epoch.
- It demands a new, higher threshold,
`spark.sql.execution.aggregate.adaptivePartialAggregation.minCompactionToSpill`
(default 2.0). The evaluation uses the larger of it and `minCompaction`.
The window advances at each periodic evaluation, so the change adds no
per-row work. This replaces the spill threshold that an earlier revision of
#59764 proposed, which judged the whole epoch, and covers the review comments
on the spill check that were deferred from there.
### Why are the changes needed?
Keeping the aggregation when the map is about to spill means aggregating
the rest of the input through spills. That pays off only if the rest of the
input keeps repeating keys: the final merge then collapses them, and the
shuffle carries a fraction of the rows. Otherwise the task writes most of its
rows to disk to save shuffling a few. With one threshold of 1.05, an input
whose keys arrive together, at a ratio of 1.5, keeps aggregating and spills
every epoch.
The ratio of the whole epoch cannot tell those inputs apart. A uniformly
distributed key with ten rows per key per task fills its map at a ratio of
about 1.65, in the same range as an input whose epochs share no keys, yet
keeping the aggregation is the right call for it. The recent rows separate the
two. When keys repeat across the input, more of the recent rows find their key
already in the map as it fills, so their ratio climbs above the epoch's, to
between 2 and 3 in that example. [...]
The two evaluations weigh different costs, which is why they get separate
thresholds. The periodic one weighs only maintaining the map, so a ratio barely
above 1 is worth keeping, while the one before a spill weighs spilling the rest
of the input.
`AdaptivePartialAggregationBenchmark` gets spill-check cases with an
`Exchange` between the Partial and the Final. Compared with 1.05, in the "Run
benchmarks" workflow on JDK 17, 21 and 25, on both paths:
- Keys that arrive together: about 1.3x faster, because the rest of the
input is passed through instead of spilled every epoch.
- 10 hot keys over a unique tail: about 1.3x faster.
- Keys that recur, with the map holding about two thirds of them: the same
decision as 1.05, keeping the aggregation.
- Keys that recur, with the maps holding 40% of them, with the fast map off
and on: the default passes the rest through. That was within about 8% of
keeping the aggregation on the runners that produced the committed results, and
about 20% slower in one earlier run on a different CPU.
Passing the rest of the input through also streams the later rows of keys
the frozen map holds, so with a few hot keys each of them reaches the final
aggregation row by row, on the one reducer that key hashes to. The docs
describe this trade-off, and that setting `minCompactionToSpill` to
`minCompaction` or below keeps inputs dominated by one or two keys aggregating.
### Does this PR introduce _any_ user-facing change?
No. Adaptive partial aggregation is new in the unreleased 4.4.0 and
disabled by default. When it is enabled, a map about to spill passes the rest
of its input through when its recent rows compact by less than 2.0, and the new
config tunes that threshold.
### How was this patch tested?
New and updated tests in `AdaptivePartialAggregationSuite`, on both paths:
- An input whose keys arrive together at a ratio of 1.5 passes the rest
through at the spill check by default, and keeps spilling with the threshold
lowered to 1.3. It runs with one input partition and with two, which put an
`Exchange` between the Partial and the Final.
- An input whose epoch compacts by 1.875 but whose recent rows compact by
3.75 keeps aggregating by default, and passes through with the threshold raised
to 4.0.
- The exact-boundary test sets the spill threshold, and checks that the
larger of the two thresholds applies, from either side.
- A new test reaches the boundary of the recent rows: they compact by
exactly 2.0, while a window starting one check earlier compacts by 1.33 and the
whole epoch by 1.5, so the task keeps aggregating at 2.0 and passes through at
2.1.
- A spill followed by a distinct-heavy epoch now flips at that epoch's
first periodic check, with exact counts. It pins `minCompactionToSpill` to 1.5
so the counts do not rest on a tie at 2.0.
- Two tests pin the window's bookkeeping: a new epoch that fills after one
periodic check does not count the previous epoch's keys, and on the codegen
path the window counts the fast map's keys.
- The new config rejects non-finite values, as `minCompaction` does.
`build/sbt "sql/testOnly *AdaptivePartialAggregationSuite"` passes.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
Closes #59823 from cloud-fan/wenchen/spark-60561-spill-check.
Authored-by: Wenchen Fan <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit 60c7b8feb9986d9c4187dd9fbd4d7bef8d5ad02d)
Signed-off-by: Wenchen Fan <[email protected]>
---
docs/sql-performance-tuning.md | 88 ++++--
.../org/apache/spark/sql/internal/SQLConf.scala | 43 ++-
...vePartialAggregationBenchmark-jdk21-results.txt | 117 ++++++--
...vePartialAggregationBenchmark-jdk25-results.txt | 117 ++++++--
...AdaptivePartialAggregationBenchmark-results.txt | 117 ++++++--
.../execution/aggregate/HashAggregateExec.scala | 76 ++++--
.../aggregate/TungstenAggregationIterator.scala | 49 +++-
.../AdaptivePartialAggregationSuite.scala | 303 ++++++++++++++++++---
.../AdaptivePartialAggregationBenchmark.scala | 68 ++++-
9 files changed, 814 insertions(+), 164 deletions(-)
diff --git a/docs/sql-performance-tuning.md b/docs/sql-performance-tuning.md
index f38522110fd6..c5f4223ae558 100644
--- a/docs/sql-performance-tuning.md
+++ b/docs/sql-performance-tuning.md
@@ -198,24 +198,48 @@ for the final aggregation to merge. Once pass-through is
active the map is froze
always comes before the passed-through rows: rows that collide with the frozen
map are held in a
queue behind it and flushed only after it drains, so a duplicate of a key
already in the map still
merges after that key's accumulated rows, keeping an order-sensitive aggregate
such as
-`first`/`last` consistent with a run that never bypasses. The ratio is
evaluated periodically, and
-again right before the aggregation map would spill, in which case the spill is
skipped entirely.
-Because pass-through cannot be reversed, the evaluation before a spill also
waits until the current
-map has processed `minRows` rows, so a map that fills sooner spills as usual
instead of committing
-the task on a small sample. The SQL UI counts the passed-through rows in
-`number of pass-through output rows`. They are not timed: `time in aggregation
build` covers the
+`first`/`last` consistent with a run that never bypasses. The SQL UI counts
the passed-through rows
+in `number of pass-through output rows`. They are not timed: `time in
aggregation build` covers the
aggregation up to the point where pass-through starts, and preparing the
aggregated map for output,
which in a task that has spilled also writes the map to disk and merges it
with the spilled data.
-Both evaluations use the cumulative rows and keys of the current map epoch
and, once triggered,
-pass-through is not reversed for the rest of the task, so a skewed prefix
biases the outcome in
-either direction. A favorable prefix can mask a later distinct-heavy tail,
keeping the aggregation
-on until a spill restarts the accounting; conversely, a distinct-heavy prefix
can trip the
-pass-through early and keep it tripped even where the rest of the input would
aggregate well. To
+The ratio is evaluated periodically, every `minRows` rows, over the rows and
keys of the current map
+epoch, against `minCompaction`. It is evaluated again right before the
aggregation map would spill,
+against the larger of `minCompaction` and `minCompactionToSpill`. Keeping the
aggregation at that
+point means aggregating the rest of the input through spills, which pays off
only if that input
+keeps repeating keys, so this evaluation measures the recent rows, at least
`minRows` and fewer than
+twice that many: those processed since the periodic evaluation before last,
against the keys they
+added. When keys repeat across the input, more of the recent rows find their
key already in the map
+as it fills, so their ratio climbs above the epoch's; when each key's rows
arrive together, the two
+ratios match. The recent rows still only show how often keys repeat within the
map: with keys spread
+evenly over the task, their ratio reflects how much of the task's distinct
keys the map holds rather
+than how often each key repeats, so at the default threshold the aggregation
is kept only when the
+map holds about half of the task's keys or more. Before the current map's
second periodic
+evaluation, or with `minRows` set to `0`, the evaluation measures the whole
map epoch instead, whose
+ratio is lower because an epoch's early rows add most of its keys, so it
passes through more
+readily. If the ratio is below the threshold, the map is not spilled and the
rest of the input is
+passed through. In a task that has already spilled, the frozen map is still
written out and merged
+with the spilled data at the end, but no later epoch spills. Because
pass-through cannot be
+reversed, the evaluation before a spill also waits until the current map has
processed `minRows`
+rows, so a map that fills sooner spills as usual instead of committing the
task on a small sample.
+
+Once triggered, pass-through is not reversed for the rest of the task, so a
skewed prefix biases the
+outcome in either direction. A favorable prefix can mask a later
distinct-heavy tail, keeping the
+aggregation on until a spill restarts the accounting; conversely, a
distinct-heavy prefix can trip
+the pass-through early and keep it tripped even where the rest of the input
would aggregate well. To
catch the former without waiting for a spill, raise `minCompaction` so the
cumulative ratio trips
the threshold on a weaker late turn (a higher threshold also bypasses more
readily on other inputs).
-To avoid committing the task to pass-through on the latter, raise `minRows` so
the evaluations
-start later.
+To avoid committing the task to pass-through on the latter, raise `minRows` so
the evaluations start
+later, or, when the task commits at the evaluation before a spill, lower
`minCompactionToSpill`.
+Raising `minRows` helps there only past the number of rows the map takes to
fill, and also widens
+the sample of recent rows that the evaluation before a spill measures. Lowering
+`minCompactionToSpill` also keeps aggregating keys that repeat across a task
whose map holds well
+under half of them; whether that pays off depends on how the cost of spilling
the rest of the input
+compares with shuffling it. Pass-through also streams the later rows of keys
the frozen map holds,
+so with a few hot keys each of them reaches the final aggregation row by row,
on the one reducer
+that key hashes to. When one or two keys, such as a NULL or default value,
hold a large share of the
+rows over many distinct ones, that reducer can receive a large share of all
rows; setting
+`minCompactionToSpill` to `minCompaction` or below keeps such inputs
aggregating through spills.
<table class="spark-config">
<thead><tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since
Version</th></tr></thead>
@@ -235,11 +259,14 @@ start later.
<td>
The number of rows between periodic compaction-ratio evaluations. The
ratio is also evaluated
when the aggregation map is about to spill, but only once the current
map has processed at
- least this many rows. Setting this to <code>0</code> disables the
periodic evaluation, leaving
- only the evaluation when the map is about to spill. A larger value also
delays the
- evaluations, so when one of them trips pass-through the frozen map tends
to hold more rows;
- the frozen map stays resident until its output is drained, so a larger
value raises that
- transient memory peak.
+ least this many rows, and over the rows since the periodic evaluation
before last, so this
+ also sets how many recent rows that evaluation measures: at least this
many and fewer than
+ twice that, or the whole map epoch when the map fills before its second
periodic evaluation.
+ Setting this to <code>0</code> disables the periodic evaluation, leaving
only the evaluation
+ when the map is about to spill, which then measures the whole map epoch.
A larger value also
+ delays the evaluations, so when one of them trips pass-through the
frozen map tends to hold
+ more rows; the frozen map stays resident until its output is drained, so
a larger value raises
+ that transient memory peak.
</td>
<td>4.4.0</td>
</tr>
@@ -247,10 +274,27 @@ start later.
<td><code>spark.sql.execution.aggregate.adaptivePartialAggregation.minCompaction</code></td>
<td>1.05</td>
<td>
- The minimum compaction ratio required to keep the partial aggregation. A
ratio of 10 means the
- partial aggregation collapses ten rows into one key; when an evaluation
finds the ratio below
- this value the partial aggregation is bypassed for the rest of the
input. A larger value
- bypasses more aggressively.
+ The minimum compaction ratio required to keep the partial aggregation at
the periodic
+ evaluation. A ratio of 10 means the partial aggregation collapses ten
rows into one key; when
+ the periodic evaluation finds the ratio below this value the partial
aggregation is bypassed
+ for the rest of the input. A larger value bypasses more aggressively.
The evaluation before a
+ spill demands at least this value too.
+ </td>
+ <td>4.4.0</td>
+ </tr>
+ <tr>
+
<td><code>spark.sql.execution.aggregate.adaptivePartialAggregation.minCompactionToSpill</code></td>
+ <td>2.0</td>
+ <td>
+ The minimum compaction ratio required to keep the partial aggregation
when its map is about to
+ spill, measured over the rows processed since the periodic evaluation
before last, or over the
+ whole map epoch before its second periodic evaluation. Keeping the
aggregation then means
+ aggregating the rest of the input through spills, which pays off only if
that input keeps
+ repeating keys. With keys spread evenly over the task, the default keeps
aggregating only when
+ the map holds about half of the task's keys or more. When the ratio is
below the larger of
+ this value and <code>minCompaction</code>, the map is not spilled and
the rest of the input is
+ passed through. The evaluation runs only once the current map has
processed
+ <code>minRows</code> rows.
</td>
<td>4.4.0</td>
</tr>
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 692aac02e565..c4793ee4de7d 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -4627,8 +4627,12 @@ object SQLConf {
.doc("The number of rows between periodic compaction-ratio evaluations
by adaptive partial " +
s"aggregation (see '${ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key}'). The
ratio is also " +
"evaluated when the aggregation map is about to spill, but only once
the current map " +
- "has processed at least this many rows. Setting this to 0 disables the
periodic " +
- "evaluation, leaving only the evaluation when the map is about to
spill.")
+ "has processed at least this many rows, and over the rows since the
periodic " +
+ "evaluation before last, so this also sets how many recent rows that
evaluation " +
+ "measures: at least this many and fewer than twice that, or the whole
map epoch when " +
+ "the map fills before its second periodic evaluation. Setting this to
0 disables the " +
+ "periodic evaluation, leaving only the evaluation when the map is
about to spill, " +
+ "which then measures all rows and keys of the current map epoch.")
.version("4.4.0")
.withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.longConf
@@ -4640,9 +4644,11 @@ object SQLConf {
.doc("The minimum compaction ratio required to keep the pre-shuffle
partial aggregation " +
s"(see '${ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key}'). The compaction
ratio is the " +
"number of processed rows divided by the number of keys held in the
aggregation maps, " +
- "so a ratio of 10 means the partial aggregation collapses ten rows
into one. When an " +
- "evaluation finds the ratio below this value, the partial aggregation
is bypassed for " +
- "the rest of the input. A larger value bypasses more aggressively.")
+ "so a ratio of 10 means the partial aggregation collapses ten rows
into one. When the " +
+ "periodic evaluation finds the ratio below this value, the partial
aggregation is " +
+ "bypassed for the rest of the input. A larger value bypasses more
aggressively. The " +
+ "evaluation when the map is about to spill demands at least this value
too, see " +
+
"'spark.sql.execution.aggregate.adaptivePartialAggregation.minCompactionToSpill'.")
.version("4.4.0")
.withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.doubleConf
@@ -4650,6 +4656,30 @@ object SQLConf {
"The minimum compaction ratio must be a finite value of at least 1.0.")
.createWithDefault(1.05)
+ val ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL =
+
buildConf("spark.sql.execution.aggregate.adaptivePartialAggregation.minCompactionToSpill")
+ .doc("The minimum compaction ratio required to keep the pre-shuffle
partial aggregation " +
+ "when its map is about to spill (see " +
+ s"'${ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key}'). Keeping the
aggregation then means " +
+ "aggregating the rest of the input through spills, which pays off only
if that input " +
+ "keeps repeating keys, so this evaluation measures the recent rows:
those processed " +
+ "since the periodic evaluation before last, against the keys they
added, or the whole " +
+ "map epoch before the map's second periodic evaluation. When keys
repeat across the " +
+ "input, that ratio climbs as the map fills; with keys spread evenly
over the task it " +
+ "reflects how much of the task's distinct keys the map holds rather
than how often " +
+ "each key repeats, so at the default the aggregation is kept only when
the map holds " +
+ "about half of them or more. When the ratio is below the larger of
this value and " +
+ s"'${ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION.key}', the map is not
spilled and the " +
+ "rest of the input is passed through. Like the periodic evaluation, it
only runs once " +
+ "the current map has processed " +
+ s"'${ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key}' rows.")
+ .version("4.4.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .doubleConf
+ .checkValue(v => v >= 1.0 && v.isFinite,
+ "The minimum compaction ratio must be a finite value of at least 1.0.")
+ .createWithDefault(2.0)
+
val JSON_GENERATOR_IGNORE_NULL_FIELDS =
buildConf("spark.sql.jsonGenerator.ignoreNullFields")
.doc("Whether to ignore null fields when generating JSON objects in JSON
data source and " +
@@ -9847,6 +9877,9 @@ class SQLConf extends Serializable with Logging with
SqlApiConf {
def adaptivePartialAggregationMinCompaction: Double =
getConf(ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION)
+ def adaptivePartialAggregationMinCompactionToSpill: Double =
+ getConf(ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL)
+
def objectAggSortBasedFallbackThreshold: Int =
getConf(OBJECT_AGG_SORT_BASED_FALLBACK_THRESHOLD)
def variableSubstituteEnabled: Boolean = getConf(VARIABLE_SUBSTITUTE_ENABLED)
diff --git
a/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-jdk21-results.txt
b/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-jdk21-results.txt
index dd08f21eca70..5bc0aab19a43 100644
--- a/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-jdk21-results.txt
+++ b/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-jdk21-results.txt
@@ -2,55 +2,132 @@
high-cardinality input, pass-through at the periodic check
================================================================================================
-OpenJDK 64-Bit Server VM 21.0.12+8-LTS on Linux 6.17.0-1020-azure
+OpenJDK 64-Bit Server VM 21.0.13+9-LTS on Linux 6.17.0-1022-azure
AMD EPYC 7763 64-Core Processor
adaptive partial agg, high card, no spill: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F 4069 4145
108 2.1 485.0 1.0X
-codegen = true, adaptive = T 2585 2632
67 3.2 308.1 1.6X
-codegen = false, adaptive = F 4818 4849
44 1.7 574.3 0.8X
-codegen = false, adaptive = T 2983 3150
236 2.8 355.6 1.4X
+codegen = true, adaptive = F 4280 4290
14 2.0 510.2 1.0X
+codegen = true, adaptive = T 2617 2625
11 3.2 312.0 1.6X
+codegen = false, adaptive = F 5007 5030
32 1.7 596.9 0.9X
+codegen = false, adaptive = T 3253 3273
28 2.6 387.8 1.3X
================================================================================================
low-cardinality input, pass-through at the periodic check
================================================================================================
-OpenJDK 64-Bit Server VM 21.0.12+8-LTS on Linux 6.17.0-1020-azure
+OpenJDK 64-Bit Server VM 21.0.13+9-LTS on Linux 6.17.0-1022-azure
AMD EPYC 7763 64-Core Processor
adaptive partial agg, low card, no spill: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F 281 292
7 59.6 16.8 1.0X
-codegen = true, adaptive = T 291 323
52 57.7 17.3 1.0X
-codegen = false, adaptive = F 1289 1303
20 13.0 76.8 0.2X
-codegen = false, adaptive = T 1266 1270
7 13.3 75.4 0.2X
+codegen = true, adaptive = F 287 306
21 58.4 17.1 1.0X
+codegen = true, adaptive = T 293 301
9 57.2 17.5 1.0X
+codegen = false, adaptive = F 1356 1359
5 12.4 80.8 0.2X
+codegen = false, adaptive = T 1360 1365
7 12.3 81.1 0.2X
================================================================================================
high-cardinality input, pass-through at the spill check
================================================================================================
-OpenJDK 64-Bit Server VM 21.0.12+8-LTS on Linux 6.17.0-1020-azure
+OpenJDK 64-Bit Server VM 21.0.13+9-LTS on Linux 6.17.0-1022-azure
AMD EPYC 7763 64-Core Processor
adaptive partial agg, high card, spill: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F 8120 8126
9 1.0 968.0 1.0X
-codegen = true, adaptive = T 4460 4476
22 1.9 531.7 1.8X
-codegen = false, adaptive = F 9331 9389
82 0.9 1112.3 0.9X
-codegen = false, adaptive = T 5369 5382
18 1.6 640.1 1.5X
+codegen = true, adaptive = F 8875 8896
29 0.9 1058.0 1.0X
+codegen = true, adaptive = T 4931 4961
43 1.7 587.8 1.8X
+codegen = false, adaptive = F 10087 10123
51 0.8 1202.5 0.9X
+codegen = false, adaptive = T 5756 5769
19 1.5 686.2 1.5X
================================================================================================
low-cardinality input, pass-through at the spill check
================================================================================================
-OpenJDK 64-Bit Server VM 21.0.12+8-LTS on Linux 6.17.0-1020-azure
+OpenJDK 64-Bit Server VM 21.0.13+9-LTS on Linux 6.17.0-1022-azure
AMD EPYC 7763 64-Core Processor
adaptive partial agg, low card, spill: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F 768 782
12 21.8 45.8 1.0X
-codegen = true, adaptive = T 769 780
13 21.8 45.8 1.0X
-codegen = false, adaptive = F 1366 1369
4 12.3 81.4 0.6X
-codegen = false, adaptive = T 1358 1358
0 12.4 80.9 0.6X
+codegen = true, adaptive = F 817 829
11 20.5 48.7 1.0X
+codegen = true, adaptive = T 843 861
18 19.9 50.3 1.0X
+codegen = false, adaptive = F 1398 1478
113 12.0 83.3 0.6X
+codegen = false, adaptive = T 1406 1414
11 11.9 83.8 0.6X
+
+
+================================================================================================
+spill check with an Exchange, keys arrive together
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.13+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys arrive together: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+---------------------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F 6653
6661 11 1.3 793.1 1.0X
+codegen = true, minCompactionToSpill = 1.05 6571
6591 28 1.3 783.3 1.0X
+codegen = true, minCompactionToSpill = 2.0 5047
5102 77 1.7 601.7 1.3X
+codegen = false, adaptive = F 8090
8094 6 1.0 964.4 0.8X
+codegen = false, minCompactionToSpill = 1.05 8004
8069 93 1.0 954.1 0.8X
+codegen = false, minCompactionToSpill = 2.0 6219
6228 13 1.3 741.4 1.1X
+
+
+================================================================================================
+spill check with an Exchange, keys recur
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.13+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys recur: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+-----------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F 3364 3443
112 2.5 401.1 1.0X
+codegen = true, minCompactionToSpill = 1.05 3394 3417
33 2.5 404.5 1.0X
+codegen = true, minCompactionToSpill = 2.0 3210 3268
82 2.6 382.6 1.0X
+codegen = false, adaptive = F 4044 4177
188 2.1 482.1 0.8X
+codegen = false, minCompactionToSpill = 1.05 4386 4422
51 1.9 522.8 0.8X
+codegen = false, minCompactionToSpill = 2.0 4452 4479
38 1.9 530.8 0.8X
+
+
+================================================================================================
+spill check with an Exchange, keys recur, map holds 40% of them, fast map off
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.13+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys recur, map holds 40% of them, fast map
off: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns)
Relative
+------------------------------------------------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F
9194 9460 377 1.8 548.0
1.0X
+codegen = true, minCompactionToSpill = 1.05
9205 9459 360 1.8 548.6
1.0X
+codegen = true, minCompactionToSpill = 2.0
8908 8949 58 1.9 531.0
1.0X
+codegen = false, adaptive = F
9129 9345 306 1.8 544.1
1.0X
+codegen = false, minCompactionToSpill = 1.05
9652 9710 82 1.7 575.3
1.0X
+codegen = false, minCompactionToSpill = 2.0
10163 10196 47 1.7 605.7
0.9X
+
+
+================================================================================================
+spill check with an Exchange, keys recur, maps hold 40% of them, fast map on
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.13+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys recur, maps hold 40% of them, fast map
on: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns)
Relative
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F
8892 8975 118 1.9 530.0
1.0X
+codegen = true, minCompactionToSpill = 1.05
9157 9166 13 1.8 545.8
1.0X
+codegen = true, minCompactionToSpill = 2.0
8390 8393 6 2.0 500.1
1.1X
+
+
+================================================================================================
+spill check with an Exchange, hot keys
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.13+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, hot keys: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+----------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F 6814 6837
32 1.2 812.3 1.0X
+codegen = true, minCompactionToSpill = 1.05 6778 6819
58 1.2 808.0 1.0X
+codegen = true, minCompactionToSpill = 2.0 5298 5430
186 1.6 631.6 1.3X
+codegen = false, adaptive = F 7995 8007
16 1.0 953.1 0.9X
+codegen = false, minCompactionToSpill = 1.05 8093 8151
83 1.0 964.7 0.8X
+codegen = false, minCompactionToSpill = 2.0 6346 6372
36 1.3 756.5 1.1X
diff --git
a/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-jdk25-results.txt
b/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-jdk25-results.txt
index f54c40877aaf..f55167c051a5 100644
--- a/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-jdk25-results.txt
+++ b/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-jdk25-results.txt
@@ -2,55 +2,132 @@
high-cardinality input, pass-through at the periodic check
================================================================================================
-OpenJDK 64-Bit Server VM 25.0.4+7-LTS on Linux 6.17.0-1021-azure
+OpenJDK 64-Bit Server VM 25.0.5+8-LTS on Linux 6.17.0-1022-azure
AMD EPYC 7763 64-Core Processor
adaptive partial agg, high card, no spill: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F 4338 4408
100 1.9 517.1 1.0X
-codegen = true, adaptive = T 2586 2608
30 3.2 308.3 1.7X
-codegen = false, adaptive = F 5146 5199
76 1.6 613.4 0.8X
-codegen = false, adaptive = T 3271 3287
22 2.6 390.0 1.3X
+codegen = true, adaptive = F 3995 4021
36 2.1 476.3 1.0X
+codegen = true, adaptive = T 2435 2439
5 3.4 290.3 1.6X
+codegen = false, adaptive = F 4704 4707
5 1.8 560.7 0.8X
+codegen = false, adaptive = T 3088 3114
37 2.7 368.1 1.3X
================================================================================================
low-cardinality input, pass-through at the periodic check
================================================================================================
-OpenJDK 64-Bit Server VM 25.0.4+7-LTS on Linux 6.17.0-1021-azure
+OpenJDK 64-Bit Server VM 25.0.5+8-LTS on Linux 6.17.0-1022-azure
AMD EPYC 7763 64-Core Processor
adaptive partial agg, low card, no spill: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F 269 275
4 62.5 16.0 1.0X
-codegen = true, adaptive = T 284 294
6 59.1 16.9 0.9X
-codegen = false, adaptive = F 1317 1321
6 12.7 78.5 0.2X
-codegen = false, adaptive = T 1309 1382
104 12.8 78.0 0.2X
+codegen = true, adaptive = F 254 267
10 66.0 15.1 1.0X
+codegen = true, adaptive = T 301 308
6 55.8 17.9 0.8X
+codegen = false, adaptive = F 1321 1324
4 12.7 78.7 0.2X
+codegen = false, adaptive = T 1316 1316
1 12.8 78.4 0.2X
================================================================================================
high-cardinality input, pass-through at the spill check
================================================================================================
-OpenJDK 64-Bit Server VM 25.0.4+7-LTS on Linux 6.17.0-1021-azure
+OpenJDK 64-Bit Server VM 25.0.5+8-LTS on Linux 6.17.0-1022-azure
AMD EPYC 7763 64-Core Processor
adaptive partial agg, high card, spill: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F 8332 8462
184 1.0 993.2 1.0X
-codegen = true, adaptive = T 4677 4683
9 1.8 557.5 1.8X
-codegen = false, adaptive = F 9932 9966
47 0.8 1184.0 0.8X
-codegen = false, adaptive = T 5412 5618
291 1.6 645.1 1.5X
+codegen = true, adaptive = F 8112 8167
78 1.0 967.0 1.0X
+codegen = true, adaptive = T 4599 4619
28 1.8 548.2 1.8X
+codegen = false, adaptive = F 9269 9273
6 0.9 1104.9 0.9X
+codegen = false, adaptive = T 5384 5390
9 1.6 641.8 1.5X
================================================================================================
low-cardinality input, pass-through at the spill check
================================================================================================
-OpenJDK 64-Bit Server VM 25.0.4+7-LTS on Linux 6.17.0-1021-azure
+OpenJDK 64-Bit Server VM 25.0.5+8-LTS on Linux 6.17.0-1022-azure
AMD EPYC 7763 64-Core Processor
adaptive partial agg, low card, spill: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F 773 786
17 21.7 46.1 1.0X
-codegen = true, adaptive = T 790 803
20 21.2 47.1 1.0X
-codegen = false, adaptive = F 1370 1377
10 12.2 81.6 0.6X
-codegen = false, adaptive = T 1366 1373
11 12.3 81.4 0.6X
+codegen = true, adaptive = F 771 780
15 21.8 45.9 1.0X
+codegen = true, adaptive = T 809 820
16 20.7 48.2 1.0X
+codegen = false, adaptive = F 1387 1393
8 12.1 82.7 0.6X
+codegen = false, adaptive = T 1382 1383
2 12.1 82.4 0.6X
+
+
+================================================================================================
+spill check with an Exchange, keys arrive together
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.5+8-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys arrive together: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+---------------------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F 6107
6110 4 1.4 728.0 1.0X
+codegen = true, minCompactionToSpill = 1.05 6115
6146 43 1.4 729.0 1.0X
+codegen = true, minCompactionToSpill = 2.0 4771
4783 16 1.8 568.8 1.3X
+codegen = false, adaptive = F 7032
7047 21 1.2 838.3 0.9X
+codegen = false, minCompactionToSpill = 1.05 7068
7092 33 1.2 842.6 0.9X
+codegen = false, minCompactionToSpill = 2.0 5590
5609 26 1.5 666.4 1.1X
+
+
+================================================================================================
+spill check with an Exchange, keys recur
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.5+8-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys recur: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+-----------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F 2739 2768
41 3.1 326.5 1.0X
+codegen = true, minCompactionToSpill = 1.05 2699 2714
21 3.1 321.8 1.0X
+codegen = true, minCompactionToSpill = 2.0 2697 2720
34 3.1 321.4 1.0X
+codegen = false, adaptive = F 3689 3722
46 2.3 439.8 0.7X
+codegen = false, minCompactionToSpill = 1.05 3701 3713
18 2.3 441.2 0.7X
+codegen = false, minCompactionToSpill = 2.0 3703 3763
85 2.3 441.4 0.7X
+
+
+================================================================================================
+spill check with an Exchange, keys recur, map holds 40% of them, fast map off
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.5+8-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys recur, map holds 40% of them, fast map
off: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns)
Relative
+------------------------------------------------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F
8161 8252 129 2.1 486.4
1.0X
+codegen = true, minCompactionToSpill = 1.05
7881 8001 171 2.1 469.7
1.0X
+codegen = true, minCompactionToSpill = 2.0
7988 8021 46 2.1 476.1
1.0X
+codegen = false, adaptive = F
8664 8827 230 1.9 516.4
0.9X
+codegen = false, minCompactionToSpill = 1.05
8389 8546 222 2.0 500.0
1.0X
+codegen = false, minCompactionToSpill = 2.0
8867 8949 116 1.9 528.5
0.9X
+
+
+================================================================================================
+spill check with an Exchange, keys recur, maps hold 40% of them, fast map on
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.5+8-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys recur, maps hold 40% of them, fast map
on: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns)
Relative
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F
7317 7361 62 2.3 436.1
1.0X
+codegen = true, minCompactionToSpill = 1.05
7416 7426 15 2.3 442.0
1.0X
+codegen = true, minCompactionToSpill = 2.0
7905 7908 5 2.1 471.2
0.9X
+
+
+================================================================================================
+spill check with an Exchange, hot keys
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.5+8-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, hot keys: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+----------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F 6246 6264
26 1.3 744.6 1.0X
+codegen = true, minCompactionToSpill = 1.05 6317 6358
58 1.3 753.0 1.0X
+codegen = true, minCompactionToSpill = 2.0 4904 4922
26 1.7 584.6 1.3X
+codegen = false, adaptive = F 7212 7236
34 1.2 859.7 0.9X
+codegen = false, minCompactionToSpill = 1.05 7214 7269
78 1.2 860.0 0.9X
+codegen = false, minCompactionToSpill = 2.0 5579 5605
36 1.5 665.1 1.1X
diff --git
a/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-results.txt
b/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-results.txt
index 5523166ca01f..7335fe8e296c 100644
--- a/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-results.txt
+++ b/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-results.txt
@@ -2,55 +2,132 @@
high-cardinality input, pass-through at the periodic check
================================================================================================
-OpenJDK 64-Bit Server VM 17.0.20+8-LTS on Linux 6.17.0-1020-azure
+OpenJDK 64-Bit Server VM 17.0.21+9-LTS on Linux 6.17.0-1022-azure
AMD EPYC 7763 64-Core Processor
adaptive partial agg, high card, no spill: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F 4118 4125
11 2.0 490.9 1.0X
-codegen = true, adaptive = T 2411 2427
23 3.5 287.4 1.7X
-codegen = false, adaptive = F 4846 4861
21 1.7 577.7 0.8X
-codegen = false, adaptive = T 2991 3009
25 2.8 356.6 1.4X
+codegen = true, adaptive = F 4303 4326
32 1.9 513.0 1.0X
+codegen = true, adaptive = T 2621 2627
8 3.2 312.4 1.6X
+codegen = false, adaptive = F 5219 5228
12 1.6 622.2 0.8X
+codegen = false, adaptive = T 3259 3298
55 2.6 388.5 1.3X
================================================================================================
low-cardinality input, pass-through at the periodic check
================================================================================================
-OpenJDK 64-Bit Server VM 17.0.20+8-LTS on Linux 6.17.0-1020-azure
+OpenJDK 64-Bit Server VM 17.0.21+9-LTS on Linux 6.17.0-1022-azure
AMD EPYC 7763 64-Core Processor
adaptive partial agg, low card, no spill: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F 270 311
37 62.2 16.1 1.0X
-codegen = true, adaptive = T 293 318
22 57.3 17.5 0.9X
-codegen = false, adaptive = F 1305 1312
10 12.9 77.8 0.2X
-codegen = false, adaptive = T 1329 1435
150 12.6 79.2 0.2X
+codegen = true, adaptive = F 317 339
18 53.0 18.9 1.0X
+codegen = true, adaptive = T 300 321
21 56.0 17.9 1.1X
+codegen = false, adaptive = F 1299 1304
7 12.9 77.4 0.2X
+codegen = false, adaptive = T 1332 1349
24 12.6 79.4 0.2X
================================================================================================
high-cardinality input, pass-through at the spill check
================================================================================================
-OpenJDK 64-Bit Server VM 17.0.20+8-LTS on Linux 6.17.0-1020-azure
+OpenJDK 64-Bit Server VM 17.0.21+9-LTS on Linux 6.17.0-1022-azure
AMD EPYC 7763 64-Core Processor
adaptive partial agg, high card, spill: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F 8196 8247
73 1.0 977.0 1.0X
-codegen = true, adaptive = T 4433 4530
137 1.9 528.5 1.8X
-codegen = false, adaptive = F 9544 9555
15 0.9 1137.8 0.9X
-codegen = false, adaptive = T 5243 5395
215 1.6 625.0 1.6X
+codegen = true, adaptive = F 8760 8789
41 1.0 1044.2 1.0X
+codegen = true, adaptive = T 4692 4712
28 1.8 559.3 1.9X
+codegen = false, adaptive = F 10134 10142
11 0.8 1208.1 0.9X
+codegen = false, adaptive = T 5904 6003
140 1.4 703.8 1.5X
================================================================================================
low-cardinality input, pass-through at the spill check
================================================================================================
-OpenJDK 64-Bit Server VM 17.0.20+8-LTS on Linux 6.17.0-1020-azure
+OpenJDK 64-Bit Server VM 17.0.21+9-LTS on Linux 6.17.0-1022-azure
AMD EPYC 7763 64-Core Processor
adaptive partial agg, low card, spill: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F 803 813
11 20.9 47.8 1.0X
-codegen = true, adaptive = T 829 839
9 20.2 49.4 1.0X
-codegen = false, adaptive = F 1362 1387
35 12.3 81.2 0.6X
-codegen = false, adaptive = T 1365 1376
15 12.3 81.4 0.6X
+codegen = true, adaptive = F 845 856
12 19.9 50.3 1.0X
+codegen = true, adaptive = T 894 908
22 18.8 53.3 0.9X
+codegen = false, adaptive = F 1508 1521
18 11.1 89.9 0.6X
+codegen = false, adaptive = T 1487 1499
16 11.3 88.7 0.6X
+
+
+================================================================================================
+spill check with an Exchange, keys arrive together
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.21+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys arrive together: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+---------------------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F 6354
6448 134 1.3 757.4 1.0X
+codegen = true, minCompactionToSpill = 1.05 6816
6848 45 1.2 812.6 0.9X
+codegen = true, minCompactionToSpill = 2.0 4959
4979 27 1.7 591.2 1.3X
+codegen = false, adaptive = F 7335
7360 35 1.1 874.4 0.9X
+codegen = false, minCompactionToSpill = 1.05 7368
7444 108 1.1 878.3 0.9X
+codegen = false, minCompactionToSpill = 2.0 5710
5781 101 1.5 680.6 1.1X
+
+
+================================================================================================
+spill check with an Exchange, keys recur
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.21+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys recur: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+-----------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F 2712 2732
27 3.1 323.3 1.0X
+codegen = true, minCompactionToSpill = 1.05 2733 2784
73 3.1 325.7 1.0X
+codegen = true, minCompactionToSpill = 2.0 2722 2741
27 3.1 324.5 1.0X
+codegen = false, adaptive = F 3738 3745
10 2.2 445.6 0.7X
+codegen = false, minCompactionToSpill = 1.05 3728 3853
177 2.3 444.4 0.7X
+codegen = false, minCompactionToSpill = 2.0 3889 3897
11 2.2 463.6 0.7X
+
+
+================================================================================================
+spill check with an Exchange, keys recur, map holds 40% of them, fast map off
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.21+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys recur, map holds 40% of them, fast map
off: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns)
Relative
+------------------------------------------------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F
7715 7775 84 2.2 459.9
1.0X
+codegen = true, minCompactionToSpill = 1.05
7764 7785 30 2.2 462.8
1.0X
+codegen = true, minCompactionToSpill = 2.0
8011 8020 12 2.1 477.5
1.0X
+codegen = false, adaptive = F
8729 8839 156 1.9 520.3
0.9X
+codegen = false, minCompactionToSpill = 1.05
9086 9099 19 1.8 541.5
0.8X
+codegen = false, minCompactionToSpill = 2.0
9106 9155 68 1.8 542.8
0.8X
+
+
+================================================================================================
+spill check with an Exchange, keys recur, maps hold 40% of them, fast map on
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.21+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys recur, maps hold 40% of them, fast map
on: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns)
Relative
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F
7826 7860 48 2.1 466.5
1.0X
+codegen = true, minCompactionToSpill = 1.05
7637 7850 301 2.2 455.2
1.0X
+codegen = true, minCompactionToSpill = 2.0
8170 8221 72 2.1 487.0
1.0X
+
+
+================================================================================================
+spill check with an Exchange, hot keys
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.21+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, hot keys: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+----------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F 6443 6507
89 1.3 768.1 1.0X
+codegen = true, minCompactionToSpill = 1.05 6544 6556
17 1.3 780.1 1.0X
+codegen = true, minCompactionToSpill = 2.0 4985 4987
2 1.7 594.2 1.3X
+codegen = false, adaptive = F 7311 7320
13 1.1 871.6 0.9X
+codegen = false, minCompactionToSpill = 1.05 7310 7317
11 1.1 871.4 0.9X
+codegen = false, minCompactionToSpill = 2.0 5761 5790
41 1.5 686.8 1.1X
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
index ffb6773bfab8..7ed81e5818ff 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
@@ -140,7 +140,8 @@ case class HashAggregateExec(
numPassthroughRows,
adaptivePartialAggEnabled,
adaptiveMinRows,
- adaptiveMinCompaction)
+ adaptiveMinCompaction,
+ adaptiveMinCompactionToSpill)
if (!hasInput && groupingExpressions.isEmpty) {
numOutputRows += 1
Iterator.single[UnsafeRow](aggregationIterator.outputForEmptyGroupingKeyWithoutInput())
@@ -208,10 +209,15 @@ case class HashAggregateExec(
requiredChildDistributionExpressions.isEmpty
}
- // The number of rows between two compaction-ratio evaluations, and the
ratio below which the
- // partial aggregation is considered ineffective. Only read when the feature
applies.
+ // The number of rows between two compaction-ratio evaluations, and the
ratios below which the
+ // partial aggregation is considered ineffective: at the periodic check, and
when the map is
+ // about to spill. Keeping the aggregation past that point means aggregating
the rest of the
+ // input through spills, so the spill check never demands less compaction
than the periodic one.
+ // Only read when the feature applies.
private val adaptiveMinRows: Long = conf.adaptivePartialAggregationMinRows
private val adaptiveMinCompaction: Double =
conf.adaptivePartialAggregationMinCompaction
+ private val adaptiveMinCompactionToSpill: Double =
+ math.max(adaptiveMinCompaction,
conf.adaptivePartialAggregationMinCompactionToSpill)
// The name for Fast HashMap
private var fastHashMapTerm: String = _
@@ -253,6 +259,13 @@ case class HashAggregateExec(
// keeps both sides of the ratio on the same epoch. Once the fast map fills
it stops accepting
// keys, so the difference settles at zero and the ratio is the regular
map's alone.
private var adaptiveFastKeysAtSpillTerm: String = _
+ // The epoch's key count at its last periodic check, and at the check before
that, where the
+ // window of recent rows that the spill check judges starts (see
`adaptiveIneffective`). The
+ // checks run at multiples of `minRows`, and the next check row advances
only past a check that
+ // keeps aggregating, so the window starts `2 * minRows` rows before the
next check row, or at
+ // the epoch start before the second check.
+ private var adaptiveCheckKeysTerm: String = _
+ private var adaptiveWindowKeysTerm: String = _
// The name of the generated output function, promoted to a field so
`doConsumeWithKeys` can emit
// pass-through rows directly from within the build loop.
private var outputFunc: String = _
@@ -575,6 +588,8 @@ case class HashAggregateExec(
ctx.addMutableState(CodeGenerator.JAVA_BOOLEAN, "adaptiveMapSetupDone")
adaptiveFastKeysAtSpillTerm =
ctx.addMutableState(CodeGenerator.JAVA_INT, "adaptiveFastKeysAtSpill")
+ adaptiveCheckKeysTerm = ctx.addMutableState(CodeGenerator.JAVA_LONG,
"adaptiveCheckKeys")
+ adaptiveWindowKeysTerm = ctx.addMutableState(CodeGenerator.JAVA_LONG,
"adaptiveWindowKeys")
adaptivePendingRowsTerm = ctx.addMutableState(
"java.util.LinkedList<UnsafeRow[]>", "adaptivePendingRows",
v => s"$v = new java.util.LinkedList<UnsafeRow[]>();", forceInline =
true)
@@ -980,25 +995,31 @@ case class HashAggregateExec(
case _ => ("true", "", "")
}
- // The compaction ratio is measured at the operator level: all processed
rows against the keys
- // held by both maps, so two-level-map routing does not change the
decision. The same predicate
- // decides both check points -- periodically every `minRows` rows, and
right before the map
- // would spill (in which case the spill is skipped entirely). Pass-through
cannot be undone
- // within the task, so the spill check, like the periodic one, waits until
the current epoch
- // has processed `minRows` rows; a map that fills sooner spills as usual.
`minRows = 0`
- // disables the periodic check: the row count is only ever compared after
being incremented
- // past 0, so it never matches and only the spill check remains, without a
row floor.
- val adaptiveIneffective = if (adaptivePartialAggEnabled) {
- val totalKeys = if (isFastHashMapEnabled) {
- s"($fastHashMapTerm.getNumKeys() - $adaptiveFastKeysAtSpillTerm + " +
- s"$hashMapTerm.getNumKeys())"
- } else {
- s"$hashMapTerm.getNumKeys()"
- }
- s"$processedRowsTerm < (double) $totalKeys * ${adaptiveMinCompaction}D"
+ // The compaction ratio is measured at the operator level: processed rows
against the keys
+ // held by both maps, so two-level-map routing does not change the
decision. It is evaluated at
+ // two check points:
+ // - periodically, every `minRows` rows, over the whole epoch against
`minCompaction`;
+ // - right before the map would spill, over the recent rows against the
higher
+ // `minCompactionToSpill`, in which case the map does not spill and
the rest of the input
+ // is passed through. Keeping the aggregation then means aggregating
the rest of the input
+ // through spills, which pays off only if that input keeps repeating
keys. The recent rows,
+ // those since the periodic check before last, measure that: when keys
repeat across the
+ // input, more of them find their key already in the map as it fills,
so their ratio
+ // climbs above the epoch's; when each key's rows arrive together, the
two ratios match.
+ // Pass-through cannot be undone within the task, so the spill check, like
the periodic one,
+ // waits until the current epoch has processed `minRows` rows; a map that
fills sooner spills
+ // as usual. `minRows = 0` disables the periodic check: the row count is
only ever compared
+ // after being incremented past 0, so it never matches and only the spill
check remains,
+ // without a row floor and over the whole epoch.
+ def adaptiveEpochKeys: String = if (isFastHashMapEnabled) {
+ s"((long) $fastHashMapTerm.getNumKeys() - $adaptiveFastKeysAtSpillTerm +
" +
+ s"$hashMapTerm.getNumKeys())"
} else {
- ""
+ s"((long) $hashMapTerm.getNumKeys())"
}
+ // Whether `rows` processed rows fail to collapse into `keys` keys by at
least `threshold`.
+ def adaptiveIneffective(rows: String, keys: String, threshold: Double):
String =
+ s"$rows < (double) $keys * ${threshold}D"
val findOrInsertRegularHashMap: String = {
// Assumes the grouping key projection (`unsafeRowKeyCode.code`) has
already run for this row,
@@ -1055,7 +1076,14 @@ case class HashAggregateExec(
|$processedRowsTerm = 0L;
|$adaptiveNextCheckRowTerm = ${adaptiveMinRows}L;
|$snapshotFastKeys
+ |$adaptiveCheckKeysTerm = 0L;
+ |$adaptiveWindowKeysTerm = 0L;
""".stripMargin
+ val windowRows = s"Math.max(0L, $adaptiveNextCheckRowTerm - ${2 *
adaptiveMinRows}L)"
+ val recentIneffective = adaptiveIneffective(
+ s"($processedRowsTerm - $windowRows)",
+ s"($adaptiveEpochKeys - $adaptiveWindowKeysTerm)",
+ adaptiveMinCompactionToSpill)
// The spill check needs at least one processed row even when `minRows
= 0`.
val spillCheckMinRows = math.max(adaptiveMinRows, 1L)
s"""
@@ -1064,7 +1092,7 @@ case class HashAggregateExec(
|if (!$adaptivePassThroughTerm) {
| $probeRegularMap
| if ($unsafeRowBuffer == null) {
- | if ($processedRowsTerm >= ${spillCheckMinRows}L &&
$adaptiveIneffective) {
+ | if ($processedRowsTerm >= ${spillCheckMinRows}L &&
$recentIneffective) {
| $adaptivePassThroughTerm = true;
| } else {
| $spillAndRestartEpoch
@@ -1138,6 +1166,8 @@ case class HashAggregateExec(
} else {
s"($unsafeRowBuffer != null)"
}
+ val epochIneffective =
+ adaptiveIneffective(processedRowsTerm, adaptiveEpochKeys,
adaptiveMinCompaction)
// The grouping key was already projected in
`findOrInsertRegularHashMap`
// (`unsafeRowKeyCode.code`), so `unsafeRowKeyCode.value` holds this
row's key. Only build
// the single-row partial buffer here: reset the reusable buffer to
the initial aggregation
@@ -1148,10 +1178,12 @@ case class HashAggregateExec(
| if (!$adaptivePassThroughTerm) {
| $processedRowsTerm += 1;
| if ($processedRowsTerm == $adaptiveNextCheckRowTerm) {
- | if ($adaptiveIneffective) {
+ | if ($epochIneffective) {
| $adaptivePassThroughTerm = true;
| } else {
| $adaptiveNextCheckRowTerm += ${adaptiveMinRows}L;
+ | $adaptiveWindowKeysTerm = $adaptiveCheckKeysTerm;
+ | $adaptiveCheckKeysTerm = $adaptiveEpochKeys;
| }
| }
| }
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala
index c5f538841505..82806ecccd8a 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala
@@ -100,7 +100,8 @@ class TungstenAggregationIterator(
numPassthroughRows: SQLMetric,
adaptivePartialAggEnabled: Boolean,
adaptiveMinRows: Long,
- adaptiveMinCompaction: Double)
+ adaptiveMinCompaction: Double,
+ adaptiveMinCompactionToSpill: Double)
extends AggregationIterator(
partIndex,
groupingExpressions,
@@ -189,14 +190,18 @@ class TungstenAggregationIterator(
// When adaptive partial aggregation is enabled
(`adaptivePartialAggEnabled`), the processing may
// stop early and switch to pass-through mode: the remaining input rows are
not added to the map
// but are instead emitted as single-row partial buffers by the output stage
(see
- // `nextPassThroughOutput`). One predicate decides both check points -- the
aggregation is
- // ineffective when `processedRows < distinctKeys * minCompaction`, i.e. it
does not collapse
- // `minCompaction` rows into one key:
- // - periodically, every `minRows` processed rows;
- // - when the map is about to spill, in which case the spill is skipped
entirely and the row
- // that could not be inserted becomes the first pass-through row.
Pass-through cannot be
- // undone within the task, so like the periodic check this waits until
the current epoch has
- // processed `minRows` rows; a map that fills sooner spills as usual.
+ // `nextPassThroughOutput`). The aggregation is ineffective when `rows`
processed rows do not
+ // collapse into their `keys` keys by at least `threshold`, checked at two
points:
+ // - periodically, every `minRows` processed rows, over the whole epoch
against
+ // `minCompaction`;
+ // - when the map is about to spill, over the recent rows (those since the
periodic check
+ // before last) against the higher `minCompactionToSpill`. Keeping the
aggregation then
+ // means aggregating the rest of the input through spills, which pays
off only if that input
+ // keeps repeating keys; when keys repeat across the input, the ratio of
the recent rows
+ // climbs above the epoch's as the map fills. Below the threshold, the
map does not spill
+ // and the row that could not be inserted becomes the first pass-through
row. Pass-through
+ // cannot be undone within the task, so like the periodic check this
waits until the current
+ // epoch has processed `minRows` rows; a map that fills sooner spills as
usual.
// A spill starts a new in-memory map epoch: the row counters restart so
that epoch is judged on
// its own rows, which lets an input whose cardinality only turns
unfavorable later still be
// passed through. Pass-through may therefore coexist with earlier spills.
The output order
@@ -225,10 +230,18 @@ class TungstenAggregationIterator(
// The spill check waits for `minRows` rows of the current epoch, and
for at least one row
// when `minRows = 0`.
val spillCheckMinRows = math.max(minRows, 1L)
- // The partial aggregation is ineffective when it does not collapse
`minCompaction` rows into
- // one key. There is no fast map on this path, so the map's keys are all
the operator holds.
- def ineffective(): Boolean =
- processedRows < hashMap.getNumKeys().toDouble * adaptiveMinCompaction
+ // The epoch's key count at its last periodic check, and at the check
before that, where the
+ // window of recent rows that the spill check judges starts. The checks
run at multiples of
+ // `minRows`, and `nextCheckRow` advances only past a check that keeps
aggregating, so the
+ // window starts `2 * minRows` rows before `nextCheckRow`, or at the
epoch start before the
+ // second check.
+ var checkKeys = 0L
+ var windowKeys = 0L
+ def windowRows: Long = math.max(0L, nextCheckRow - 2 * minRows)
+ // Whether `rows` processed rows fail to collapse into `keys` keys by at
least `threshold`.
+ // There is no fast map on this path, so the map's keys are all the
operator holds.
+ def ineffective(rows: Long, keys: Long, threshold: Double): Boolean =
+ rows < keys.toDouble * threshold
while (inputIter.hasNext && !passThrough) {
val newInput = inputIter.next()
val groupingKey = groupingProjection.apply(newInput)
@@ -240,7 +253,9 @@ class TungstenAggregationIterator(
// The map is full and would normally spill. Adaptive partial
aggregation may instead
// bypass: keep the in-memory map as-is, pass this row and all
remaining rows through,
// and skip the spill entirely.
- if (adaptivePartialAggEnabled && processedRows >= spillCheckMinRows
&& ineffective()) {
+ if (adaptivePartialAggEnabled && processedRows >= spillCheckMinRows
&&
+ ineffective(processedRows - windowRows, hashMap.getNumKeys() -
windowKeys,
+ adaptiveMinCompactionToSpill)) {
passThrough = true
// `newInput` could not be inserted; stash a copy as the first
pass-through row so it
// is not lost when we drain the rest of `inputIter`.
@@ -257,6 +272,8 @@ class TungstenAggregationIterator(
// epoch alone decides whether the remaining rows are passed
through.
processedRows = 0L
nextCheckRow = minRows
+ checkKeys = 0L
+ windowKeys = 0L
buffer = hashMap.getAggregationBufferFromUnsafeRow(groupingKey)
if (buffer == null) {
// failed to allocate the first page
@@ -271,10 +288,12 @@ class TungstenAggregationIterator(
// Periodic check: if the aggregation is not collapsing enough rows,
bypass it for the
// rest of the input.
if (adaptivePartialAggEnabled && processedRows == nextCheckRow) {
- if (ineffective()) {
+ if (ineffective(processedRows, hashMap.getNumKeys(),
adaptiveMinCompaction)) {
passThrough = true
} else {
nextCheckRow += minRows
+ windowKeys = checkKeys
+ checkKeys = hashMap.getNumKeys()
}
}
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/aggregate/AdaptivePartialAggregationSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/aggregate/AdaptivePartialAggregationSuite.scala
index c4240ca308cf..d88b24cb8510 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/aggregate/AdaptivePartialAggregationSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/aggregate/AdaptivePartialAggregationSuite.scala
@@ -714,32 +714,39 @@ class AdaptivePartialAggregationSuite extends QueryTest
with SharedSparkSession
// compaction ratio high enough that they decline to bypass -- they cover
the eligible-but-
// declining side. (Widening the rollup lowers the ratio: with five distinct
columns the same
// shape does bypass.) The GROUPING SETS and multi-distinct tests, and
`pass-through fires for
- // high-cardinality input below an Expand`, cover an Expand that bypasses.
+ // high-cardinality input below an Expand`, cover an Expand that bypasses.
The forced-spill
+ // epochs of ROLLUP and CUBE compact too little to keep aggregating past a
spill under the
+ // default `minCompactionToSpill`, which rightly passes them through, so
they pin it to 1.0 to
+ // judge the spill check by `minCompaction` as well.
test("results unchanged for ROLLUP (Expand below partial aggregate)") {
// The grand-total group repeats on every expanded row, so the compaction
ratio stays above the
// threshold and the partial aggregate declines to bypass in every cell.
- checkAdaptiveMatchesReference(
- expectBypass = false,
- build = { parts =>
- spark.range(0, 400, 1, parts)
- .select(($"id" % 200) as "k1", ($"id" % 7) as "k2", $"id" as "v")
- .rollup($"k1", $"k2")
- .agg(sum($"v") as "s", count(lit(1)) as "c")
- })
+
withSQLConf(SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key ->
"1.0") {
+ checkAdaptiveMatchesReference(
+ expectBypass = false,
+ build = { parts =>
+ spark.range(0, 400, 1, parts)
+ .select(($"id" % 200) as "k1", ($"id" % 7) as "k2", $"id" as "v")
+ .rollup($"k1", $"k2")
+ .agg(sum($"v") as "s", count(lit(1)) as "c")
+ })
+ }
}
test("results unchanged for CUBE (Expand below partial aggregate)") {
// Same as ROLLUP: the grand-total group keeps the ratio above the
threshold, so nothing
// bypasses.
- checkAdaptiveMatchesReference(
- expectBypass = false,
- build = { parts =>
- spark.range(0, 400, 1, parts)
- .select(($"id" % 150) as "k1", ($"id" % 5) as "k2", $"id" as "v")
- .cube($"k1", $"k2")
- .agg(sum($"v") as "s", count(lit(1)) as "c")
- })
+
withSQLConf(SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key ->
"1.0") {
+ checkAdaptiveMatchesReference(
+ expectBypass = false,
+ build = { parts =>
+ spark.range(0, 400, 1, parts)
+ .select(($"id" % 150) as "k1", ($"id" % 5) as "k2", $"id" as "v")
+ .cube($"k1", $"k2")
+ .agg(sum($"v") as "s", count(lit(1)) as "c")
+ })
+ }
}
test("results unchanged for GROUPING SETS (Expand below partial aggregate)")
{
@@ -1171,11 +1178,12 @@ class AdaptivePartialAggregationSuite extends QueryTest
with SharedSparkSession
test("the spill check bypasses between periodic checks once minRows rows are
processed") {
// With `minRows = 10` the periodic check runs at the 10th aggregated row,
when ids 0-9 have
- // collapsed into two keys (ratio 5.0, above `minCompaction = 2.0`), so it
keeps aggregating.
- // Ids 10-18 then add nine distinct keys, and the forced fallback makes
the map full when id
- // 19 arrives. That is past the floor but before the next periodic check
at 20 rows, and the
- // ratio of 19 rows over 11 keys is below 2.0, so the spill check bypasses
at exactly that row:
- // ids 19-399 stream through and the sorter is never created.
+ // collapsed into two keys (ratio 5.0, above `minCompaction`), so it keeps
aggregating. Ids
+ // 10-18 then add nine distinct keys, and the forced fallback makes the
map full when id 19
+ // arrives. That is past the floor but before the next periodic check at
20 rows. With a single
+ // periodic check so far, the spill check judges the whole epoch, and 19
rows over 11 keys is
+ // below `minCompactionToSpill = 2.0`, so it bypasses at exactly that row:
ids 19-399 stream
+ // through and the sorter is never created.
Seq(true, false).foreach { wholeStage =>
withSQLConf(
(Seq(
@@ -1183,7 +1191,7 @@ class AdaptivePartialAggregationSuite extends QueryTest
with SharedSparkSession
SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> wholeStage.toString,
SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> "false",
SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key -> "10",
- SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION.key -> "2.0",
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key ->
"2.0",
"spark.sql.TungstenAggregate.testFallbackStartsAt" -> "1, 19") ++
fixedPlanConfs): _*) {
val df = () => spark.range(0, 400, 1, 1)
@@ -1207,12 +1215,104 @@ class AdaptivePartialAggregationSuite extends
QueryTest with SharedSparkSession
}
}
+ test("the spill check passes through an epoch that compacts too little to
spill") {
+ // Keys `floor(2 * id / 3)` collapse every three rows into two keys, a
steady ratio of about
+ // 1.5 with no key repeating once its rows have passed. That is above
`minCompaction`, so the
+ // periodic checks at 10, 20 and 30 rows keep aggregating. The forced
fallback fills the map
+ // when the next row arrives, and the spill check judges the rows since
the check before last,
+ // which compact about the same, 1.4 to 1.7. Keeping the aggregation would
aggregate the rest
+ // through spills for that little, so the default `minCompactionToSpill`
(2.0) passes the rest
+ // through instead and the sorter is never created. Lowering it to 1.3
keeps the old behavior:
+ // every epoch spills, and nothing is passed through. Two input partitions
put an `Exchange`
+ // between the Partial and the Final, so passed-through rows go through
the output buffer and
+ // the resumed build; one partition fuses the two.
+ for {
+ (minCompactionToSpill, shouldBypass) <- Seq(None -> true, Some("1.3") ->
false)
+ partitions <- Seq(1, 2)
+ wholeStage <- Seq(true, false)
+ } {
+ withSQLConf(
+ (Seq(
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key -> "true",
+ SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> wholeStage.toString,
+ SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> "false",
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key -> "10",
+ "spark.sql.TungstenAggregate.testFallbackStartsAt" -> "1, 30") ++
+ minCompactionToSpill.map(
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key
-> _) ++
+ fixedPlanConfs): _*) {
+ val df = () => spark.range(0, 400, 1, partitions)
+ .select(($"id" * 2 / 3).cast("long").cast("string") as "k", $"id" as
"v")
+ .groupBy($"k")
+ .agg(sum($"v") as "s")
+ withClue(s"minCompactionToSpill=$minCompactionToSpill
partitions=$partitions " +
+ s"wholeStage=$wholeStage") {
+ val c = runAndReadCounters(df)
+ if (shouldBypass) {
+ assert(c.passedThrough == 400 - 30 * partitions,
+ "each task should pass through from the row that found its map
full")
+ assert(c.tasksFallBacked == 0,
+ "the spill check must replace the sort fallback, not trigger it")
+ } else {
+ assert(c.passedThrough == 0,
+ "an epoch above minCompactionToSpill must keep aggregating")
+ assert(c.tasksFallBacked > 0, "an epoch that keeps aggregating
must spill")
+ }
+ }
+ }
+ }
+ }
+
+ test("the spill check keeps aggregating when the recent rows repeat keys") {
+ // Ids 0-35 map to `floor(2 * id / 3)`, keys 0-23, and ids 36-53 to `id %
20`, keys the map
+ // already holds. The periodic checks at 10, 20, 30 and 40 rows see ratios
from 1.4 to 1.7, so
+ // they keep aggregating, and the forced fallback fills the map when id 45
arrives, after 45
+ // rows over 24 keys. That ratio of 1.875 is below `minCompactionToSpill`,
but the spill check
+ // judges the rows since the check before last: the 15 rows since the
check at 30 rows added
+ // only 4 keys, a ratio of 3.75, because the rest found their key in the
map. So the map spills
+ // and keeps aggregating, and the last 9 rows are too few for another
check. Demanding 4.0 at
+ // the spill check passes those 9 rows through instead.
+ for {
+ (minCompactionToSpill, passedThrough) <- Seq(None -> 0, Some("4.0") -> 9)
+ wholeStage <- Seq(true, false)
+ } {
+ withSQLConf(
+ (Seq(
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key -> "true",
+ SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> wholeStage.toString,
+ SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> "false",
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key -> "10",
+ "spark.sql.TungstenAggregate.testFallbackStartsAt" -> "1, 45") ++
+ minCompactionToSpill.map(
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key
-> _) ++
+ fixedPlanConfs): _*) {
+ val df = () => spark.range(0, 54, 1, 1)
+ .select(when($"id" < 36, ($"id" * 2 /
3).cast("long")).otherwise($"id" % 20)
+ .cast("string") as "k", $"id" as "v")
+ .groupBy($"k")
+ .agg(sum($"v") as "s")
+ withClue(s"minCompactionToSpill=$minCompactionToSpill
wholeStage=$wholeStage") {
+ val c = runAndReadCounters(df)
+ assert(c.passedThrough == passedThrough)
+ assert((c.tasksFallBacked > 0) == (passedThrough == 0),
+ "the map spills exactly when the spill check keeps aggregating")
+ }
+ }
+ }
+ }
+
test("a new in-memory map epoch after a spill can still bypass") {
// A spill starts a new in-memory map epoch and restarts the row counters,
so an input whose
- // cardinality only turns unfavorable after an early spill is still
caught. The first 100 rows
+ // cardinality only turns unfavorable after an early spill is still
caught. The first 80 rows
// repeat 5 keys and keep the aggregation effective while the forced
fallback makes the map
- // spill; the remaining 300 rows are fully distinct, so the new epoch is
judged ineffective and
- // the rest of the input is passed through. Both the spill and the bypass
must be observable.
+ // spill at ids 40 and 80; the remaining 320 rows are fully distinct, so
the third epoch's
+ // first periodic check, at its 8th row (id 87), judges it ineffective and
the rest of the
+ // input is passed through. With the two-level map on the codegen path,
the fast map absorbs
+ // keys 0-3, so each forced spill waits for the next row that needs the
regular map, and the
+ // third epoch starts at id 84 instead. The exact count also pins that the
check row restarts
+ // with the epoch. `minCompactionToSpill` is pinned to 1.5 so the count
does not rest on a tie:
+ // in the two-level codegen cell, the window when id 84 finds the map full
holds 8 rows over 4
+ // keys, exactly the default 2.0.
for {
wholeStage <- Seq(true, false)
twoLevelMap <- Seq(true, false)
@@ -1224,18 +1324,20 @@ class AdaptivePartialAggregationSuite extends QueryTest
with SharedSparkSession
SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> wholeStage.toString,
SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> twoLevelMap.toString,
SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key -> "8",
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key ->
"1.5",
"spark.sql.TungstenAggregate.testFallbackStartsAt" -> s"$fastCap,
40") ++
fixedPlanConfs): _*) {
val df = () => spark.range(0, 400, 1, 1)
- .select(when($"id" < 100, $"id" % 5).otherwise($"id") as "k", $"id"
as "v")
+ .select(when($"id" < 80, $"id" % 5).otherwise($"id") as "k", $"id"
as "v")
.groupBy($"k")
.agg(sum($"v") as "s")
withClue(s"wholeStage=$wholeStage twoLevelMap=$twoLevelMap") {
val c = runAndReadCounters(df)
assert(c.tasksFallBacked > 0,
"the low-cardinality prefix should still spill and fall back to
sort")
- assert(c.passedThrough > 0,
- "the high-cardinality tail after the spill should be passed
through")
+ val firstPassedThrough = if (wholeStage && twoLevelMap) 92 else 88
+ assert(c.passedThrough == 400 - firstPassedThrough,
+ "the high-cardinality tail after the spills should be passed
through")
}
}
}
@@ -1278,8 +1380,15 @@ class AdaptivePartialAggregationSuite extends QueryTest
with SharedSparkSession
// failed insertion that becomes the first pass-through row, so both
execution paths must judge
// the same row set and reach the same decision. `id % 40` over 400 rows
gives 40 keys when the
// map fills at 50 aggregated rows, i.e. a compaction ratio of exactly
1.25: demanding 1.25 the
- // aggregation is kept (`50 < 40 * 1.25` is false), demanding 1.3 it is
bypassed.
- Seq(1.25 -> false, 1.3 -> true).foreach { case (minCompaction,
shouldBypass) =>
+ // aggregation is kept (`50 < 40 * 1.25` is false), demanding 1.3 it is
bypassed. The spill
+ // check demands the larger of `minCompaction` and `minCompactionToSpill`,
so either one can
+ // set the boundary, from either side.
+ Seq(
+ (1.0, 1.25, false),
+ (1.0, 1.3, true),
+ (1.25, 1.0, false),
+ (1.3, 1.0, true)
+ ).foreach { case (minCompaction, minCompactionToSpill, shouldBypass) =>
val passedThroughPerCodegen = Seq(true, false).map { wholeStage =>
withSQLConf(
(Seq(
@@ -1287,21 +1396,25 @@ class AdaptivePartialAggregationSuite extends QueryTest
with SharedSparkSession
SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> wholeStage.toString,
SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> "false",
// `minRows = 0` keeps the periodic check out of the picture and
leaves the spill check
- // without a row floor, so the spill check alone decides at the
boundary.
+ // without a row floor and judging the whole epoch, so it alone
decides at the boundary.
SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key -> "0",
SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION.key ->
minCompaction.toString,
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key ->
+ minCompactionToSpill.toString,
"spark.sql.TungstenAggregate.testFallbackStartsAt" -> "1, 50") ++
fixedPlanConfs): _*) {
val df = () => spark.range(0, 400, 1, 1)
.select(($"id" % 40).cast("string") as "k", $"id" as "v")
.groupBy($"k")
.agg(sum($"v") as "s")
- withClue(s"minCompaction=$minCompaction wholeStage=$wholeStage") {
+ withClue(s"minCompaction=$minCompaction
minCompactionToSpill=$minCompactionToSpill " +
+ s"wholeStage=$wholeStage") {
numPassthroughRows(df)
}
}
}
- withClue(s"minCompaction=$minCompaction
passedThrough=$passedThroughPerCodegen") {
+ withClue(s"minCompaction=$minCompaction
minCompactionToSpill=$minCompactionToSpill " +
+ s"passedThrough=$passedThroughPerCodegen") {
assert(passedThroughPerCodegen.forall(_ > 0) == shouldBypass,
s"expected bypass=$shouldBypass at the ratio boundary")
assert(passedThroughPerCodegen.map(_ > 0).distinct.length == 1,
@@ -1310,6 +1423,114 @@ class AdaptivePartialAggregationSuite extends QueryTest
with SharedSparkSession
}
}
+ test("the spill check judges the recent rows at their exact ratio boundary")
{
+ // With `minRows = 10`, ids 0-9 add 5 keys, two rows each, ids 10-19 add
10 more, one each, and
+ // ids 20-29 add 5 more, two rows each, so the periodic checks at 10, 20
and 30 rows see 2.0,
+ // 1.33 and 1.5 and keep aggregating. The check at 30 rows runs on id 29,
so when id 30 finds
+ // the map full, the rows since the check before last are ids 20-29: 10
rows over 5 keys, a
+ // ratio of exactly 2.0. Demanding 2.0 the aggregation is kept (`10 < 5 *
2.0` is false) and
+ // the map spills; demanding 2.1 the last 10 rows are passed through. A
window starting one
+ // check earlier (20 rows over 15 keys) or the whole epoch (30 rows over
20 keys) would pass
+ // them through at both.
+ for {
+ (minCompactionToSpill, passedThrough) <- Seq(2.0 -> 0, 2.1 -> 10)
+ wholeStage <- Seq(true, false)
+ } {
+ withSQLConf(
+ (Seq(
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key -> "true",
+ SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> wholeStage.toString,
+ SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> "false",
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key -> "10",
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key ->
+ minCompactionToSpill.toString,
+ "spark.sql.TungstenAggregate.testFallbackStartsAt" -> "1, 30") ++
+ fixedPlanConfs): _*) {
+ val df = () => spark.range(0, 40, 1, 1)
+ .select(when($"id" < 10, $"id" / 2).when($"id" < 20, $"id" - 5)
+ .when($"id" < 30, ($"id" - 20) / 2 + 15).otherwise(($"id" - 30) /
2 + 20)
+ .cast("long").cast("string") as "k", $"id" as "v")
+ .groupBy($"k")
+ .agg(sum($"v") as "s")
+ withClue(s"minCompactionToSpill=$minCompactionToSpill
wholeStage=$wholeStage") {
+ val c = runAndReadCounters(df)
+ assert(c.passedThrough == passedThrough)
+ assert((c.tasksFallBacked > 0) == (passedThrough == 0),
+ "the map spills exactly when the spill check keeps aggregating")
+ }
+ }
+ }
+ }
+
+ test("the spill check of a new map epoch does not count the previous epoch's
keys") {
+ // With `minRows = 10`, the forced fallback fills every epoch at 19 rows,
after one periodic
+ // check, so the spill check judges the whole epoch. The first epoch, ids
0-18 over 5 keys,
+ // compacts by 3.8 and spills. The second, ids 19-37, has 8 keys at its
check at 10 rows and
+ // 12 at the fill, so it compacts by 19 / 12 and the last 12 rows are
passed through. Keeping
+ // the key count of the first epoch's check as the start of the second
epoch's window would
+ // judge 19 rows over 7 keys and keep aggregating.
+ Seq(true, false).foreach { wholeStage =>
+ withSQLConf(
+ (Seq(
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key -> "true",
+ SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> wholeStage.toString,
+ SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> "false",
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key -> "10",
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key ->
"2.0",
+ "spark.sql.TungstenAggregate.testFallbackStartsAt" -> "1, 19") ++
+ fixedPlanConfs): _*) {
+ val df = () => spark.range(0, 50, 1, 1)
+ .select(when($"id" < 19, $"id" % 5).when($"id" < 29, ($"id" - 19) %
8 + 100)
+ .when($"id" < 38, ($"id" - 29) % 4 + 108).otherwise(0L)
+ .cast("string") as "k", $"id" as "v")
+ .groupBy($"k")
+ .agg(sum($"v") as "s")
+ withClue(s"wholeStage=$wholeStage") {
+ val c = runAndReadCounters(df)
+ assert(c.passedThrough == 12)
+ assert(c.tasksFallBacked > 0, "the first epoch should spill")
+ }
+ }
+ }
+ }
+
+ test("the spill check's window counts the keys of both maps") {
+ // With the two-level map on the codegen path and a 16-key fast map, ids
0-9 add 10 keys and
+ // ids 10-19 add 5 more, two rows each, all held by the fast map, and id
20 fills it. Id 21
+ // needs the regular map, which the forced fallback reports full after 20
rows, so the spill
+ // check judges the rows since the check before last, at 10 rows: ids
10-20, 11 rows over 6
+ // keys, a ratio of 1.83. Demanding 1.5 the aggregation is kept and the
map spills; demanding
+ // 1.9 the last 9 rows are passed through. Leaving the fast map's keys out
of the window's
+ // start would judge 11 rows over 16 keys and pass them through at both,
while counting them
+ // twice would keep aggregating at both. `minCompaction = 1.0` keeps the
periodic check at 10
+ // rows, 10 rows over 10 keys, aggregating.
+ Seq(1.5 -> 0, 1.9 -> 9).foreach { case (minCompactionToSpill,
passedThrough) =>
+ withSQLConf(
+ (Seq(
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key -> "true",
+ SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> "true",
+ SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> "true",
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key -> "10",
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION.key -> "1.0",
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key ->
+ minCompactionToSpill.toString,
+ "spark.sql.TungstenAggregate.testFallbackStartsAt" -> "16, 20") ++
+ fixedPlanConfs): _*) {
+ val df = () => spark.range(0, 30, 1, 1)
+ .select(when($"id" < 10, $"id").when($"id" < 20, ($"id" - 10) / 2 +
10)
+ .when($"id" === 20, 15L).otherwise($"id" + 16).cast("long") as
"k", $"id" as "v")
+ .groupBy($"k")
+ .agg(sum($"v") as "s")
+ withClue(s"minCompactionToSpill=$minCompactionToSpill") {
+ val c = runAndReadCounters(df)
+ assert(c.passedThrough == passedThrough)
+ assert((c.tasksFallBacked > 0) == (passedThrough == 0),
+ "the map spills exactly when the spill check keeps aggregating")
+ }
+ }
+ }
+ }
+
test("a very high minCompaction always bypasses once minRows has been
processed") {
// Demanding an unreachable compaction ratio is the most aggressive
setting: even a
// low-cardinality input that the default threshold keeps aggregating is
bypassed at the first
@@ -1361,17 +1582,21 @@ class AdaptivePartialAggregationSuite extends QueryTest
with SharedSparkSession
}
}
- test("minCompaction rejects values that cannot be generated as a Java
literal") {
+ test("the compaction thresholds reject values that cannot be generated as a
Java literal") {
// The threshold is interpolated straight into the generated source, where
a non-finite double
// renders as `InfinityD` -- not a valid Java literal. Reject it at
configuration time rather
// than failing to compile the stage (or silently deoptimizing when
codegen falls back).
- Seq("Infinity", "-Infinity", "NaN", "1e309").foreach { v =>
+ for {
+ key <- Seq(SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION.key,
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key)
+ v <- Seq("Infinity", "-Infinity", "NaN", "1e309")
+ } {
val e = intercept[IllegalArgumentException] {
-
spark.conf.set(SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION.key, v)
+ spark.conf.set(key, v)
}
- assert(e.getMessage.contains("finite"), s"expected a finiteness error
for '$v': $e")
+ assert(e.getMessage.contains("finite"), s"expected a finiteness error
for $key='$v': $e")
+ spark.conf.unset(key)
}
- spark.conf.unset(SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION.key)
}
test("a larger minRows defers the decision so a small high-cardinality input
is not bypassed") {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/AdaptivePartialAggregationBenchmark.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/AdaptivePartialAggregationBenchmark.scala
index 7c23d5e296bc..01121e6a28e4 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/AdaptivePartialAggregationBenchmark.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/AdaptivePartialAggregationBenchmark.scala
@@ -28,7 +28,7 @@ import org.apache.spark.sql.internal.SQLConf
* remaining rows through as single-row partial buffers instead of maintaining
(and possibly
* spilling) a large aggregation map.
*
- * Each scenario runs the query across the full matrix of whole-stage codegen
on/off and the
+ * The first scenarios run the query across the full matrix of whole-stage
codegen on/off and the
* feature disabled (`adaptive = F`, the pre-change baseline) vs enabled
(`adaptive = T`), over a
* {high, low}-cardinality x {no-spill, on-spill} grid:
* - high-cardinality, no spill: the periodic check bypasses, which should
win.
@@ -39,6 +39,11 @@ import org.apache.spark.sql.internal.SQLConf
* - low-cardinality, forced regular-map spill: the compaction ratio is far
above the threshold,
* so the spill check keeps aggregating and both runs spill identically
(no regression).
*
+ * The spill-check scenarios read four partitions, so an `Exchange` separates
the Partial from the
+ * Final, force the map to fill every few hundred thousand rows, and compare
the feature disabled
+ * with `minCompactionToSpill` at 1.05 and at its default, 2.0, over inputs
whose keys arrive
+ * together, recur across the task, or concentrate on a few hot keys.
+ *
* To run this benchmark:
* {{{
* 1. build/sbt "sql/Test/runMain
@@ -135,6 +140,67 @@ object AdaptivePartialAggregationBenchmark extends
SqlBasedBenchmark {
extraConf = spillCheckConf)
benchmark.run()
}
+
+ // Inputs that compact by about 1.5 within each map epoch, read from four
partitions so an
+ // `Exchange` separates the Partial from the Final, and with a map forced
to fill every few
+ // hundred thousand rows. Whether keeping the aggregation through the
spills pays off depends
+ // on whether the rest of the input keeps repeating keys:
+ // - each key's rows arrive together, so later epochs share no keys and
passing the rest
+ // through at the first fill avoids spilling every epoch;
+ // - keys recur uniformly across the task (10 rows per key), so the
final merge collapses
+ // them and keeping the aggregation shuffles a tenth of the rows;
+ // - the same over twice the rows and keys, so the maps fill when they
hold 40% of a task's
+ // keys: the recent rows compact by about 1.5 and the default passes
the rest through,
+ // although each key repeats about 10 times. It runs with the fast map
off, and with it on,
+ // where the fast map's keys never spill. The interpreted path has no
fast map, so the
+ // latter runs with codegen only;
+ // - 10 hot keys hold 30% of the rows and the rest are unique, so
passing through avoids the
+ // spills but sends every hot-key row to the reducer of its key.
+ // `minCompactionToSpill = 1.05` judges the spill check by
`minCompaction`; the default, 2.0,
+ // demands that the recent rows compact by 2.0.
+ case class SpillCheckCase(name: String, key: String, fillRows: Int,
+ numRows: Long = 8L << 20, twoLevelMap: Boolean = true, codegenOnly:
Boolean = false)
+ val spillCheckCases = Seq(
+ SpillCheckCase("keys arrive together", "cast(id * 2 / 3 as bigint)",
300000),
+ SpillCheckCase("keys recur", "pmod(xxhash64(id), 200000)", 220000),
+ SpillCheckCase("keys recur, map holds 40% of them, fast map off",
+ "pmod(xxhash64(id), 400000)", 204330, numRows = 16L << 20, twoLevelMap
= false),
+ SpillCheckCase("keys recur, maps hold 40% of them, fast map on",
+ "pmod(xxhash64(id), 400000)", 204330, numRows = 16L << 20, codegenOnly
= true),
+ SpillCheckCase("hot keys", "if(id % 100 < 30, id % 10, id + 1000)",
300000))
+ spillCheckCases.foreach { c =>
+ runBenchmark(s"spill check with an Exchange, ${c.name}") {
+ val N = c.numRows
+ val benchmark = new Benchmark(s"adaptive partial agg, spill check,
${c.name}", N,
+ output = output)
+ val query = () => spark.range(0, N, 1, 4).selectExpr(s"${c.key} as k",
"id as v")
+ .groupBy("k").agg("v" -> "sum")
+ val spillConf = Seq(
+ SQLConf.SHUFFLE_PARTITIONS.key -> "8",
+ SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> c.twoLevelMap.toString,
+ "spark.sql.TungstenAggregate.testFallbackStartsAt" -> s"65536,
${c.fillRows}")
+ for {
+ wholeStage <- if (c.codegenOnly) Seq(true) else Seq(true, false)
+ minCompactionToSpill <- Seq(None, Some("1.05"), Some("2.0"))
+ } {
+ val label = minCompactionToSpill.map(v => s"minCompactionToSpill =
$v")
+ .getOrElse("adaptive = F")
+ benchmark.addCase(s"codegen = $wholeStage, $label") { _ =>
+ withSQLConf(
+ (Seq(
+ SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> wholeStage.toString,
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key ->
+ minCompactionToSpill.isDefined.toString) ++
+ minCompactionToSpill.map(
+
SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key -> _) ++
+ fixedPlanConfs ++ spillConf): _*) {
+ query().noop()
+ }
+ }
+ }
+ benchmark.run()
+ }
+ }
}
private def distinctKeyedDf(N: Long): DataFrame =
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]