This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 51ee165ce10e [SPARK-58511][SQL][FOLLOWUP] Apply the minRows floor to
the adaptive partial aggregation spill check
51ee165ce10e is described below
commit 51ee165ce10e3a53cdda9f00c386d9ed26dfe837
Author: Wenchen Fan <[email protected]>
AuthorDate: Fri Oct 23 15:48:47 2026 +0000
[SPARK-58511][SQL][FOLLOWUP] Apply the minRows floor to the adaptive
partial aggregation spill check
### What changes were proposed in this pull request?
This is a followup of https://github.com/apache/spark/pull/57742.
Adaptive partial aggregation evaluates the compaction ratio at two points:
periodically, every `minRows` processed rows, and right before the aggregation
map would spill. The periodic check never decides on fewer than `minRows` rows,
but the spill check decides on whatever the current map epoch has processed,
which can be very few rows when the task is short on memory.
This PR applies the same `minRows` floor to the spill check, on both the
codegen path (`HashAggregateExec`) and the interpreted path
(`TungstenAggregationIterator`). When the map fills before the current epoch
has processed `minRows` rows, it spills as usual and the next epoch is judged
on its own rows. `minRows = 0`, which disables the periodic check, still leaves
the spill check without a floor. The `minRows` config doc and the SQL
performance tuning guide are updated to match.
### Why are the changes needed?
Pass-through cannot be undone within a task, so a decision made from a
small sample is expensive when it is wrong. If duplicate keys are spread more
widely than one small epoch, every epoch looks nearly distinct and the spill
check bypasses, even though spilling and merging would have reduced the rows
well. The task then sends roughly every input row to the shuffle instead of
about one row per key. A missed bypass only costs the spill the operator would
have done anyway, so waiting fo [...]
### Does this PR introduce _any_ user-facing change?
No. Adaptive partial aggregation is new in the unreleased 4.4.0 and
disabled by default. Within master, when the feature is enabled, a map that
fills before its current epoch has processed `minRows` rows now spills instead
of bypassing.
### How was this patch tested?
Updated `AdaptivePartialAggregationSuite`:
- The spill-check test is replaced by two tests. One checks that an epoch
smaller than `minRows` spills and falls back to sort-based aggregation instead
of bypassing. The other checks that, once past `minRows`, the spill check
bypasses between two periodic checks, starting from the row that found the map
full, on both the codegen and interpreted paths.
- The exact-ratio boundary test now uses `minRows = 0`, so it still
exercises the spill check without a floor.
`build/sbt "sql/testOnly *AdaptivePartialAggregationSuite"` passes. The
spill cases of `AdaptivePartialAggregationBenchmark` use `minRows = 0`, so
their results are unaffected.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
Closes #59741 from cloud-fan/wenchen/spark-58511-spill-floor.
Authored-by: Wenchen Fan <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit 7d16a2ba4afdf64731aa75f77b1dac54e07686be)
Signed-off-by: Wenchen Fan <[email protected]>
---
docs/sql-performance-tuning.md | 20 +++---
.../org/apache/spark/sql/internal/SQLConf.scala | 7 +-
.../execution/aggregate/HashAggregateExec.scala | 12 ++--
.../aggregate/TungstenAggregationIterator.scala | 9 ++-
.../AdaptivePartialAggregationSuite.scala | 78 ++++++++++++++++------
.../AdaptivePartialAggregationBenchmark.scala | 8 +--
6 files changed, 93 insertions(+), 41 deletions(-)
diff --git a/docs/sql-performance-tuning.md b/docs/sql-performance-tuning.md
index 97c40c83d7ea..62ea051a8aed 100644
--- a/docs/sql-performance-tuning.md
+++ b/docs/sql-performance-tuning.md
@@ -200,6 +200,9 @@ queue behind it and flushed only after it drains, so a
duplicate of a key alread
merges after that key's accumulated rows, keeping an order-sensitive aggregate
such as
`first`/`last` consistent with a run that never bypasses. The ratio is
evaluated periodically, and
again right before the aggregation map would spill, in which case the spill is
skipped entirely.
+Because pass-through cannot be reversed, the evaluation before a spill also
waits until the current
+map has processed `minRows` rows, so a map that fills sooner spills as usual
instead of committing
+the task on a small sample.
Both evaluations use the cumulative rows and keys of the current map epoch
and, once triggered,
pass-through is not reversed for the rest of the task, so a skewed prefix
biases the outcome in
@@ -208,8 +211,8 @@ on until a spill restarts the accounting; conversely, a
distinct-heavy prefix ca
pass-through early and keep it tripped even where the rest of the input would
aggregate well. To
catch the former without waiting for a spill, raise `minCompaction` so the
cumulative ratio trips
the threshold on a weaker late turn (a higher threshold also bypasses more
readily on other inputs).
-To avoid committing the task to pass-through on the latter, raise `minRows` so
the periodic
-evaluations start later.
+To avoid committing the task to pass-through on the latter, raise `minRows` so
the evaluations
+start later.
<table class="spark-config">
<thead><tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since
Version</th></tr></thead>
@@ -227,12 +230,13 @@ evaluations start later.
<td><code>spark.sql.execution.aggregate.adaptivePartialAggregation.minRows</code></td>
<td>100000</td>
<td>
- The number of rows between periodic compaction-ratio evaluations.
Setting this to
- <code>0</code> disables the periodic evaluation. The ratio may still be
evaluated when the
- aggregation map is about to spill. A larger value also delays the
periodic evaluations, so
- when one of them trips pass-through the frozen map tends to hold more
rows; the frozen map
- stays resident until its output is drained, so a larger value raises
that transient memory
- peak.
+ The number of rows between periodic compaction-ratio evaluations. The
ratio is also evaluated
+ when the aggregation map is about to spill, but only once the current
map has processed at
+ least this many rows. Setting this to <code>0</code> disables the
periodic evaluation, leaving
+ only the evaluation when the map is about to spill. A larger value also
delays the
+ evaluations, so when one of them trips pass-through the frozen map tends
to hold more rows;
+ the frozen map stays resident until its output is drained, so a larger
value raises that
+ transient memory peak.
</td>
<td>4.4.0</td>
</tr>
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 16ce8d8cc4a8..176a44805932 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -4623,9 +4623,10 @@ object SQLConf {
val ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS =
buildConf("spark.sql.execution.aggregate.adaptivePartialAggregation.minRows")
.doc("The number of rows between periodic compaction-ratio evaluations
by adaptive partial " +
- s"aggregation (see '${ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key}').
Setting this to 0 " +
- "disables the periodic evaluation. The ratio may still be evaluated
when the aggregation " +
- "map is about to spill.")
+ s"aggregation (see '${ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key}'). The
ratio is also " +
+ "evaluated when the aggregation map is about to spill, but only once
the current map " +
+ "has processed at least this many rows. Setting this to 0 disables the
periodic " +
+ "evaluation, leaving only the evaluation when the map is about to
spill.")
.version("4.4.0")
.withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.longConf
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
index d33eee47a944..1dec77de3bbf 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
@@ -949,9 +949,11 @@ case class HashAggregateExec(
// The compaction ratio is measured at the operator level: all processed
rows against the keys
// held by both maps, so two-level-map routing does not change the
decision. The same predicate
// decides both check points -- periodically every `minRows` rows, and
right before the map
- // would spill (in which case the spill is skipped entirely). `minRows =
0` disables the
- // periodic check: the row count is only ever compared after being
incremented past 0, so it
- // never matches and only the spill check remains.
+ // would spill (in which case the spill is skipped entirely). Pass-through
cannot be undone
+ // within the task, so the spill check, like the periodic one, waits until
the current epoch
+ // has processed `minRows` rows; a map that fills sooner spills as usual.
`minRows = 0`
+ // disables the periodic check: the row count is only ever compared after
being incremented
+ // past 0, so it never matches and only the spill check remains, without a
row floor.
val adaptiveIneffective = if (adaptivePartialAggEnabled) {
val totalKeys = if (isFastHashMapEnabled) {
s"($fastHashMapTerm.getNumKeys() - $adaptiveFastKeysAtSpillTerm + " +
@@ -1020,13 +1022,15 @@ case class HashAggregateExec(
|$adaptiveNextCheckRowTerm = ${adaptiveMinRows}L;
|$snapshotFastKeys
""".stripMargin
+ // The spill check needs at least one processed row even when `minRows
= 0`.
+ val spillCheckMinRows = math.max(adaptiveMinRows, 1L)
s"""
|// generate grouping key
|${unsafeRowKeyCode.code}
|if (!$adaptivePassThroughTerm) {
| $probeRegularMap
| if ($unsafeRowBuffer == null) {
- | if ($processedRowsTerm > 0 && $adaptiveIneffective) {
+ | if ($processedRowsTerm >= ${spillCheckMinRows}L &&
$adaptiveIneffective) {
| $adaptivePassThroughTerm = true;
| } else {
| $spillAndRestartEpoch
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala
index 779d3a80ad8a..69c0d892eb39 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala
@@ -197,7 +197,9 @@ class TungstenAggregationIterator(
// `minCompaction` rows into one key:
// - periodically, every `minRows` processed rows;
// - when the map is about to spill, in which case the spill is skipped
entirely and the row
- // that could not be inserted becomes the first pass-through row.
+ // that could not be inserted becomes the first pass-through row.
Pass-through cannot be
+ // undone within the task, so like the periodic check this waits until
the current epoch has
+ // processed `minRows` rows; a map that fills sooner spills as usual.
// A spill starts a new in-memory map epoch: the row counters restart so
that epoch is judged on
// its own rows, which lets an input whose cardinality only turns
unfavorable later still be
// passed through. Pass-through may therefore coexist with earlier spills.
The output order
@@ -223,6 +225,9 @@ class TungstenAggregationIterator(
// compared after being incremented past 0, so it never matches and only
the spill check
// below remains.
var nextCheckRow = minRows
+ // The spill check waits for `minRows` rows of the current epoch, and
for at least one row
+ // when `minRows = 0`.
+ val spillCheckMinRows = math.max(minRows, 1L)
// The partial aggregation is ineffective when it does not collapse
`minCompaction` rows into
// one key. There is no fast map on this path, so the map's keys are all
the operator holds.
def ineffective(): Boolean =
@@ -238,7 +243,7 @@ class TungstenAggregationIterator(
// The map is full and would normally spill. Adaptive partial
aggregation may instead
// bypass: keep the in-memory map as-is, pass this row and all
remaining rows through,
// and skip the spill entirely.
- if (adaptivePartialAggEnabled && processedRows > 0 && ineffective())
{
+ if (adaptivePartialAggEnabled && processedRows >= spillCheckMinRows
&& ineffective()) {
passThrough = true
// `newInput` could not be inserted; stash a copy as the first
pass-through row so it
// is not lost when we drain the rest of `inputIter`.
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/aggregate/AdaptivePartialAggregationSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/aggregate/AdaptivePartialAggregationSuite.scala
index 5af3793c9f88..d20848fd5345 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/aggregate/AdaptivePartialAggregationSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/aggregate/AdaptivePartialAggregationSuite.scala
@@ -1139,26 +1139,62 @@ class AdaptivePartialAggregationSuite extends QueryTest
with SharedSparkSession
}
}
- test("the spill check fires when the map would spill on high-cardinality
input") {
- // Force the regular map to fall back quickly. High-cardinality input that
reaches the fallback
- // point should bypass via the spill check rather than spilling. Use a
`minRows` larger than the
- // input so the periodic check cannot fire first and the spill check is
the one exercised.
- forEachCodegenAndMap(minRows = 100000, regularFallback = 16) { clue =>
+ test("the spill check does not bypass an epoch smaller than minRows") {
+ // The spill check, like the periodic check, judges an epoch only once it
holds `minRows` rows,
+ // because the bypass cannot be undone within the task. Force the regular
map to fall back
+ // every 16 rows with `minRows = 32`: every epoch is below the floor, so
even fully distinct
+ // input must spill and fall back to sort-based aggregation instead of
bypassing. The 200-row
+ // input as a whole exceeds `minRows`, so this also pins that the floor
counts the rows of the
+ // current epoch rather than of the task, which would bypass at the second
spill.
+ forEachCodegenAndMap(minRows = 32, regularFallback = 16) { clue =>
val df = () => spark.range(0, 200, 1, 1)
.select($"id".cast("string") as "k", $"id" as "v")
.groupBy($"k")
.agg(sum($"v") as "s")
withClue(clue) {
val c = runAndReadCounters(df)
- assert(c.skipped > 0,
- "the spill check should bypass high-cardinality input at the spill
boundary")
- // The whole point of the spill check is to bypass *instead of*
falling back to sort-based
- // aggregation, so the sorter is never created. `numTasksFallBacked`
is the reliable
- // per-operator signal for that (the `spillSize` metric on the
interpreted path is derived
- // from the task-cumulative memory-spill counter and can be inflated
by unrelated spilling
- // such as the downstream shuffle write, so it is not asserted here).
- assert(c.tasksFallBacked == 0,
- "the spill check must replace the sort fallback, not trigger it")
+ assert(c.skipped == 0,
+ "the spill check must not bypass an epoch smaller than minRows")
+ assert(c.tasksFallBacked > 0,
+ "an epoch below the minRows floor must spill and fall back to sort")
+ }
+ }
+ }
+
+ test("the spill check bypasses between periodic checks once minRows rows are
processed") {
+ // With `minRows = 10` the periodic check runs at the 10th aggregated row,
when ids 0-9 have
+ // collapsed into two keys (ratio 5.0, above `minCompaction = 2.0`), so it
keeps aggregating.
+ // Ids 10-18 then add nine distinct keys, and the forced fallback makes
the map full when id
+ // 19 arrives. That is past the floor but before the next periodic check
at 20 rows, and the
+ // ratio of 19 rows over 11 keys is below 2.0, so the spill check bypasses
at exactly that row:
+ // ids 19-399 stream through and the sorter is never created.
+ Seq(true, false).foreach { wholeStage =>
+ withSQLConf(
+ (Seq(
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key -> "true",
+ SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> wholeStage.toString,
+ SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> "false",
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key -> "10",
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION.key -> "2.0",
+ "spark.sql.TungstenAggregate.testFallbackStartsAt" -> "1, 19") ++
+ fixedPlanConfs): _*) {
+ val df = () => spark.range(0, 400, 1, 1)
+ .select(when($"id" < 10, $"id" % 2).otherwise($"id").cast("string")
as "k",
+ $"id" as "v")
+ .groupBy($"k")
+ .agg(sum($"v") as "s")
+ withClue(s"wholeStage=$wholeStage") {
+ val c = runAndReadCounters(df)
+ assert(c.skipped == 400 - 19,
+ "the spill check should bypass from the row that found the map
full")
+ // The spill check bypasses *instead of* falling back to sort-based
aggregation, so the
+ // sorter is never created. `numTasksFallBacked` is the reliable
per-operator signal for
+ // that (the `spillSize` metric on the interpreted path is derived
from the
+ // task-cumulative memory-spill counter and can be inflated by
unrelated spilling such as
+ // the downstream shuffle write, so it is not asserted here).
+ assert(c.tasksFallBacked == 0,
+ "the spill check must replace the sort fallback, not trigger it")
+ }
}
}
}
@@ -1198,10 +1234,11 @@ class AdaptivePartialAggregationSuite extends QueryTest
with SharedSparkSession
}
test("without the feature the same input really does fall back to sort") {
- // Sanity check for the spill check assertion above: with adaptive
disabled, the identical
- // high-cardinality input under the same forced fallback genuinely falls
back to sort-based
- // aggregation. This proves the spill check's `tasksFallBacked == 0`
reflects the bypass and
- // not merely an input that never reached the spill boundary.
+ // Sanity check for the spill check assertions: with adaptive disabled,
the high-cardinality
+ // input that "minRows = 0 still bypasses at the spill boundary" uses,
under the same forced
+ // fallback, genuinely falls back to sort-based aggregation. This proves
that test's
+ // `tasksFallBacked == 0` reflects the bypass and not merely an input that
never reached the
+ // spill boundary.
for {
wholeStage <- Seq(true, false)
twoLevelMap <- Seq(true, false)
@@ -1241,8 +1278,9 @@ class AdaptivePartialAggregationSuite extends QueryTest
with SharedSparkSession
SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key -> "true",
SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> wholeStage.toString,
SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> "false",
- // A `minRows` larger than the input keeps the periodic check out
of the picture.
- SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key -> "100000",
+ // `minRows = 0` keeps the periodic check out of the picture and
leaves the spill check
+ // without a row floor, so the spill check alone decides at the
boundary.
+ SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key -> "0",
SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION.key ->
minCompaction.toString,
"spark.sql.TungstenAggregate.testFallbackStartsAt" -> "1, 50") ++
fixedPlanConfs): _*) {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/AdaptivePartialAggregationBenchmark.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/AdaptivePartialAggregationBenchmark.scala
index 0482128b5bf4..7c23d5e296bc 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/AdaptivePartialAggregationBenchmark.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/AdaptivePartialAggregationBenchmark.scala
@@ -106,10 +106,10 @@ object AdaptivePartialAggregationBenchmark extends
SqlBasedBenchmark {
benchmark.run()
}
- // Force the regular map to spill quickly and disable the periodic check
(huge minRows). With
- // fully distinct keys the compaction ratio is 1.0, so at the spill
boundary the spill check
- // bypasses instead of spilling; the baseline spills repeatedly and falls
back to sort-based
- // aggregation.
+ // Force the regular map to spill quickly and disable the periodic check
(minRows = 0, which
+ // also leaves the spill check without a row floor). With fully distinct
keys the compaction
+ // ratio is 1.0, so at the spill boundary the spill check bypasses instead
of spilling; the
+ // baseline spills repeatedly and falls back to sort-based aggregation.
runBenchmark("high-cardinality input, pass-through at the spill check") {
val N = 8L << 20
val benchmark = new Benchmark("adaptive partial agg, high card, spill",
N, output = output)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]