This is an automated email from the ASF dual-hosted git repository.

cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new e6fc64f58dc2 [SPARK-60561][SQL] Judge the adaptive partial aggregation 
spill check by the recent rows, with a higher threshold
e6fc64f58dc2 is described below

commit e6fc64f58dc26663060af092d3845456f9c3b198
Author: Wenchen Fan <[email protected]>
AuthorDate: Fri Oct 30 07:10:19 2026 +0000

    [SPARK-60561][SQL] Judge the adaptive partial aggregation spill check by 
the recent rows, with a higher threshold
    
    ### What changes were proposed in this pull request?
    
    Adaptive partial aggregation (#57742) evaluates its compaction ratio 
periodically and right before its map would spill, against one threshold, 
`minCompaction` (default 1.05). This PR changes the evaluation before a spill 
in two ways, on both the codegen and the interpreted path:
    
    - It measures the recent rows, those processed since the periodic 
evaluation before last, against the keys they added, instead of the whole map 
epoch.
    - It demands a new, higher threshold, 
`spark.sql.execution.aggregate.adaptivePartialAggregation.minCompactionToSpill` 
(default 2.0). The evaluation uses the larger of it and `minCompaction`.
    
    The window advances at each periodic evaluation, so the change adds no 
per-row work. This replaces the spill threshold that an earlier revision of 
#59764 proposed, which judged the whole epoch, and covers the review comments 
on the spill check that were deferred from there.
    
    ### Why are the changes needed?
    
    Keeping the aggregation when the map is about to spill means aggregating 
the rest of the input through spills. That pays off only if the rest of the 
input keeps repeating keys: the final merge then collapses them, and the 
shuffle carries a fraction of the rows. Otherwise the task writes most of its 
rows to disk to save shuffling a few. With one threshold of 1.05, an input 
whose keys arrive together, at a ratio of 1.5, keeps aggregating and spills 
every epoch.
    
    The ratio of the whole epoch cannot tell those inputs apart. A uniformly 
distributed key with ten rows per key per task fills its map at a ratio of 
about 1.65, in the same range as an input whose epochs share no keys, yet 
keeping the aggregation is the right call for it. The recent rows separate the 
two. When keys repeat across the input, more of the recent rows find their key 
already in the map as it fills, so their ratio climbs above the epoch's, to 
between 2 and 3 in that example.  [...]
    
    The two evaluations weigh different costs, which is why they get separate 
thresholds. The periodic one weighs only maintaining the map, so a ratio barely 
above 1 is worth keeping, while the one before a spill weighs spilling the rest 
of the input.
    
    `AdaptivePartialAggregationBenchmark` gets spill-check cases with an 
`Exchange` between the Partial and the Final. Compared with 1.05, in the "Run 
benchmarks" workflow on JDK 17, 21 and 25, on both paths:
    
    - Keys that arrive together: about 1.3x faster, because the rest of the 
input is passed through instead of spilled every epoch.
    - 10 hot keys over a unique tail: about 1.3x faster.
    - Keys that recur, with the map holding about two thirds of them: the same 
decision as 1.05, keeping the aggregation.
    - Keys that recur, with the maps holding 40% of them, with the fast map off 
and on: the default passes the rest through. That was within about 8% of 
keeping the aggregation on the runners that produced the committed results, and 
about 20% slower in one earlier run on a different CPU.
    
    Passing the rest of the input through also streams the later rows of keys 
the frozen map holds, so with a few hot keys each of them reaches the final 
aggregation row by row, on the one reducer that key hashes to. The docs 
describe this trade-off, and that setting `minCompactionToSpill` to 
`minCompaction` or below keeps inputs dominated by one or two keys aggregating.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. Adaptive partial aggregation is new in the unreleased 4.4.0 and 
disabled by default. When it is enabled, a map about to spill passes the rest 
of its input through when its recent rows compact by less than 2.0, and the new 
config tunes that threshold.
    
    ### How was this patch tested?
    
    New and updated tests in `AdaptivePartialAggregationSuite`, on both paths:
    - An input whose keys arrive together at a ratio of 1.5 passes the rest 
through at the spill check by default, and keeps spilling with the threshold 
lowered to 1.3. It runs with one input partition and with two, which put an 
`Exchange` between the Partial and the Final.
    - An input whose epoch compacts by 1.875 but whose recent rows compact by 
3.75 keeps aggregating by default, and passes through with the threshold raised 
to 4.0.
    - The exact-boundary test sets the spill threshold, and checks that the 
larger of the two thresholds applies, from either side.
    - A new test reaches the boundary of the recent rows: they compact by 
exactly 2.0, while a window starting one check earlier compacts by 1.33 and the 
whole epoch by 1.5, so the task keeps aggregating at 2.0 and passes through at 
2.1.
    - A spill followed by a distinct-heavy epoch now flips at that epoch's 
first periodic check, with exact counts. It pins `minCompactionToSpill` to 1.5 
so the counts do not rest on a tie at 2.0.
    - Two tests pin the window's bookkeeping: a new epoch that fills after one 
periodic check does not count the previous epoch's keys, and on the codegen 
path the window counts the fast map's keys.
    - The new config rejects non-finite values, as `minCompaction` does.
    
    `build/sbt "sql/testOnly *AdaptivePartialAggregationSuite"` passes.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5.5)
    
    This pull request and its description were written by Isaac.
    
    Closes #59823 from cloud-fan/wenchen/spark-60561-spill-check.
    
    Authored-by: Wenchen Fan <[email protected]>
    Signed-off-by: Wenchen Fan <[email protected]>
    (cherry picked from commit 60c7b8feb9986d9c4187dd9fbd4d7bef8d5ad02d)
    Signed-off-by: Wenchen Fan <[email protected]>
---
 docs/sql-performance-tuning.md                     |  88 ++++--
 .../org/apache/spark/sql/internal/SQLConf.scala    |  43 ++-
 ...vePartialAggregationBenchmark-jdk21-results.txt | 117 ++++++--
 ...vePartialAggregationBenchmark-jdk25-results.txt | 117 ++++++--
 ...AdaptivePartialAggregationBenchmark-results.txt | 117 ++++++--
 .../execution/aggregate/HashAggregateExec.scala    |  76 ++++--
 .../aggregate/TungstenAggregationIterator.scala    |  49 +++-
 .../AdaptivePartialAggregationSuite.scala          | 303 ++++++++++++++++++---
 .../AdaptivePartialAggregationBenchmark.scala      |  68 ++++-
 9 files changed, 814 insertions(+), 164 deletions(-)

diff --git a/docs/sql-performance-tuning.md b/docs/sql-performance-tuning.md
index f38522110fd6..c5f4223ae558 100644
--- a/docs/sql-performance-tuning.md
+++ b/docs/sql-performance-tuning.md
@@ -198,24 +198,48 @@ for the final aggregation to merge. Once pass-through is 
active the map is froze
 always comes before the passed-through rows: rows that collide with the frozen 
map are held in a
 queue behind it and flushed only after it drains, so a duplicate of a key 
already in the map still
 merges after that key's accumulated rows, keeping an order-sensitive aggregate 
such as
-`first`/`last` consistent with a run that never bypasses. The ratio is 
evaluated periodically, and
-again right before the aggregation map would spill, in which case the spill is 
skipped entirely.
-Because pass-through cannot be reversed, the evaluation before a spill also 
waits until the current
-map has processed `minRows` rows, so a map that fills sooner spills as usual 
instead of committing
-the task on a small sample. The SQL UI counts the passed-through rows in
-`number of pass-through output rows`. They are not timed: `time in aggregation 
build` covers the
+`first`/`last` consistent with a run that never bypasses. The SQL UI counts 
the passed-through rows
+in `number of pass-through output rows`. They are not timed: `time in 
aggregation build` covers the
 aggregation up to the point where pass-through starts, and preparing the 
aggregated map for output,
 which in a task that has spilled also writes the map to disk and merges it 
with the spilled data.
 
-Both evaluations use the cumulative rows and keys of the current map epoch 
and, once triggered,
-pass-through is not reversed for the rest of the task, so a skewed prefix 
biases the outcome in
-either direction. A favorable prefix can mask a later distinct-heavy tail, 
keeping the aggregation
-on until a spill restarts the accounting; conversely, a distinct-heavy prefix 
can trip the
-pass-through early and keep it tripped even where the rest of the input would 
aggregate well. To
+The ratio is evaluated periodically, every `minRows` rows, over the rows and 
keys of the current map
+epoch, against `minCompaction`. It is evaluated again right before the 
aggregation map would spill,
+against the larger of `minCompaction` and `minCompactionToSpill`. Keeping the 
aggregation at that
+point means aggregating the rest of the input through spills, which pays off 
only if that input
+keeps repeating keys, so this evaluation measures the recent rows, at least 
`minRows` and fewer than
+twice that many: those processed since the periodic evaluation before last, 
against the keys they
+added. When keys repeat across the input, more of the recent rows find their 
key already in the map
+as it fills, so their ratio climbs above the epoch's; when each key's rows 
arrive together, the two
+ratios match. The recent rows still only show how often keys repeat within the 
map: with keys spread
+evenly over the task, their ratio reflects how much of the task's distinct 
keys the map holds rather
+than how often each key repeats, so at the default threshold the aggregation 
is kept only when the
+map holds about half of the task's keys or more. Before the current map's 
second periodic
+evaluation, or with `minRows` set to `0`, the evaluation measures the whole 
map epoch instead, whose
+ratio is lower because an epoch's early rows add most of its keys, so it 
passes through more
+readily. If the ratio is below the threshold, the map is not spilled and the 
rest of the input is
+passed through. In a task that has already spilled, the frozen map is still 
written out and merged
+with the spilled data at the end, but no later epoch spills. Because 
pass-through cannot be
+reversed, the evaluation before a spill also waits until the current map has 
processed `minRows`
+rows, so a map that fills sooner spills as usual instead of committing the 
task on a small sample.
+
+Once triggered, pass-through is not reversed for the rest of the task, so a 
skewed prefix biases the
+outcome in either direction. A favorable prefix can mask a later 
distinct-heavy tail, keeping the
+aggregation on until a spill restarts the accounting; conversely, a 
distinct-heavy prefix can trip
+the pass-through early and keep it tripped even where the rest of the input 
would aggregate well. To
 catch the former without waiting for a spill, raise `minCompaction` so the 
cumulative ratio trips
 the threshold on a weaker late turn (a higher threshold also bypasses more 
readily on other inputs).
-To avoid committing the task to pass-through on the latter, raise `minRows` so 
the evaluations
-start later.
+To avoid committing the task to pass-through on the latter, raise `minRows` so 
the evaluations start
+later, or, when the task commits at the evaluation before a spill, lower 
`minCompactionToSpill`.
+Raising `minRows` helps there only past the number of rows the map takes to 
fill, and also widens
+the sample of recent rows that the evaluation before a spill measures. Lowering
+`minCompactionToSpill` also keeps aggregating keys that repeat across a task 
whose map holds well
+under half of them; whether that pays off depends on how the cost of spilling 
the rest of the input
+compares with shuffling it. Pass-through also streams the later rows of keys 
the frozen map holds,
+so with a few hot keys each of them reaches the final aggregation row by row, 
on the one reducer
+that key hashes to. When one or two keys, such as a NULL or default value, 
hold a large share of the
+rows over many distinct ones, that reducer can receive a large share of all 
rows; setting
+`minCompactionToSpill` to `minCompaction` or below keeps such inputs 
aggregating through spills.
 
 <table class="spark-config">
   <thead><tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since 
Version</th></tr></thead>
@@ -235,11 +259,14 @@ start later.
     <td>
       The number of rows between periodic compaction-ratio evaluations. The 
ratio is also evaluated
       when the aggregation map is about to spill, but only once the current 
map has processed at
-      least this many rows. Setting this to <code>0</code> disables the 
periodic evaluation, leaving
-      only the evaluation when the map is about to spill. A larger value also 
delays the
-      evaluations, so when one of them trips pass-through the frozen map tends 
to hold more rows;
-      the frozen map stays resident until its output is drained, so a larger 
value raises that
-      transient memory peak.
+      least this many rows, and over the rows since the periodic evaluation 
before last, so this
+      also sets how many recent rows that evaluation measures: at least this 
many and fewer than
+      twice that, or the whole map epoch when the map fills before its second 
periodic evaluation.
+      Setting this to <code>0</code> disables the periodic evaluation, leaving 
only the evaluation
+      when the map is about to spill, which then measures the whole map epoch. 
A larger value also
+      delays the evaluations, so when one of them trips pass-through the 
frozen map tends to hold
+      more rows; the frozen map stays resident until its output is drained, so 
a larger value raises
+      that transient memory peak.
     </td>
     <td>4.4.0</td>
   </tr>
@@ -247,10 +274,27 @@ start later.
     
<td><code>spark.sql.execution.aggregate.adaptivePartialAggregation.minCompaction</code></td>
     <td>1.05</td>
     <td>
-      The minimum compaction ratio required to keep the partial aggregation. A 
ratio of 10 means the
-      partial aggregation collapses ten rows into one key; when an evaluation 
finds the ratio below
-      this value the partial aggregation is bypassed for the rest of the 
input. A larger value
-      bypasses more aggressively.
+      The minimum compaction ratio required to keep the partial aggregation at 
the periodic
+      evaluation. A ratio of 10 means the partial aggregation collapses ten 
rows into one key; when
+      the periodic evaluation finds the ratio below this value the partial 
aggregation is bypassed
+      for the rest of the input. A larger value bypasses more aggressively. 
The evaluation before a
+      spill demands at least this value too.
+    </td>
+    <td>4.4.0</td>
+  </tr>
+  <tr>
+    
<td><code>spark.sql.execution.aggregate.adaptivePartialAggregation.minCompactionToSpill</code></td>
+    <td>2.0</td>
+    <td>
+      The minimum compaction ratio required to keep the partial aggregation 
when its map is about to
+      spill, measured over the rows processed since the periodic evaluation 
before last, or over the
+      whole map epoch before its second periodic evaluation. Keeping the 
aggregation then means
+      aggregating the rest of the input through spills, which pays off only if 
that input keeps
+      repeating keys. With keys spread evenly over the task, the default keeps 
aggregating only when
+      the map holds about half of the task's keys or more. When the ratio is 
below the larger of
+      this value and <code>minCompaction</code>, the map is not spilled and 
the rest of the input is
+      passed through. The evaluation runs only once the current map has 
processed
+      <code>minRows</code> rows.
     </td>
     <td>4.4.0</td>
   </tr>
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 692aac02e565..c4793ee4de7d 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -4627,8 +4627,12 @@ object SQLConf {
       .doc("The number of rows between periodic compaction-ratio evaluations 
by adaptive partial " +
         s"aggregation (see '${ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key}'). The 
ratio is also " +
         "evaluated when the aggregation map is about to spill, but only once 
the current map " +
-        "has processed at least this many rows. Setting this to 0 disables the 
periodic " +
-        "evaluation, leaving only the evaluation when the map is about to 
spill.")
+        "has processed at least this many rows, and over the rows since the 
periodic " +
+        "evaluation before last, so this also sets how many recent rows that 
evaluation " +
+        "measures: at least this many and fewer than twice that, or the whole 
map epoch when " +
+        "the map fills before its second periodic evaluation. Setting this to 
0 disables the " +
+        "periodic evaluation, leaving only the evaluation when the map is 
about to spill, " +
+        "which then measures all rows and keys of the current map epoch.")
       .version("4.4.0")
       .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .longConf
@@ -4640,9 +4644,11 @@ object SQLConf {
       .doc("The minimum compaction ratio required to keep the pre-shuffle 
partial aggregation " +
         s"(see '${ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key}'). The compaction 
ratio is the " +
         "number of processed rows divided by the number of keys held in the 
aggregation maps, " +
-        "so a ratio of 10 means the partial aggregation collapses ten rows 
into one. When an " +
-        "evaluation finds the ratio below this value, the partial aggregation 
is bypassed for " +
-        "the rest of the input. A larger value bypasses more aggressively.")
+        "so a ratio of 10 means the partial aggregation collapses ten rows 
into one. When the " +
+        "periodic evaluation finds the ratio below this value, the partial 
aggregation is " +
+        "bypassed for the rest of the input. A larger value bypasses more 
aggressively. The " +
+        "evaluation when the map is about to spill demands at least this value 
too, see " +
+        
"'spark.sql.execution.aggregate.adaptivePartialAggregation.minCompactionToSpill'.")
       .version("4.4.0")
       .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .doubleConf
@@ -4650,6 +4656,30 @@ object SQLConf {
         "The minimum compaction ratio must be a finite value of at least 1.0.")
       .createWithDefault(1.05)
 
+  val ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL =
+    
buildConf("spark.sql.execution.aggregate.adaptivePartialAggregation.minCompactionToSpill")
+      .doc("The minimum compaction ratio required to keep the pre-shuffle 
partial aggregation " +
+        "when its map is about to spill (see " +
+        s"'${ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key}'). Keeping the 
aggregation then means " +
+        "aggregating the rest of the input through spills, which pays off only 
if that input " +
+        "keeps repeating keys, so this evaluation measures the recent rows: 
those processed " +
+        "since the periodic evaluation before last, against the keys they 
added, or the whole " +
+        "map epoch before the map's second periodic evaluation. When keys 
repeat across the " +
+        "input, that ratio climbs as the map fills; with keys spread evenly 
over the task it " +
+        "reflects how much of the task's distinct keys the map holds rather 
than how often " +
+        "each key repeats, so at the default the aggregation is kept only when 
the map holds " +
+        "about half of them or more. When the ratio is below the larger of 
this value and " +
+        s"'${ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION.key}', the map is not 
spilled and the " +
+        "rest of the input is passed through. Like the periodic evaluation, it 
only runs once " +
+        "the current map has processed " +
+        s"'${ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key}' rows.")
+      .version("4.4.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+      .doubleConf
+      .checkValue(v => v >= 1.0 && v.isFinite,
+        "The minimum compaction ratio must be a finite value of at least 1.0.")
+      .createWithDefault(2.0)
+
   val JSON_GENERATOR_IGNORE_NULL_FIELDS =
     buildConf("spark.sql.jsonGenerator.ignoreNullFields")
       .doc("Whether to ignore null fields when generating JSON objects in JSON 
data source and " +
@@ -9847,6 +9877,9 @@ class SQLConf extends Serializable with Logging with 
SqlApiConf {
   def adaptivePartialAggregationMinCompaction: Double =
     getConf(ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION)
 
+  def adaptivePartialAggregationMinCompactionToSpill: Double =
+    getConf(ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL)
+
   def objectAggSortBasedFallbackThreshold: Int = 
getConf(OBJECT_AGG_SORT_BASED_FALLBACK_THRESHOLD)
 
   def variableSubstituteEnabled: Boolean = getConf(VARIABLE_SUBSTITUTE_ENABLED)
diff --git 
a/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-jdk21-results.txt 
b/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-jdk21-results.txt
index dd08f21eca70..5bc0aab19a43 100644
--- a/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-jdk21-results.txt
+++ b/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-jdk21-results.txt
@@ -2,55 +2,132 @@
 high-cardinality input, pass-through at the periodic check
 
================================================================================================
 
-OpenJDK 64-Bit Server VM 21.0.12+8-LTS on Linux 6.17.0-1020-azure
+OpenJDK 64-Bit Server VM 21.0.13+9-LTS on Linux 6.17.0-1022-azure
 AMD EPYC 7763 64-Core Processor
 adaptive partial agg, high card, no spill:  Best Time(ms)   Avg Time(ms)   
Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 
-------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F                        4069           4145        
 108          2.1         485.0       1.0X
-codegen = true, adaptive = T                        2585           2632        
  67          3.2         308.1       1.6X
-codegen = false, adaptive = F                       4818           4849        
  44          1.7         574.3       0.8X
-codegen = false, adaptive = T                       2983           3150        
 236          2.8         355.6       1.4X
+codegen = true, adaptive = F                        4280           4290        
  14          2.0         510.2       1.0X
+codegen = true, adaptive = T                        2617           2625        
  11          3.2         312.0       1.6X
+codegen = false, adaptive = F                       5007           5030        
  32          1.7         596.9       0.9X
+codegen = false, adaptive = T                       3253           3273        
  28          2.6         387.8       1.3X
 
 
 
================================================================================================
 low-cardinality input, pass-through at the periodic check
 
================================================================================================
 
-OpenJDK 64-Bit Server VM 21.0.12+8-LTS on Linux 6.17.0-1020-azure
+OpenJDK 64-Bit Server VM 21.0.13+9-LTS on Linux 6.17.0-1022-azure
 AMD EPYC 7763 64-Core Processor
 adaptive partial agg, low card, no spill:  Best Time(ms)   Avg Time(ms)   
Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 
------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F                        281            292         
  7         59.6          16.8       1.0X
-codegen = true, adaptive = T                        291            323         
 52         57.7          17.3       1.0X
-codegen = false, adaptive = F                      1289           1303         
 20         13.0          76.8       0.2X
-codegen = false, adaptive = T                      1266           1270         
  7         13.3          75.4       0.2X
+codegen = true, adaptive = F                        287            306         
 21         58.4          17.1       1.0X
+codegen = true, adaptive = T                        293            301         
  9         57.2          17.5       1.0X
+codegen = false, adaptive = F                      1356           1359         
  5         12.4          80.8       0.2X
+codegen = false, adaptive = T                      1360           1365         
  7         12.3          81.1       0.2X
 
 
 
================================================================================================
 high-cardinality input, pass-through at the spill check
 
================================================================================================
 
-OpenJDK 64-Bit Server VM 21.0.12+8-LTS on Linux 6.17.0-1020-azure
+OpenJDK 64-Bit Server VM 21.0.13+9-LTS on Linux 6.17.0-1022-azure
 AMD EPYC 7763 64-Core Processor
 adaptive partial agg, high card, spill:   Best Time(ms)   Avg Time(ms)   
Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 
------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F                       8120           8126         
  9          1.0         968.0       1.0X
-codegen = true, adaptive = T                       4460           4476         
 22          1.9         531.7       1.8X
-codegen = false, adaptive = F                      9331           9389         
 82          0.9        1112.3       0.9X
-codegen = false, adaptive = T                      5369           5382         
 18          1.6         640.1       1.5X
+codegen = true, adaptive = F                       8875           8896         
 29          0.9        1058.0       1.0X
+codegen = true, adaptive = T                       4931           4961         
 43          1.7         587.8       1.8X
+codegen = false, adaptive = F                     10087          10123         
 51          0.8        1202.5       0.9X
+codegen = false, adaptive = T                      5756           5769         
 19          1.5         686.2       1.5X
 
 
 
================================================================================================
 low-cardinality input, pass-through at the spill check
 
================================================================================================
 
-OpenJDK 64-Bit Server VM 21.0.12+8-LTS on Linux 6.17.0-1020-azure
+OpenJDK 64-Bit Server VM 21.0.13+9-LTS on Linux 6.17.0-1022-azure
 AMD EPYC 7763 64-Core Processor
 adaptive partial agg, low card, spill:    Best Time(ms)   Avg Time(ms)   
Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 
------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F                        768            782         
 12         21.8          45.8       1.0X
-codegen = true, adaptive = T                        769            780         
 13         21.8          45.8       1.0X
-codegen = false, adaptive = F                      1366           1369         
  4         12.3          81.4       0.6X
-codegen = false, adaptive = T                      1358           1358         
  0         12.4          80.9       0.6X
+codegen = true, adaptive = F                        817            829         
 11         20.5          48.7       1.0X
+codegen = true, adaptive = T                        843            861         
 18         19.9          50.3       1.0X
+codegen = false, adaptive = F                      1398           1478         
113         12.0          83.3       0.6X
+codegen = false, adaptive = T                      1406           1414         
 11         11.9          83.8       0.6X
+
+
+================================================================================================
+spill check with an Exchange, keys arrive together
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.13+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys arrive together:  Best Time(ms)   Avg 
Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
+---------------------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F                                      6653         
  6661          11          1.3         793.1       1.0X
+codegen = true, minCompactionToSpill = 1.05                       6571         
  6591          28          1.3         783.3       1.0X
+codegen = true, minCompactionToSpill = 2.0                        5047         
  5102          77          1.7         601.7       1.3X
+codegen = false, adaptive = F                                     8090         
  8094           6          1.0         964.4       0.8X
+codegen = false, minCompactionToSpill = 1.05                      8004         
  8069          93          1.0         954.1       0.8X
+codegen = false, minCompactionToSpill = 2.0                       6219         
  6228          13          1.3         741.4       1.1X
+
+
+================================================================================================
+spill check with an Exchange, keys recur
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.13+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys recur:  Best Time(ms)   Avg Time(ms)   
Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
+-----------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F                            3364           3443    
     112          2.5         401.1       1.0X
+codegen = true, minCompactionToSpill = 1.05             3394           3417    
      33          2.5         404.5       1.0X
+codegen = true, minCompactionToSpill = 2.0              3210           3268    
      82          2.6         382.6       1.0X
+codegen = false, adaptive = F                           4044           4177    
     188          2.1         482.1       0.8X
+codegen = false, minCompactionToSpill = 1.05            4386           4422    
      51          1.9         522.8       0.8X
+codegen = false, minCompactionToSpill = 2.0             4452           4479    
      38          1.9         530.8       0.8X
+
+
+================================================================================================
+spill check with an Exchange, keys recur, map holds 40% of them, fast map off
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.13+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys recur, map holds 40% of them, fast map 
off:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   
Relative
+------------------------------------------------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F                                                   
              9194           9460         377          1.8         548.0       
1.0X
+codegen = true, minCompactionToSpill = 1.05                                    
              9205           9459         360          1.8         548.6       
1.0X
+codegen = true, minCompactionToSpill = 2.0                                     
              8908           8949          58          1.9         531.0       
1.0X
+codegen = false, adaptive = F                                                  
              9129           9345         306          1.8         544.1       
1.0X
+codegen = false, minCompactionToSpill = 1.05                                   
              9652           9710          82          1.7         575.3       
1.0X
+codegen = false, minCompactionToSpill = 2.0                                    
             10163          10196          47          1.7         605.7       
0.9X
+
+
+================================================================================================
+spill check with an Exchange, keys recur, maps hold 40% of them, fast map on
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.13+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys recur, maps hold 40% of them, fast map 
on:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   
Relative
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F                                                   
             8892           8975         118          1.9         530.0       
1.0X
+codegen = true, minCompactionToSpill = 1.05                                    
             9157           9166          13          1.8         545.8       
1.0X
+codegen = true, minCompactionToSpill = 2.0                                     
             8390           8393           6          2.0         500.1       
1.1X
+
+
+================================================================================================
+spill check with an Exchange, hot keys
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.13+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, hot keys:  Best Time(ms)   Avg Time(ms)   
Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
+----------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F                           6814           6837     
     32          1.2         812.3       1.0X
+codegen = true, minCompactionToSpill = 1.05            6778           6819     
     58          1.2         808.0       1.0X
+codegen = true, minCompactionToSpill = 2.0             5298           5430     
    186          1.6         631.6       1.3X
+codegen = false, adaptive = F                          7995           8007     
     16          1.0         953.1       0.9X
+codegen = false, minCompactionToSpill = 1.05           8093           8151     
     83          1.0         964.7       0.8X
+codegen = false, minCompactionToSpill = 2.0            6346           6372     
     36          1.3         756.5       1.1X
 
 
diff --git 
a/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-jdk25-results.txt 
b/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-jdk25-results.txt
index f54c40877aaf..f55167c051a5 100644
--- a/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-jdk25-results.txt
+++ b/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-jdk25-results.txt
@@ -2,55 +2,132 @@
 high-cardinality input, pass-through at the periodic check
 
================================================================================================
 
-OpenJDK 64-Bit Server VM 25.0.4+7-LTS on Linux 6.17.0-1021-azure
+OpenJDK 64-Bit Server VM 25.0.5+8-LTS on Linux 6.17.0-1022-azure
 AMD EPYC 7763 64-Core Processor
 adaptive partial agg, high card, no spill:  Best Time(ms)   Avg Time(ms)   
Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 
-------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F                        4338           4408        
 100          1.9         517.1       1.0X
-codegen = true, adaptive = T                        2586           2608        
  30          3.2         308.3       1.7X
-codegen = false, adaptive = F                       5146           5199        
  76          1.6         613.4       0.8X
-codegen = false, adaptive = T                       3271           3287        
  22          2.6         390.0       1.3X
+codegen = true, adaptive = F                        3995           4021        
  36          2.1         476.3       1.0X
+codegen = true, adaptive = T                        2435           2439        
   5          3.4         290.3       1.6X
+codegen = false, adaptive = F                       4704           4707        
   5          1.8         560.7       0.8X
+codegen = false, adaptive = T                       3088           3114        
  37          2.7         368.1       1.3X
 
 
 
================================================================================================
 low-cardinality input, pass-through at the periodic check
 
================================================================================================
 
-OpenJDK 64-Bit Server VM 25.0.4+7-LTS on Linux 6.17.0-1021-azure
+OpenJDK 64-Bit Server VM 25.0.5+8-LTS on Linux 6.17.0-1022-azure
 AMD EPYC 7763 64-Core Processor
 adaptive partial agg, low card, no spill:  Best Time(ms)   Avg Time(ms)   
Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 
------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F                        269            275         
  4         62.5          16.0       1.0X
-codegen = true, adaptive = T                        284            294         
  6         59.1          16.9       0.9X
-codegen = false, adaptive = F                      1317           1321         
  6         12.7          78.5       0.2X
-codegen = false, adaptive = T                      1309           1382         
104         12.8          78.0       0.2X
+codegen = true, adaptive = F                        254            267         
 10         66.0          15.1       1.0X
+codegen = true, adaptive = T                        301            308         
  6         55.8          17.9       0.8X
+codegen = false, adaptive = F                      1321           1324         
  4         12.7          78.7       0.2X
+codegen = false, adaptive = T                      1316           1316         
  1         12.8          78.4       0.2X
 
 
 
================================================================================================
 high-cardinality input, pass-through at the spill check
 
================================================================================================
 
-OpenJDK 64-Bit Server VM 25.0.4+7-LTS on Linux 6.17.0-1021-azure
+OpenJDK 64-Bit Server VM 25.0.5+8-LTS on Linux 6.17.0-1022-azure
 AMD EPYC 7763 64-Core Processor
 adaptive partial agg, high card, spill:   Best Time(ms)   Avg Time(ms)   
Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 
------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F                       8332           8462         
184          1.0         993.2       1.0X
-codegen = true, adaptive = T                       4677           4683         
  9          1.8         557.5       1.8X
-codegen = false, adaptive = F                      9932           9966         
 47          0.8        1184.0       0.8X
-codegen = false, adaptive = T                      5412           5618         
291          1.6         645.1       1.5X
+codegen = true, adaptive = F                       8112           8167         
 78          1.0         967.0       1.0X
+codegen = true, adaptive = T                       4599           4619         
 28          1.8         548.2       1.8X
+codegen = false, adaptive = F                      9269           9273         
  6          0.9        1104.9       0.9X
+codegen = false, adaptive = T                      5384           5390         
  9          1.6         641.8       1.5X
 
 
 
================================================================================================
 low-cardinality input, pass-through at the spill check
 
================================================================================================
 
-OpenJDK 64-Bit Server VM 25.0.4+7-LTS on Linux 6.17.0-1021-azure
+OpenJDK 64-Bit Server VM 25.0.5+8-LTS on Linux 6.17.0-1022-azure
 AMD EPYC 7763 64-Core Processor
 adaptive partial agg, low card, spill:    Best Time(ms)   Avg Time(ms)   
Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 
------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F                        773            786         
 17         21.7          46.1       1.0X
-codegen = true, adaptive = T                        790            803         
 20         21.2          47.1       1.0X
-codegen = false, adaptive = F                      1370           1377         
 10         12.2          81.6       0.6X
-codegen = false, adaptive = T                      1366           1373         
 11         12.3          81.4       0.6X
+codegen = true, adaptive = F                        771            780         
 15         21.8          45.9       1.0X
+codegen = true, adaptive = T                        809            820         
 16         20.7          48.2       1.0X
+codegen = false, adaptive = F                      1387           1393         
  8         12.1          82.7       0.6X
+codegen = false, adaptive = T                      1382           1383         
  2         12.1          82.4       0.6X
+
+
+================================================================================================
+spill check with an Exchange, keys arrive together
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.5+8-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys arrive together:  Best Time(ms)   Avg 
Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
+---------------------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F                                      6107         
  6110           4          1.4         728.0       1.0X
+codegen = true, minCompactionToSpill = 1.05                       6115         
  6146          43          1.4         729.0       1.0X
+codegen = true, minCompactionToSpill = 2.0                        4771         
  4783          16          1.8         568.8       1.3X
+codegen = false, adaptive = F                                     7032         
  7047          21          1.2         838.3       0.9X
+codegen = false, minCompactionToSpill = 1.05                      7068         
  7092          33          1.2         842.6       0.9X
+codegen = false, minCompactionToSpill = 2.0                       5590         
  5609          26          1.5         666.4       1.1X
+
+
+================================================================================================
+spill check with an Exchange, keys recur
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.5+8-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys recur:  Best Time(ms)   Avg Time(ms)   
Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
+-----------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F                            2739           2768    
      41          3.1         326.5       1.0X
+codegen = true, minCompactionToSpill = 1.05             2699           2714    
      21          3.1         321.8       1.0X
+codegen = true, minCompactionToSpill = 2.0              2697           2720    
      34          3.1         321.4       1.0X
+codegen = false, adaptive = F                           3689           3722    
      46          2.3         439.8       0.7X
+codegen = false, minCompactionToSpill = 1.05            3701           3713    
      18          2.3         441.2       0.7X
+codegen = false, minCompactionToSpill = 2.0             3703           3763    
      85          2.3         441.4       0.7X
+
+
+================================================================================================
+spill check with an Exchange, keys recur, map holds 40% of them, fast map off
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.5+8-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys recur, map holds 40% of them, fast map 
off:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   
Relative
+------------------------------------------------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F                                                   
              8161           8252         129          2.1         486.4       
1.0X
+codegen = true, minCompactionToSpill = 1.05                                    
              7881           8001         171          2.1         469.7       
1.0X
+codegen = true, minCompactionToSpill = 2.0                                     
              7988           8021          46          2.1         476.1       
1.0X
+codegen = false, adaptive = F                                                  
              8664           8827         230          1.9         516.4       
0.9X
+codegen = false, minCompactionToSpill = 1.05                                   
              8389           8546         222          2.0         500.0       
1.0X
+codegen = false, minCompactionToSpill = 2.0                                    
              8867           8949         116          1.9         528.5       
0.9X
+
+
+================================================================================================
+spill check with an Exchange, keys recur, maps hold 40% of them, fast map on
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.5+8-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys recur, maps hold 40% of them, fast map 
on:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   
Relative
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F                                                   
             7317           7361          62          2.3         436.1       
1.0X
+codegen = true, minCompactionToSpill = 1.05                                    
             7416           7426          15          2.3         442.0       
1.0X
+codegen = true, minCompactionToSpill = 2.0                                     
             7905           7908           5          2.1         471.2       
0.9X
+
+
+================================================================================================
+spill check with an Exchange, hot keys
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.5+8-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, hot keys:  Best Time(ms)   Avg Time(ms)   
Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
+----------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F                           6246           6264     
     26          1.3         744.6       1.0X
+codegen = true, minCompactionToSpill = 1.05            6317           6358     
     58          1.3         753.0       1.0X
+codegen = true, minCompactionToSpill = 2.0             4904           4922     
     26          1.7         584.6       1.3X
+codegen = false, adaptive = F                          7212           7236     
     34          1.2         859.7       0.9X
+codegen = false, minCompactionToSpill = 1.05           7214           7269     
     78          1.2         860.0       0.9X
+codegen = false, minCompactionToSpill = 2.0            5579           5605     
     36          1.5         665.1       1.1X
 
 
diff --git 
a/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-results.txt 
b/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-results.txt
index 5523166ca01f..7335fe8e296c 100644
--- a/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-results.txt
+++ b/sql/core/benchmarks/AdaptivePartialAggregationBenchmark-results.txt
@@ -2,55 +2,132 @@
 high-cardinality input, pass-through at the periodic check
 
================================================================================================
 
-OpenJDK 64-Bit Server VM 17.0.20+8-LTS on Linux 6.17.0-1020-azure
+OpenJDK 64-Bit Server VM 17.0.21+9-LTS on Linux 6.17.0-1022-azure
 AMD EPYC 7763 64-Core Processor
 adaptive partial agg, high card, no spill:  Best Time(ms)   Avg Time(ms)   
Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 
-------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F                        4118           4125        
  11          2.0         490.9       1.0X
-codegen = true, adaptive = T                        2411           2427        
  23          3.5         287.4       1.7X
-codegen = false, adaptive = F                       4846           4861        
  21          1.7         577.7       0.8X
-codegen = false, adaptive = T                       2991           3009        
  25          2.8         356.6       1.4X
+codegen = true, adaptive = F                        4303           4326        
  32          1.9         513.0       1.0X
+codegen = true, adaptive = T                        2621           2627        
   8          3.2         312.4       1.6X
+codegen = false, adaptive = F                       5219           5228        
  12          1.6         622.2       0.8X
+codegen = false, adaptive = T                       3259           3298        
  55          2.6         388.5       1.3X
 
 
 
================================================================================================
 low-cardinality input, pass-through at the periodic check
 
================================================================================================
 
-OpenJDK 64-Bit Server VM 17.0.20+8-LTS on Linux 6.17.0-1020-azure
+OpenJDK 64-Bit Server VM 17.0.21+9-LTS on Linux 6.17.0-1022-azure
 AMD EPYC 7763 64-Core Processor
 adaptive partial agg, low card, no spill:  Best Time(ms)   Avg Time(ms)   
Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 
------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F                        270            311         
 37         62.2          16.1       1.0X
-codegen = true, adaptive = T                        293            318         
 22         57.3          17.5       0.9X
-codegen = false, adaptive = F                      1305           1312         
 10         12.9          77.8       0.2X
-codegen = false, adaptive = T                      1329           1435         
150         12.6          79.2       0.2X
+codegen = true, adaptive = F                        317            339         
 18         53.0          18.9       1.0X
+codegen = true, adaptive = T                        300            321         
 21         56.0          17.9       1.1X
+codegen = false, adaptive = F                      1299           1304         
  7         12.9          77.4       0.2X
+codegen = false, adaptive = T                      1332           1349         
 24         12.6          79.4       0.2X
 
 
 
================================================================================================
 high-cardinality input, pass-through at the spill check
 
================================================================================================
 
-OpenJDK 64-Bit Server VM 17.0.20+8-LTS on Linux 6.17.0-1020-azure
+OpenJDK 64-Bit Server VM 17.0.21+9-LTS on Linux 6.17.0-1022-azure
 AMD EPYC 7763 64-Core Processor
 adaptive partial agg, high card, spill:   Best Time(ms)   Avg Time(ms)   
Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 
------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F                       8196           8247         
 73          1.0         977.0       1.0X
-codegen = true, adaptive = T                       4433           4530         
137          1.9         528.5       1.8X
-codegen = false, adaptive = F                      9544           9555         
 15          0.9        1137.8       0.9X
-codegen = false, adaptive = T                      5243           5395         
215          1.6         625.0       1.6X
+codegen = true, adaptive = F                       8760           8789         
 41          1.0        1044.2       1.0X
+codegen = true, adaptive = T                       4692           4712         
 28          1.8         559.3       1.9X
+codegen = false, adaptive = F                     10134          10142         
 11          0.8        1208.1       0.9X
+codegen = false, adaptive = T                      5904           6003         
140          1.4         703.8       1.5X
 
 
 
================================================================================================
 low-cardinality input, pass-through at the spill check
 
================================================================================================
 
-OpenJDK 64-Bit Server VM 17.0.20+8-LTS on Linux 6.17.0-1020-azure
+OpenJDK 64-Bit Server VM 17.0.21+9-LTS on Linux 6.17.0-1022-azure
 AMD EPYC 7763 64-Core Processor
 adaptive partial agg, low card, spill:    Best Time(ms)   Avg Time(ms)   
Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 
------------------------------------------------------------------------------------------------------------------------
-codegen = true, adaptive = F                        803            813         
 11         20.9          47.8       1.0X
-codegen = true, adaptive = T                        829            839         
  9         20.2          49.4       1.0X
-codegen = false, adaptive = F                      1362           1387         
 35         12.3          81.2       0.6X
-codegen = false, adaptive = T                      1365           1376         
 15         12.3          81.4       0.6X
+codegen = true, adaptive = F                        845            856         
 12         19.9          50.3       1.0X
+codegen = true, adaptive = T                        894            908         
 22         18.8          53.3       0.9X
+codegen = false, adaptive = F                      1508           1521         
 18         11.1          89.9       0.6X
+codegen = false, adaptive = T                      1487           1499         
 16         11.3          88.7       0.6X
+
+
+================================================================================================
+spill check with an Exchange, keys arrive together
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.21+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys arrive together:  Best Time(ms)   Avg 
Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
+---------------------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F                                      6354         
  6448         134          1.3         757.4       1.0X
+codegen = true, minCompactionToSpill = 1.05                       6816         
  6848          45          1.2         812.6       0.9X
+codegen = true, minCompactionToSpill = 2.0                        4959         
  4979          27          1.7         591.2       1.3X
+codegen = false, adaptive = F                                     7335         
  7360          35          1.1         874.4       0.9X
+codegen = false, minCompactionToSpill = 1.05                      7368         
  7444         108          1.1         878.3       0.9X
+codegen = false, minCompactionToSpill = 2.0                       5710         
  5781         101          1.5         680.6       1.1X
+
+
+================================================================================================
+spill check with an Exchange, keys recur
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.21+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys recur:  Best Time(ms)   Avg Time(ms)   
Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
+-----------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F                            2712           2732    
      27          3.1         323.3       1.0X
+codegen = true, minCompactionToSpill = 1.05             2733           2784    
      73          3.1         325.7       1.0X
+codegen = true, minCompactionToSpill = 2.0              2722           2741    
      27          3.1         324.5       1.0X
+codegen = false, adaptive = F                           3738           3745    
      10          2.2         445.6       0.7X
+codegen = false, minCompactionToSpill = 1.05            3728           3853    
     177          2.3         444.4       0.7X
+codegen = false, minCompactionToSpill = 2.0             3889           3897    
      11          2.2         463.6       0.7X
+
+
+================================================================================================
+spill check with an Exchange, keys recur, map holds 40% of them, fast map off
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.21+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys recur, map holds 40% of them, fast map 
off:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   
Relative
+------------------------------------------------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F                                                   
              7715           7775          84          2.2         459.9       
1.0X
+codegen = true, minCompactionToSpill = 1.05                                    
              7764           7785          30          2.2         462.8       
1.0X
+codegen = true, minCompactionToSpill = 2.0                                     
              8011           8020          12          2.1         477.5       
1.0X
+codegen = false, adaptive = F                                                  
              8729           8839         156          1.9         520.3       
0.9X
+codegen = false, minCompactionToSpill = 1.05                                   
              9086           9099          19          1.8         541.5       
0.8X
+codegen = false, minCompactionToSpill = 2.0                                    
              9106           9155          68          1.8         542.8       
0.8X
+
+
+================================================================================================
+spill check with an Exchange, keys recur, maps hold 40% of them, fast map on
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.21+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, keys recur, maps hold 40% of them, fast map 
on:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   
Relative
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F                                                   
             7826           7860          48          2.1         466.5       
1.0X
+codegen = true, minCompactionToSpill = 1.05                                    
             7637           7850         301          2.2         455.2       
1.0X
+codegen = true, minCompactionToSpill = 2.0                                     
             8170           8221          72          2.1         487.0       
1.0X
+
+
+================================================================================================
+spill check with an Exchange, hot keys
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.21+9-LTS on Linux 6.17.0-1022-azure
+AMD EPYC 7763 64-Core Processor
+adaptive partial agg, spill check, hot keys:  Best Time(ms)   Avg Time(ms)   
Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
+----------------------------------------------------------------------------------------------------------------------------
+codegen = true, adaptive = F                           6443           6507     
     89          1.3         768.1       1.0X
+codegen = true, minCompactionToSpill = 1.05            6544           6556     
     17          1.3         780.1       1.0X
+codegen = true, minCompactionToSpill = 2.0             4985           4987     
      2          1.7         594.2       1.3X
+codegen = false, adaptive = F                          7311           7320     
     13          1.1         871.6       0.9X
+codegen = false, minCompactionToSpill = 1.05           7310           7317     
     11          1.1         871.4       0.9X
+codegen = false, minCompactionToSpill = 2.0            5761           5790     
     41          1.5         686.8       1.1X
 
 
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
index ffb6773bfab8..7ed81e5818ff 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
@@ -140,7 +140,8 @@ case class HashAggregateExec(
             numPassthroughRows,
             adaptivePartialAggEnabled,
             adaptiveMinRows,
-            adaptiveMinCompaction)
+            adaptiveMinCompaction,
+            adaptiveMinCompactionToSpill)
         if (!hasInput && groupingExpressions.isEmpty) {
           numOutputRows += 1
           
Iterator.single[UnsafeRow](aggregationIterator.outputForEmptyGroupingKeyWithoutInput())
@@ -208,10 +209,15 @@ case class HashAggregateExec(
       requiredChildDistributionExpressions.isEmpty
   }
 
-  // The number of rows between two compaction-ratio evaluations, and the 
ratio below which the
-  // partial aggregation is considered ineffective. Only read when the feature 
applies.
+  // The number of rows between two compaction-ratio evaluations, and the 
ratios below which the
+  // partial aggregation is considered ineffective: at the periodic check, and 
when the map is
+  // about to spill. Keeping the aggregation past that point means aggregating 
the rest of the
+  // input through spills, so the spill check never demands less compaction 
than the periodic one.
+  // Only read when the feature applies.
   private val adaptiveMinRows: Long = conf.adaptivePartialAggregationMinRows
   private val adaptiveMinCompaction: Double = 
conf.adaptivePartialAggregationMinCompaction
+  private val adaptiveMinCompactionToSpill: Double =
+    math.max(adaptiveMinCompaction, 
conf.adaptivePartialAggregationMinCompactionToSpill)
 
   // The name for Fast HashMap
   private var fastHashMapTerm: String = _
@@ -253,6 +259,13 @@ case class HashAggregateExec(
   // keeps both sides of the ratio on the same epoch. Once the fast map fills 
it stops accepting
   // keys, so the difference settles at zero and the ratio is the regular 
map's alone.
   private var adaptiveFastKeysAtSpillTerm: String = _
+  // The epoch's key count at its last periodic check, and at the check before 
that, where the
+  // window of recent rows that the spill check judges starts (see 
`adaptiveIneffective`). The
+  // checks run at multiples of `minRows`, and the next check row advances 
only past a check that
+  // keeps aggregating, so the window starts `2 * minRows` rows before the 
next check row, or at
+  // the epoch start before the second check.
+  private var adaptiveCheckKeysTerm: String = _
+  private var adaptiveWindowKeysTerm: String = _
   // The name of the generated output function, promoted to a field so 
`doConsumeWithKeys` can emit
   // pass-through rows directly from within the build loop.
   private var outputFunc: String = _
@@ -575,6 +588,8 @@ case class HashAggregateExec(
         ctx.addMutableState(CodeGenerator.JAVA_BOOLEAN, "adaptiveMapSetupDone")
       adaptiveFastKeysAtSpillTerm =
         ctx.addMutableState(CodeGenerator.JAVA_INT, "adaptiveFastKeysAtSpill")
+      adaptiveCheckKeysTerm = ctx.addMutableState(CodeGenerator.JAVA_LONG, 
"adaptiveCheckKeys")
+      adaptiveWindowKeysTerm = ctx.addMutableState(CodeGenerator.JAVA_LONG, 
"adaptiveWindowKeys")
       adaptivePendingRowsTerm = ctx.addMutableState(
         "java.util.LinkedList<UnsafeRow[]>", "adaptivePendingRows",
         v => s"$v = new java.util.LinkedList<UnsafeRow[]>();", forceInline = 
true)
@@ -980,25 +995,31 @@ case class HashAggregateExec(
       case _ => ("true", "", "")
     }
 
-    // The compaction ratio is measured at the operator level: all processed 
rows against the keys
-    // held by both maps, so two-level-map routing does not change the 
decision. The same predicate
-    // decides both check points -- periodically every `minRows` rows, and 
right before the map
-    // would spill (in which case the spill is skipped entirely). Pass-through 
cannot be undone
-    // within the task, so the spill check, like the periodic one, waits until 
the current epoch
-    // has processed `minRows` rows; a map that fills sooner spills as usual. 
`minRows = 0`
-    // disables the periodic check: the row count is only ever compared after 
being incremented
-    // past 0, so it never matches and only the spill check remains, without a 
row floor.
-    val adaptiveIneffective = if (adaptivePartialAggEnabled) {
-      val totalKeys = if (isFastHashMapEnabled) {
-        s"($fastHashMapTerm.getNumKeys() - $adaptiveFastKeysAtSpillTerm + " +
-          s"$hashMapTerm.getNumKeys())"
-      } else {
-        s"$hashMapTerm.getNumKeys()"
-      }
-      s"$processedRowsTerm < (double) $totalKeys * ${adaptiveMinCompaction}D"
+    // The compaction ratio is measured at the operator level: processed rows 
against the keys
+    // held by both maps, so two-level-map routing does not change the 
decision. It is evaluated at
+    // two check points:
+    //   - periodically, every `minRows` rows, over the whole epoch against 
`minCompaction`;
+    //   - right before the map would spill, over the recent rows against the 
higher
+    //     `minCompactionToSpill`, in which case the map does not spill and 
the rest of the input
+    //     is passed through. Keeping the aggregation then means aggregating 
the rest of the input
+    //     through spills, which pays off only if that input keeps repeating 
keys. The recent rows,
+    //     those since the periodic check before last, measure that: when keys 
repeat across the
+    //     input, more of them find their key already in the map as it fills, 
so their ratio
+    //     climbs above the epoch's; when each key's rows arrive together, the 
two ratios match.
+    // Pass-through cannot be undone within the task, so the spill check, like 
the periodic one,
+    // waits until the current epoch has processed `minRows` rows; a map that 
fills sooner spills
+    // as usual. `minRows = 0` disables the periodic check: the row count is 
only ever compared
+    // after being incremented past 0, so it never matches and only the spill 
check remains,
+    // without a row floor and over the whole epoch.
+    def adaptiveEpochKeys: String = if (isFastHashMapEnabled) {
+      s"((long) $fastHashMapTerm.getNumKeys() - $adaptiveFastKeysAtSpillTerm + 
" +
+        s"$hashMapTerm.getNumKeys())"
     } else {
-      ""
+      s"((long) $hashMapTerm.getNumKeys())"
     }
+    // Whether `rows` processed rows fail to collapse into `keys` keys by at 
least `threshold`.
+    def adaptiveIneffective(rows: String, keys: String, threshold: Double): 
String =
+      s"$rows < (double) $keys * ${threshold}D"
 
     val findOrInsertRegularHashMap: String = {
       // Assumes the grouping key projection (`unsafeRowKeyCode.code`) has 
already run for this row,
@@ -1055,7 +1076,14 @@ case class HashAggregateExec(
              |$processedRowsTerm = 0L;
              |$adaptiveNextCheckRowTerm = ${adaptiveMinRows}L;
              |$snapshotFastKeys
+             |$adaptiveCheckKeysTerm = 0L;
+             |$adaptiveWindowKeysTerm = 0L;
            """.stripMargin
+        val windowRows = s"Math.max(0L, $adaptiveNextCheckRowTerm - ${2 * 
adaptiveMinRows}L)"
+        val recentIneffective = adaptiveIneffective(
+          s"($processedRowsTerm - $windowRows)",
+          s"($adaptiveEpochKeys - $adaptiveWindowKeysTerm)",
+          adaptiveMinCompactionToSpill)
         // The spill check needs at least one processed row even when `minRows 
= 0`.
         val spillCheckMinRows = math.max(adaptiveMinRows, 1L)
         s"""
@@ -1064,7 +1092,7 @@ case class HashAggregateExec(
            |if (!$adaptivePassThroughTerm) {
            |  $probeRegularMap
            |  if ($unsafeRowBuffer == null) {
-           |    if ($processedRowsTerm >= ${spillCheckMinRows}L && 
$adaptiveIneffective) {
+           |    if ($processedRowsTerm >= ${spillCheckMinRows}L && 
$recentIneffective) {
            |      $adaptivePassThroughTerm = true;
            |    } else {
            |      $spillAndRestartEpoch
@@ -1138,6 +1166,8 @@ case class HashAggregateExec(
         } else {
           s"($unsafeRowBuffer != null)"
         }
+        val epochIneffective =
+          adaptiveIneffective(processedRowsTerm, adaptiveEpochKeys, 
adaptiveMinCompaction)
         // The grouping key was already projected in 
`findOrInsertRegularHashMap`
         // (`unsafeRowKeyCode.code`), so `unsafeRowKeyCode.value` holds this 
row's key. Only build
         // the single-row partial buffer here: reset the reusable buffer to 
the initial aggregation
@@ -1148,10 +1178,12 @@ case class HashAggregateExec(
            |  if (!$adaptivePassThroughTerm) {
            |    $processedRowsTerm += 1;
            |    if ($processedRowsTerm == $adaptiveNextCheckRowTerm) {
-           |      if ($adaptiveIneffective) {
+           |      if ($epochIneffective) {
            |        $adaptivePassThroughTerm = true;
            |      } else {
            |        $adaptiveNextCheckRowTerm += ${adaptiveMinRows}L;
+           |        $adaptiveWindowKeysTerm = $adaptiveCheckKeysTerm;
+           |        $adaptiveCheckKeysTerm = $adaptiveEpochKeys;
            |      }
            |    }
            |  }
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala
index c5f538841505..82806ecccd8a 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala
@@ -100,7 +100,8 @@ class TungstenAggregationIterator(
     numPassthroughRows: SQLMetric,
     adaptivePartialAggEnabled: Boolean,
     adaptiveMinRows: Long,
-    adaptiveMinCompaction: Double)
+    adaptiveMinCompaction: Double,
+    adaptiveMinCompactionToSpill: Double)
   extends AggregationIterator(
     partIndex,
     groupingExpressions,
@@ -189,14 +190,18 @@ class TungstenAggregationIterator(
   // When adaptive partial aggregation is enabled 
(`adaptivePartialAggEnabled`), the processing may
   // stop early and switch to pass-through mode: the remaining input rows are 
not added to the map
   // but are instead emitted as single-row partial buffers by the output stage 
(see
-  // `nextPassThroughOutput`). One predicate decides both check points -- the 
aggregation is
-  // ineffective when `processedRows < distinctKeys * minCompaction`, i.e. it 
does not collapse
-  // `minCompaction` rows into one key:
-  //   - periodically, every `minRows` processed rows;
-  //   - when the map is about to spill, in which case the spill is skipped 
entirely and the row
-  //     that could not be inserted becomes the first pass-through row. 
Pass-through cannot be
-  //     undone within the task, so like the periodic check this waits until 
the current epoch has
-  //     processed `minRows` rows; a map that fills sooner spills as usual.
+  // `nextPassThroughOutput`). The aggregation is ineffective when `rows` 
processed rows do not
+  // collapse into their `keys` keys by at least `threshold`, checked at two 
points:
+  //   - periodically, every `minRows` processed rows, over the whole epoch 
against
+  //     `minCompaction`;
+  //   - when the map is about to spill, over the recent rows (those since the 
periodic check
+  //     before last) against the higher `minCompactionToSpill`. Keeping the 
aggregation then
+  //     means aggregating the rest of the input through spills, which pays 
off only if that input
+  //     keeps repeating keys; when keys repeat across the input, the ratio of 
the recent rows
+  //     climbs above the epoch's as the map fills. Below the threshold, the 
map does not spill
+  //     and the row that could not be inserted becomes the first pass-through 
row. Pass-through
+  //     cannot be undone within the task, so like the periodic check this 
waits until the current
+  //     epoch has processed `minRows` rows; a map that fills sooner spills as 
usual.
   // A spill starts a new in-memory map epoch: the row counters restart so 
that epoch is judged on
   // its own rows, which lets an input whose cardinality only turns 
unfavorable later still be
   // passed through. Pass-through may therefore coexist with earlier spills. 
The output order
@@ -225,10 +230,18 @@ class TungstenAggregationIterator(
       // The spill check waits for `minRows` rows of the current epoch, and 
for at least one row
       // when `minRows = 0`.
       val spillCheckMinRows = math.max(minRows, 1L)
-      // The partial aggregation is ineffective when it does not collapse 
`minCompaction` rows into
-      // one key. There is no fast map on this path, so the map's keys are all 
the operator holds.
-      def ineffective(): Boolean =
-        processedRows < hashMap.getNumKeys().toDouble * adaptiveMinCompaction
+      // The epoch's key count at its last periodic check, and at the check 
before that, where the
+      // window of recent rows that the spill check judges starts. The checks 
run at multiples of
+      // `minRows`, and `nextCheckRow` advances only past a check that keeps 
aggregating, so the
+      // window starts `2 * minRows` rows before `nextCheckRow`, or at the 
epoch start before the
+      // second check.
+      var checkKeys = 0L
+      var windowKeys = 0L
+      def windowRows: Long = math.max(0L, nextCheckRow - 2 * minRows)
+      // Whether `rows` processed rows fail to collapse into `keys` keys by at 
least `threshold`.
+      // There is no fast map on this path, so the map's keys are all the 
operator holds.
+      def ineffective(rows: Long, keys: Long, threshold: Double): Boolean =
+        rows < keys.toDouble * threshold
       while (inputIter.hasNext && !passThrough) {
         val newInput = inputIter.next()
         val groupingKey = groupingProjection.apply(newInput)
@@ -240,7 +253,9 @@ class TungstenAggregationIterator(
           // The map is full and would normally spill. Adaptive partial 
aggregation may instead
           // bypass: keep the in-memory map as-is, pass this row and all 
remaining rows through,
           // and skip the spill entirely.
-          if (adaptivePartialAggEnabled && processedRows >= spillCheckMinRows 
&& ineffective()) {
+          if (adaptivePartialAggEnabled && processedRows >= spillCheckMinRows 
&&
+              ineffective(processedRows - windowRows, hashMap.getNumKeys() - 
windowKeys,
+                adaptiveMinCompactionToSpill)) {
             passThrough = true
             // `newInput` could not be inserted; stash a copy as the first 
pass-through row so it
             // is not lost when we drain the rest of `inputIter`.
@@ -257,6 +272,8 @@ class TungstenAggregationIterator(
             // epoch alone decides whether the remaining rows are passed 
through.
             processedRows = 0L
             nextCheckRow = minRows
+            checkKeys = 0L
+            windowKeys = 0L
             buffer = hashMap.getAggregationBufferFromUnsafeRow(groupingKey)
             if (buffer == null) {
               // failed to allocate the first page
@@ -271,10 +288,12 @@ class TungstenAggregationIterator(
           // Periodic check: if the aggregation is not collapsing enough rows, 
bypass it for the
           // rest of the input.
           if (adaptivePartialAggEnabled && processedRows == nextCheckRow) {
-            if (ineffective()) {
+            if (ineffective(processedRows, hashMap.getNumKeys(), 
adaptiveMinCompaction)) {
               passThrough = true
             } else {
               nextCheckRow += minRows
+              windowKeys = checkKeys
+              checkKeys = hashMap.getNumKeys()
             }
           }
         }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/aggregate/AdaptivePartialAggregationSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/aggregate/AdaptivePartialAggregationSuite.scala
index c4240ca308cf..d88b24cb8510 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/aggregate/AdaptivePartialAggregationSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/aggregate/AdaptivePartialAggregationSuite.scala
@@ -714,32 +714,39 @@ class AdaptivePartialAggregationSuite extends QueryTest 
with SharedSparkSession
   // compaction ratio high enough that they decline to bypass -- they cover 
the eligible-but-
   // declining side. (Widening the rollup lowers the ratio: with five distinct 
columns the same
   // shape does bypass.) The GROUPING SETS and multi-distinct tests, and 
`pass-through fires for
-  // high-cardinality input below an Expand`, cover an Expand that bypasses.
+  // high-cardinality input below an Expand`, cover an Expand that bypasses. 
The forced-spill
+  // epochs of ROLLUP and CUBE compact too little to keep aggregating past a 
spill under the
+  // default `minCompactionToSpill`, which rightly passes them through, so 
they pin it to 1.0 to
+  // judge the spill check by `minCompaction` as well.
 
   test("results unchanged for ROLLUP (Expand below partial aggregate)") {
     // The grand-total group repeats on every expanded row, so the compaction 
ratio stays above the
     // threshold and the partial aggregate declines to bypass in every cell.
-    checkAdaptiveMatchesReference(
-      expectBypass = false,
-      build = { parts =>
-        spark.range(0, 400, 1, parts)
-          .select(($"id" % 200) as "k1", ($"id" % 7) as "k2", $"id" as "v")
-          .rollup($"k1", $"k2")
-          .agg(sum($"v") as "s", count(lit(1)) as "c")
-      })
+    
withSQLConf(SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key -> 
"1.0") {
+      checkAdaptiveMatchesReference(
+        expectBypass = false,
+        build = { parts =>
+          spark.range(0, 400, 1, parts)
+            .select(($"id" % 200) as "k1", ($"id" % 7) as "k2", $"id" as "v")
+            .rollup($"k1", $"k2")
+            .agg(sum($"v") as "s", count(lit(1)) as "c")
+        })
+    }
   }
 
   test("results unchanged for CUBE (Expand below partial aggregate)") {
     // Same as ROLLUP: the grand-total group keeps the ratio above the 
threshold, so nothing
     // bypasses.
-    checkAdaptiveMatchesReference(
-      expectBypass = false,
-      build = { parts =>
-        spark.range(0, 400, 1, parts)
-          .select(($"id" % 150) as "k1", ($"id" % 5) as "k2", $"id" as "v")
-          .cube($"k1", $"k2")
-          .agg(sum($"v") as "s", count(lit(1)) as "c")
-      })
+    
withSQLConf(SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key -> 
"1.0") {
+      checkAdaptiveMatchesReference(
+        expectBypass = false,
+        build = { parts =>
+          spark.range(0, 400, 1, parts)
+            .select(($"id" % 150) as "k1", ($"id" % 5) as "k2", $"id" as "v")
+            .cube($"k1", $"k2")
+            .agg(sum($"v") as "s", count(lit(1)) as "c")
+        })
+    }
   }
 
   test("results unchanged for GROUPING SETS (Expand below partial aggregate)") 
{
@@ -1171,11 +1178,12 @@ class AdaptivePartialAggregationSuite extends QueryTest 
with SharedSparkSession
 
   test("the spill check bypasses between periodic checks once minRows rows are 
processed") {
     // With `minRows = 10` the periodic check runs at the 10th aggregated row, 
when ids 0-9 have
-    // collapsed into two keys (ratio 5.0, above `minCompaction = 2.0`), so it 
keeps aggregating.
-    // Ids 10-18 then add nine distinct keys, and the forced fallback makes 
the map full when id
-    // 19 arrives. That is past the floor but before the next periodic check 
at 20 rows, and the
-    // ratio of 19 rows over 11 keys is below 2.0, so the spill check bypasses 
at exactly that row:
-    // ids 19-399 stream through and the sorter is never created.
+    // collapsed into two keys (ratio 5.0, above `minCompaction`), so it keeps 
aggregating. Ids
+    // 10-18 then add nine distinct keys, and the forced fallback makes the 
map full when id 19
+    // arrives. That is past the floor but before the next periodic check at 
20 rows. With a single
+    // periodic check so far, the spill check judges the whole epoch, and 19 
rows over 11 keys is
+    // below `minCompactionToSpill = 2.0`, so it bypasses at exactly that row: 
ids 19-399 stream
+    // through and the sorter is never created.
     Seq(true, false).foreach { wholeStage =>
       withSQLConf(
         (Seq(
@@ -1183,7 +1191,7 @@ class AdaptivePartialAggregationSuite extends QueryTest 
with SharedSparkSession
           SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> wholeStage.toString,
           SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> "false",
           SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key -> "10",
-          SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION.key -> "2.0",
+          SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key -> 
"2.0",
           "spark.sql.TungstenAggregate.testFallbackStartsAt" -> "1, 19") ++
           fixedPlanConfs): _*) {
         val df = () => spark.range(0, 400, 1, 1)
@@ -1207,12 +1215,104 @@ class AdaptivePartialAggregationSuite extends 
QueryTest with SharedSparkSession
     }
   }
 
+  test("the spill check passes through an epoch that compacts too little to 
spill") {
+    // Keys `floor(2 * id / 3)` collapse every three rows into two keys, a 
steady ratio of about
+    // 1.5 with no key repeating once its rows have passed. That is above 
`minCompaction`, so the
+    // periodic checks at 10, 20 and 30 rows keep aggregating. The forced 
fallback fills the map
+    // when the next row arrives, and the spill check judges the rows since 
the check before last,
+    // which compact about the same, 1.4 to 1.7. Keeping the aggregation would 
aggregate the rest
+    // through spills for that little, so the default `minCompactionToSpill` 
(2.0) passes the rest
+    // through instead and the sorter is never created. Lowering it to 1.3 
keeps the old behavior:
+    // every epoch spills, and nothing is passed through. Two input partitions 
put an `Exchange`
+    // between the Partial and the Final, so passed-through rows go through 
the output buffer and
+    // the resumed build; one partition fuses the two.
+    for {
+      (minCompactionToSpill, shouldBypass) <- Seq(None -> true, Some("1.3") -> 
false)
+      partitions <- Seq(1, 2)
+      wholeStage <- Seq(true, false)
+    } {
+      withSQLConf(
+        (Seq(
+          SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key -> "true",
+          SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> wholeStage.toString,
+          SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> "false",
+          SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key -> "10",
+          "spark.sql.TungstenAggregate.testFallbackStartsAt" -> "1, 30") ++
+          minCompactionToSpill.map(
+            SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key 
-> _) ++
+          fixedPlanConfs): _*) {
+        val df = () => spark.range(0, 400, 1, partitions)
+          .select(($"id" * 2 / 3).cast("long").cast("string") as "k", $"id" as 
"v")
+          .groupBy($"k")
+          .agg(sum($"v") as "s")
+        withClue(s"minCompactionToSpill=$minCompactionToSpill 
partitions=$partitions " +
+            s"wholeStage=$wholeStage") {
+          val c = runAndReadCounters(df)
+          if (shouldBypass) {
+            assert(c.passedThrough == 400 - 30 * partitions,
+              "each task should pass through from the row that found its map 
full")
+            assert(c.tasksFallBacked == 0,
+              "the spill check must replace the sort fallback, not trigger it")
+          } else {
+            assert(c.passedThrough == 0,
+              "an epoch above minCompactionToSpill must keep aggregating")
+            assert(c.tasksFallBacked > 0, "an epoch that keeps aggregating 
must spill")
+          }
+        }
+      }
+    }
+  }
+
+  test("the spill check keeps aggregating when the recent rows repeat keys") {
+    // Ids 0-35 map to `floor(2 * id / 3)`, keys 0-23, and ids 36-53 to `id % 
20`, keys the map
+    // already holds. The periodic checks at 10, 20, 30 and 40 rows see ratios 
from 1.4 to 1.7, so
+    // they keep aggregating, and the forced fallback fills the map when id 45 
arrives, after 45
+    // rows over 24 keys. That ratio of 1.875 is below `minCompactionToSpill`, 
but the spill check
+    // judges the rows since the check before last: the 15 rows since the 
check at 30 rows added
+    // only 4 keys, a ratio of 3.75, because the rest found their key in the 
map. So the map spills
+    // and keeps aggregating, and the last 9 rows are too few for another 
check. Demanding 4.0 at
+    // the spill check passes those 9 rows through instead.
+    for {
+      (minCompactionToSpill, passedThrough) <- Seq(None -> 0, Some("4.0") -> 9)
+      wholeStage <- Seq(true, false)
+    } {
+      withSQLConf(
+        (Seq(
+          SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key -> "true",
+          SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> wholeStage.toString,
+          SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> "false",
+          SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key -> "10",
+          "spark.sql.TungstenAggregate.testFallbackStartsAt" -> "1, 45") ++
+          minCompactionToSpill.map(
+            SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key 
-> _) ++
+          fixedPlanConfs): _*) {
+        val df = () => spark.range(0, 54, 1, 1)
+          .select(when($"id" < 36, ($"id" * 2 / 
3).cast("long")).otherwise($"id" % 20)
+            .cast("string") as "k", $"id" as "v")
+          .groupBy($"k")
+          .agg(sum($"v") as "s")
+        withClue(s"minCompactionToSpill=$minCompactionToSpill 
wholeStage=$wholeStage") {
+          val c = runAndReadCounters(df)
+          assert(c.passedThrough == passedThrough)
+          assert((c.tasksFallBacked > 0) == (passedThrough == 0),
+            "the map spills exactly when the spill check keeps aggregating")
+        }
+      }
+    }
+  }
+
   test("a new in-memory map epoch after a spill can still bypass") {
     // A spill starts a new in-memory map epoch and restarts the row counters, 
so an input whose
-    // cardinality only turns unfavorable after an early spill is still 
caught. The first 100 rows
+    // cardinality only turns unfavorable after an early spill is still 
caught. The first 80 rows
     // repeat 5 keys and keep the aggregation effective while the forced 
fallback makes the map
-    // spill; the remaining 300 rows are fully distinct, so the new epoch is 
judged ineffective and
-    // the rest of the input is passed through. Both the spill and the bypass 
must be observable.
+    // spill at ids 40 and 80; the remaining 320 rows are fully distinct, so 
the third epoch's
+    // first periodic check, at its 8th row (id 87), judges it ineffective and 
the rest of the
+    // input is passed through. With the two-level map on the codegen path, 
the fast map absorbs
+    // keys 0-3, so each forced spill waits for the next row that needs the 
regular map, and the
+    // third epoch starts at id 84 instead. The exact count also pins that the 
check row restarts
+    // with the epoch. `minCompactionToSpill` is pinned to 1.5 so the count 
does not rest on a tie:
+    // in the two-level codegen cell, the window when id 84 finds the map full 
holds 8 rows over 4
+    // keys, exactly the default 2.0.
     for {
       wholeStage <- Seq(true, false)
       twoLevelMap <- Seq(true, false)
@@ -1224,18 +1324,20 @@ class AdaptivePartialAggregationSuite extends QueryTest 
with SharedSparkSession
           SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> wholeStage.toString,
           SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> twoLevelMap.toString,
           SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key -> "8",
+          SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key -> 
"1.5",
           "spark.sql.TungstenAggregate.testFallbackStartsAt" -> s"$fastCap, 
40") ++
           fixedPlanConfs): _*) {
         val df = () => spark.range(0, 400, 1, 1)
-          .select(when($"id" < 100, $"id" % 5).otherwise($"id") as "k", $"id" 
as "v")
+          .select(when($"id" < 80, $"id" % 5).otherwise($"id") as "k", $"id" 
as "v")
           .groupBy($"k")
           .agg(sum($"v") as "s")
         withClue(s"wholeStage=$wholeStage twoLevelMap=$twoLevelMap") {
           val c = runAndReadCounters(df)
           assert(c.tasksFallBacked > 0,
             "the low-cardinality prefix should still spill and fall back to 
sort")
-          assert(c.passedThrough > 0,
-            "the high-cardinality tail after the spill should be passed 
through")
+          val firstPassedThrough = if (wholeStage && twoLevelMap) 92 else 88
+          assert(c.passedThrough == 400 - firstPassedThrough,
+            "the high-cardinality tail after the spills should be passed 
through")
         }
       }
     }
@@ -1278,8 +1380,15 @@ class AdaptivePartialAggregationSuite extends QueryTest 
with SharedSparkSession
     // failed insertion that becomes the first pass-through row, so both 
execution paths must judge
     // the same row set and reach the same decision. `id % 40` over 400 rows 
gives 40 keys when the
     // map fills at 50 aggregated rows, i.e. a compaction ratio of exactly 
1.25: demanding 1.25 the
-    // aggregation is kept (`50 < 40 * 1.25` is false), demanding 1.3 it is 
bypassed.
-    Seq(1.25 -> false, 1.3 -> true).foreach { case (minCompaction, 
shouldBypass) =>
+    // aggregation is kept (`50 < 40 * 1.25` is false), demanding 1.3 it is 
bypassed. The spill
+    // check demands the larger of `minCompaction` and `minCompactionToSpill`, 
so either one can
+    // set the boundary, from either side.
+    Seq(
+      (1.0, 1.25, false),
+      (1.0, 1.3, true),
+      (1.25, 1.0, false),
+      (1.3, 1.0, true)
+    ).foreach { case (minCompaction, minCompactionToSpill, shouldBypass) =>
       val passedThroughPerCodegen = Seq(true, false).map { wholeStage =>
         withSQLConf(
           (Seq(
@@ -1287,21 +1396,25 @@ class AdaptivePartialAggregationSuite extends QueryTest 
with SharedSparkSession
             SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> wholeStage.toString,
             SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> "false",
             // `minRows = 0` keeps the periodic check out of the picture and 
leaves the spill check
-            // without a row floor, so the spill check alone decides at the 
boundary.
+            // without a row floor and judging the whole epoch, so it alone 
decides at the boundary.
             SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key -> "0",
             SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION.key -> 
minCompaction.toString,
+            SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key ->
+              minCompactionToSpill.toString,
             "spark.sql.TungstenAggregate.testFallbackStartsAt" -> "1, 50") ++
             fixedPlanConfs): _*) {
           val df = () => spark.range(0, 400, 1, 1)
             .select(($"id" % 40).cast("string") as "k", $"id" as "v")
             .groupBy($"k")
             .agg(sum($"v") as "s")
-          withClue(s"minCompaction=$minCompaction wholeStage=$wholeStage") {
+          withClue(s"minCompaction=$minCompaction 
minCompactionToSpill=$minCompactionToSpill " +
+              s"wholeStage=$wholeStage") {
             numPassthroughRows(df)
           }
         }
       }
-      withClue(s"minCompaction=$minCompaction 
passedThrough=$passedThroughPerCodegen") {
+      withClue(s"minCompaction=$minCompaction 
minCompactionToSpill=$minCompactionToSpill " +
+          s"passedThrough=$passedThroughPerCodegen") {
         assert(passedThroughPerCodegen.forall(_ > 0) == shouldBypass,
           s"expected bypass=$shouldBypass at the ratio boundary")
         assert(passedThroughPerCodegen.map(_ > 0).distinct.length == 1,
@@ -1310,6 +1423,114 @@ class AdaptivePartialAggregationSuite extends QueryTest 
with SharedSparkSession
     }
   }
 
+  test("the spill check judges the recent rows at their exact ratio boundary") 
{
+    // With `minRows = 10`, ids 0-9 add 5 keys, two rows each, ids 10-19 add 
10 more, one each, and
+    // ids 20-29 add 5 more, two rows each, so the periodic checks at 10, 20 
and 30 rows see 2.0,
+    // 1.33 and 1.5 and keep aggregating. The check at 30 rows runs on id 29, 
so when id 30 finds
+    // the map full, the rows since the check before last are ids 20-29: 10 
rows over 5 keys, a
+    // ratio of exactly 2.0. Demanding 2.0 the aggregation is kept (`10 < 5 * 
2.0` is false) and
+    // the map spills; demanding 2.1 the last 10 rows are passed through. A 
window starting one
+    // check earlier (20 rows over 15 keys) or the whole epoch (30 rows over 
20 keys) would pass
+    // them through at both.
+    for {
+      (minCompactionToSpill, passedThrough) <- Seq(2.0 -> 0, 2.1 -> 10)
+      wholeStage <- Seq(true, false)
+    } {
+      withSQLConf(
+        (Seq(
+          SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key -> "true",
+          SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> wholeStage.toString,
+          SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> "false",
+          SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key -> "10",
+          SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key ->
+            minCompactionToSpill.toString,
+          "spark.sql.TungstenAggregate.testFallbackStartsAt" -> "1, 30") ++
+          fixedPlanConfs): _*) {
+        val df = () => spark.range(0, 40, 1, 1)
+          .select(when($"id" < 10, $"id" / 2).when($"id" < 20, $"id" - 5)
+            .when($"id" < 30, ($"id" - 20) / 2 + 15).otherwise(($"id" - 30) / 
2 + 20)
+            .cast("long").cast("string") as "k", $"id" as "v")
+          .groupBy($"k")
+          .agg(sum($"v") as "s")
+        withClue(s"minCompactionToSpill=$minCompactionToSpill 
wholeStage=$wholeStage") {
+          val c = runAndReadCounters(df)
+          assert(c.passedThrough == passedThrough)
+          assert((c.tasksFallBacked > 0) == (passedThrough == 0),
+            "the map spills exactly when the spill check keeps aggregating")
+        }
+      }
+    }
+  }
+
+  test("the spill check of a new map epoch does not count the previous epoch's 
keys") {
+    // With `minRows = 10`, the forced fallback fills every epoch at 19 rows, 
after one periodic
+    // check, so the spill check judges the whole epoch. The first epoch, ids 
0-18 over 5 keys,
+    // compacts by 3.8 and spills. The second, ids 19-37, has 8 keys at its 
check at 10 rows and
+    // 12 at the fill, so it compacts by 19 / 12 and the last 12 rows are 
passed through. Keeping
+    // the key count of the first epoch's check as the start of the second 
epoch's window would
+    // judge 19 rows over 7 keys and keep aggregating.
+    Seq(true, false).foreach { wholeStage =>
+      withSQLConf(
+        (Seq(
+          SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key -> "true",
+          SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> wholeStage.toString,
+          SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> "false",
+          SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key -> "10",
+          SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key -> 
"2.0",
+          "spark.sql.TungstenAggregate.testFallbackStartsAt" -> "1, 19") ++
+          fixedPlanConfs): _*) {
+        val df = () => spark.range(0, 50, 1, 1)
+          .select(when($"id" < 19, $"id" % 5).when($"id" < 29, ($"id" - 19) % 
8 + 100)
+            .when($"id" < 38, ($"id" - 29) % 4 + 108).otherwise(0L)
+            .cast("string") as "k", $"id" as "v")
+          .groupBy($"k")
+          .agg(sum($"v") as "s")
+        withClue(s"wholeStage=$wholeStage") {
+          val c = runAndReadCounters(df)
+          assert(c.passedThrough == 12)
+          assert(c.tasksFallBacked > 0, "the first epoch should spill")
+        }
+      }
+    }
+  }
+
+  test("the spill check's window counts the keys of both maps") {
+    // With the two-level map on the codegen path and a 16-key fast map, ids 
0-9 add 10 keys and
+    // ids 10-19 add 5 more, two rows each, all held by the fast map, and id 
20 fills it. Id 21
+    // needs the regular map, which the forced fallback reports full after 20 
rows, so the spill
+    // check judges the rows since the check before last, at 10 rows: ids 
10-20, 11 rows over 6
+    // keys, a ratio of 1.83. Demanding 1.5 the aggregation is kept and the 
map spills; demanding
+    // 1.9 the last 9 rows are passed through. Leaving the fast map's keys out 
of the window's
+    // start would judge 11 rows over 16 keys and pass them through at both, 
while counting them
+    // twice would keep aggregating at both. `minCompaction = 1.0` keeps the 
periodic check at 10
+    // rows, 10 rows over 10 keys, aggregating.
+    Seq(1.5 -> 0, 1.9 -> 9).foreach { case (minCompactionToSpill, 
passedThrough) =>
+      withSQLConf(
+        (Seq(
+          SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key -> "true",
+          SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> "true",
+          SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> "true",
+          SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_ROWS.key -> "10",
+          SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION.key -> "1.0",
+          SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key ->
+            minCompactionToSpill.toString,
+          "spark.sql.TungstenAggregate.testFallbackStartsAt" -> "16, 20") ++
+          fixedPlanConfs): _*) {
+        val df = () => spark.range(0, 30, 1, 1)
+          .select(when($"id" < 10, $"id").when($"id" < 20, ($"id" - 10) / 2 + 
10)
+            .when($"id" === 20, 15L).otherwise($"id" + 16).cast("long") as 
"k", $"id" as "v")
+          .groupBy($"k")
+          .agg(sum($"v") as "s")
+        withClue(s"minCompactionToSpill=$minCompactionToSpill") {
+          val c = runAndReadCounters(df)
+          assert(c.passedThrough == passedThrough)
+          assert((c.tasksFallBacked > 0) == (passedThrough == 0),
+            "the map spills exactly when the spill check keeps aggregating")
+        }
+      }
+    }
+  }
+
   test("a very high minCompaction always bypasses once minRows has been 
processed") {
     // Demanding an unreachable compaction ratio is the most aggressive 
setting: even a
     // low-cardinality input that the default threshold keeps aggregating is 
bypassed at the first
@@ -1361,17 +1582,21 @@ class AdaptivePartialAggregationSuite extends QueryTest 
with SharedSparkSession
     }
   }
 
-  test("minCompaction rejects values that cannot be generated as a Java 
literal") {
+  test("the compaction thresholds reject values that cannot be generated as a 
Java literal") {
     // The threshold is interpolated straight into the generated source, where 
a non-finite double
     // renders as `InfinityD` -- not a valid Java literal. Reject it at 
configuration time rather
     // than failing to compile the stage (or silently deoptimizing when 
codegen falls back).
-    Seq("Infinity", "-Infinity", "NaN", "1e309").foreach { v =>
+    for {
+      key <- Seq(SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION.key,
+        SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key)
+      v <- Seq("Infinity", "-Infinity", "NaN", "1e309")
+    } {
       val e = intercept[IllegalArgumentException] {
-        
spark.conf.set(SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION.key, v)
+        spark.conf.set(key, v)
       }
-      assert(e.getMessage.contains("finite"), s"expected a finiteness error 
for '$v': $e")
+      assert(e.getMessage.contains("finite"), s"expected a finiteness error 
for $key='$v': $e")
+      spark.conf.unset(key)
     }
-    spark.conf.unset(SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION.key)
   }
 
   test("a larger minRows defers the decision so a small high-cardinality input 
is not bypassed") {
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/AdaptivePartialAggregationBenchmark.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/AdaptivePartialAggregationBenchmark.scala
index 7c23d5e296bc..01121e6a28e4 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/AdaptivePartialAggregationBenchmark.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/AdaptivePartialAggregationBenchmark.scala
@@ -28,7 +28,7 @@ import org.apache.spark.sql.internal.SQLConf
  * remaining rows through as single-row partial buffers instead of maintaining 
(and possibly
  * spilling) a large aggregation map.
  *
- * Each scenario runs the query across the full matrix of whole-stage codegen 
on/off and the
+ * The first scenarios run the query across the full matrix of whole-stage 
codegen on/off and the
  * feature disabled (`adaptive = F`, the pre-change baseline) vs enabled 
(`adaptive = T`), over a
  * {high, low}-cardinality x {no-spill, on-spill} grid:
  *   - high-cardinality, no spill: the periodic check bypasses, which should 
win.
@@ -39,6 +39,11 @@ import org.apache.spark.sql.internal.SQLConf
  *   - low-cardinality, forced regular-map spill: the compaction ratio is far 
above the threshold,
  *     so the spill check keeps aggregating and both runs spill identically 
(no regression).
  *
+ * The spill-check scenarios read four partitions, so an `Exchange` separates 
the Partial from the
+ * Final, force the map to fill every few hundred thousand rows, and compare 
the feature disabled
+ * with `minCompactionToSpill` at 1.05 and at its default, 2.0, over inputs 
whose keys arrive
+ * together, recur across the task, or concentrate on a few hot keys.
+ *
  * To run this benchmark:
  * {{{
  *   1. build/sbt "sql/Test/runMain
@@ -135,6 +140,67 @@ object AdaptivePartialAggregationBenchmark extends 
SqlBasedBenchmark {
         extraConf = spillCheckConf)
       benchmark.run()
     }
+
+    // Inputs that compact by about 1.5 within each map epoch, read from four 
partitions so an
+    // `Exchange` separates the Partial from the Final, and with a map forced 
to fill every few
+    // hundred thousand rows. Whether keeping the aggregation through the 
spills pays off depends
+    // on whether the rest of the input keeps repeating keys:
+    //   - each key's rows arrive together, so later epochs share no keys and 
passing the rest
+    //     through at the first fill avoids spilling every epoch;
+    //   - keys recur uniformly across the task (10 rows per key), so the 
final merge collapses
+    //     them and keeping the aggregation shuffles a tenth of the rows;
+    //   - the same over twice the rows and keys, so the maps fill when they 
hold 40% of a task's
+    //     keys: the recent rows compact by about 1.5 and the default passes 
the rest through,
+    //     although each key repeats about 10 times. It runs with the fast map 
off, and with it on,
+    //     where the fast map's keys never spill. The interpreted path has no 
fast map, so the
+    //     latter runs with codegen only;
+    //   - 10 hot keys hold 30% of the rows and the rest are unique, so 
passing through avoids the
+    //     spills but sends every hot-key row to the reducer of its key.
+    // `minCompactionToSpill = 1.05` judges the spill check by 
`minCompaction`; the default, 2.0,
+    // demands that the recent rows compact by 2.0.
+    case class SpillCheckCase(name: String, key: String, fillRows: Int,
+        numRows: Long = 8L << 20, twoLevelMap: Boolean = true, codegenOnly: 
Boolean = false)
+    val spillCheckCases = Seq(
+      SpillCheckCase("keys arrive together", "cast(id * 2 / 3 as bigint)", 
300000),
+      SpillCheckCase("keys recur", "pmod(xxhash64(id), 200000)", 220000),
+      SpillCheckCase("keys recur, map holds 40% of them, fast map off",
+        "pmod(xxhash64(id), 400000)", 204330, numRows = 16L << 20, twoLevelMap 
= false),
+      SpillCheckCase("keys recur, maps hold 40% of them, fast map on",
+        "pmod(xxhash64(id), 400000)", 204330, numRows = 16L << 20, codegenOnly 
= true),
+      SpillCheckCase("hot keys", "if(id % 100 < 30, id % 10, id + 1000)", 
300000))
+    spillCheckCases.foreach { c =>
+      runBenchmark(s"spill check with an Exchange, ${c.name}") {
+        val N = c.numRows
+        val benchmark = new Benchmark(s"adaptive partial agg, spill check, 
${c.name}", N,
+          output = output)
+        val query = () => spark.range(0, N, 1, 4).selectExpr(s"${c.key} as k", 
"id as v")
+          .groupBy("k").agg("v" -> "sum")
+        val spillConf = Seq(
+          SQLConf.SHUFFLE_PARTITIONS.key -> "8",
+          SQLConf.ENABLE_TWOLEVEL_AGG_MAP.key -> c.twoLevelMap.toString,
+          "spark.sql.TungstenAggregate.testFallbackStartsAt" -> s"65536, 
${c.fillRows}")
+        for {
+          wholeStage <- if (c.codegenOnly) Seq(true) else Seq(true, false)
+          minCompactionToSpill <- Seq(None, Some("1.05"), Some("2.0"))
+        } {
+          val label = minCompactionToSpill.map(v => s"minCompactionToSpill = 
$v")
+            .getOrElse("adaptive = F")
+          benchmark.addCase(s"codegen = $wholeStage, $label") { _ =>
+            withSQLConf(
+              (Seq(
+                SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> wholeStage.toString,
+                SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key ->
+                  minCompactionToSpill.isDefined.toString) ++
+                minCompactionToSpill.map(
+                  
SQLConf.ADAPTIVE_PARTIAL_AGGREGATION_MIN_COMPACTION_TO_SPILL.key -> _) ++
+                fixedPlanConfs ++ spillConf): _*) {
+              query().noop()
+            }
+          }
+        }
+        benchmark.run()
+      }
+    }
   }
 
   private def distinctKeyedDf(N: Long): DataFrame =


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to