This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 91c03e36f0f4 [SPARK-58404][SQL] Support bypassing partial
WindowGroupLimit
91c03e36f0f4 is described below
commit 91c03e36f0f4135759bfe6a823e911f5446b34df
Author: Xiduo You <[email protected]>
AuthorDate: Mon Aug 3 22:09:35 2026 -0700
[SPARK-58404][SQL] Support bypassing partial WindowGroupLimit
### What changes were proposed in this pull request?
This PR adds a new config
`spark.sql.execution.bypassPartialWindowGroupLimit` (default `false`). When
enabled, the `WindowGroupLimit` planner strategy emits only the final
`WindowGroupLimitExec` after the shuffle and skips constructing the pre-shuffle
partial `WindowGroupLimitExec`, provided the window has a non-empty partition
spec.
The final node still declares its required child distribution
(`ClusteredDistribution(partitionSpec)`), so `EnsureRequirements` inserts the
shuffle regardless; dropping the partial node removes both the pre-shuffle
local top-k filter and the local `SortExec` that the partial node's
`requiredChildOrdering` forces via `EnsureRequirements`. Skipping that local
sort is a good part of the win. This mirrors the shape of the existing
partial/final split for window group limit and is analogou [...]
The bypass is gated on a non-empty partition spec. With an empty partition
spec the final node requires `AllTuples`, so the shuffle funnels the whole
input into a single reducer, while the partial pass we would drop bounds each
input partition to `limit` rank groups before that shuffle. Bypassing it there
would shuffle all raw rows to a single partition and is strictly worse than the
normal Partial+Final path, so the partial is always kept. This mirrors the
grouping-keys carve-out in [...]
### Why are the changes needed?
The pre-shuffle partial `WindowGroupLimitExec` only pays off when it
actually reduces rows. When each pre-shuffle partition already has few rows per
window-partition group (a low reduction ratio), the partial node adds
sorting/iteration cost with little benefit. This config lets users bypass the
partial phase for such workloads.
### Does this PR introduce _any_ user-facing change?
Yes. A new config `spark.sql.execution.bypassPartialWindowGroupLimit` is
added, defaulting to `false`, which preserves the existing behavior. It is also
documented in the SQL performance tuning guide.
### How was this patch tested?
Added a unit test in `DataFrameWindowFunctionsSuite` covering
`row_number`/`rank`/`dense_rank`. For a partitioned window it asserts the
executed plan contains a single `WindowGroupLimitExec` when the config is on
and two when it is off; for an unpartitioned window it asserts both nodes
remain regardless of the config, since the bypass is gated off there. Results
are unchanged across all settings.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57602 from ulysses-you/bypass-partial-window-group-limit.
Authored-by: Xiduo You <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/sql-performance-tuning.md | 25 +++++++++
.../org/apache/spark/sql/internal/SQLConf.scala | 16 ++++++
.../spark/sql/execution/SparkStrategies.scala | 21 ++++++-
.../spark/sql/DataFrameWindowFunctionsSuite.scala | 64 +++++++++++++++++++++-
4 files changed, 122 insertions(+), 4 deletions(-)
diff --git a/docs/sql-performance-tuning.md b/docs/sql-performance-tuning.md
index 3ba3a6c749ca..6acb36413d93 100644
--- a/docs/sql-performance-tuning.md
+++ b/docs/sql-performance-tuning.md
@@ -168,6 +168,31 @@ SELECT /*+ REBALANCE_BY_SIZE('128m', c) */ * FROM t;
For more details please refer to the documentation of [Partitioning
Hints](sql-ref-syntax-qry-select-hints.html#partitioning-hints).
+## Tuning Window Functions
+
+Top-k window queries (such as filtering on <code>row_number</code>,
<code>rank</code> or
+<code>dense_rank</code>) plan a window group limit on both sides of the
shuffle: a partial one
+before the shuffle to reduce the data feeding it, and a final one after. The
following configuration
+tunes that behavior.
+
+<table class="spark-config">
+ <thead><tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since
Version</th></tr></thead>
+ <tr>
+ <td><code>spark.sql.execution.bypassPartialWindowGroupLimit</code></td>
+ <td>false</td>
+ <td>
+ When true, skips the pre-shuffle partial window group limit for
partitioned top-k window
+ queries and runs only a single window group limit after the shuffle.
Bypassing the partial
+ window group limit can improve performance when the pre-shuffle
reduction ratio is low. The
+ bypass only applies to windows with a non-empty partition spec; for an
unpartitioned window
+ the partial pass is always kept, since for a multi-partition input the
shuffle funnels the
+ whole input into a single reducer. When false, a partial window group
limit runs before the
+ shuffle and a final one runs after it.
+ </td>
+ <td>4.3.0</td>
+ </tr>
+</table>
+
## Leveraging Statistics
Apache Spark's ability to choose the best execution plan among many possible
options is determined in part by its estimates of how many rows will be output
by every node in the execution plan (read, filter, join, etc.). Those estimates
in turn are based on statistics that are made available to Spark in one of
several ways:
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 403de67e56e7..64d732172960 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -4689,6 +4689,20 @@ object SQLConf {
"The threshold of window group limit must be -1, 0 or positive
integer.")
.createWithDefault(1000)
+ val BYPASS_PARTIAL_WINDOW_GROUP_LIMIT =
+ buildConf("spark.sql.execution.bypassPartialWindowGroupLimit")
+ .doc("When true, skips the pre-shuffle partial WindowGroupLimit and runs
only a single " +
+ "WindowGroupLimit after the shuffle. Bypassing the partial window
group limit can " +
+ "improve performance when the pre-shuffle reduction ratio is low. This
only applies to " +
+ "windows with a non-empty partition spec; an unpartitioned window
always keeps the " +
+ "partial pass, since for a multi-partition input the shuffle funnels
the whole input " +
+ "into a single reducer. When false (default), a partial
WindowGroupLimit runs before " +
+ "the shuffle and a final one runs after it.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .booleanConf
+ .createWithDefault(false)
+
val WINDOW_SEGMENT_TREE_ENABLED =
buildConf("spark.sql.window.segmentTree.enabled")
.withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
@@ -9109,6 +9123,8 @@ class SQLConf extends Serializable with Logging with
SqlApiConf {
def windowGroupLimitThreshold: Int = getConf(WINDOW_GROUP_LIMIT_THRESHOLD)
+ def bypassPartialWindowGroupLimit: Boolean =
getConf(BYPASS_PARTIAL_WINDOW_GROUP_LIMIT)
+
def windowSegmentTreeEnabled: Boolean = getConf(WINDOW_SEGMENT_TREE_ENABLED)
def windowSegmentTreeMinPartitionRows: Int =
getConf(WINDOW_SEGMENT_TREE_MIN_PARTITION_ROWS)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
index 996ecac61b72..1d0917266ea8 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
@@ -802,10 +802,25 @@ abstract class SparkStrategies extends
QueryPlanner[SparkPlan] {
object WindowGroupLimit extends Strategy {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
case logical.WindowGroupLimit(partitionSpec, orderSpec,
rankLikeFunction, limit, child) =>
- val partialWindowGroupLimit =
execution.window.WindowGroupLimitExec(partitionSpec,
- orderSpec, rankLikeFunction, limit, execution.window.Partial,
planLater(child))
+ // When the partial window group limit is bypassed, skip the
pre-shuffle partial
+ // WindowGroupLimit and run only a single WindowGroupLimit after the
shuffle. This can
+ // improve performance when the pre-shuffle reduction ratio is low.
+ //
+ // The bypass is only gated on a non-empty partitionSpec. With an
empty partitionSpec the
+ // final WindowGroupLimit requires AllTuples, so for a multi-partition
input the shuffle
+ // funnels everything into a single reducer; the partial pass we would
drop bounds each
+ // input partition to `limit` rank groups before that shuffle, so
keeping it is a sensible
+ // default (an already-single-partition child needs no shuffle, so the
partial cannot reduce
+ // the shuffle input there and only adds another pass). This mirrors
the grouping-keys
+ // carve-out in bypassPartialAggregation (see
AggUtils.planAggregateWithoutDistinct).
+ val finalChild = if (conf.bypassPartialWindowGroupLimit &&
partitionSpec.nonEmpty) {
+ planLater(child)
+ } else {
+ execution.window.WindowGroupLimitExec(partitionSpec,
+ orderSpec, rankLikeFunction, limit, execution.window.Partial,
planLater(child))
+ }
val finalWindowGroupLimit =
execution.window.WindowGroupLimitExec(partitionSpec, orderSpec,
- rankLikeFunction, limit, execution.window.Final,
partialWindowGroupLimit)
+ rankLikeFunction, limit, execution.window.Final, finalChild)
finalWindowGroupLimit :: Nil
case _ => Nil
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFunctionsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFunctionsSuite.scala
index c1d1f13915a6..cf73633e74ed 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFunctionsSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFunctionsSuite.scala
@@ -27,7 +27,7 @@ import
org.apache.spark.sql.catalyst.plans.physical.HashPartitioning
import org.apache.spark.sql.catalyst.trees.UnaryLike
import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
import org.apache.spark.sql.execution.exchange.{ENSURE_REQUIREMENTS, Exchange,
ShuffleExchangeExec}
-import org.apache.spark.sql.execution.window.WindowExec
+import org.apache.spark.sql.execution.window.{WindowExec, WindowGroupLimitExec}
import org.apache.spark.sql.expressions.{Aggregator, MutableAggregationBuffer,
UserDefinedAggregateFunction, Window}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.SQLConf
@@ -1682,6 +1682,68 @@ class DataFrameWindowFunctionsSuite extends
SharedSparkSession
}
}
+ test("SPARK-58404: bypass partial WindowGroupLimit") {
+ val df = Seq(
+ ("a", 0, "c"),
+ ("a", 1, "x"),
+ ("a", 2, "y"),
+ ("b", 1, "h"),
+ ("b", 1, "n"),
+ ("c", 1, "z"),
+ ("c", 2, "a")).toDF("key", "value", "order")
+
+ def countWindowGroupLimits(df: DataFrame): Int = {
+ collect(df.queryExecution.executedPlan) {
+ case w: WindowGroupLimitExec => w
+ }.size
+ }
+
+ val rankLikeFunctions = Seq(row_number(), rank(), dense_rank())
+
+ // Partitioned window: a shuffle is required, so without the bypass both
the partial and the
+ // final WindowGroupLimit are present; with the bypass only the final one
remains.
+ val window = Window.partitionBy($"key").orderBy($"order")
+ val expected = Seq(
+ Row("a", 0, "c", 1),
+ Row("b", 1, "h", 1),
+ Row("c", 2, "a", 1))
+
+ Seq(true, false).foreach { bypass =>
+ withSQLConf(
+ SQLConf.BYPASS_PARTIAL_WINDOW_GROUP_LIMIT.key -> bypass.toString,
+ SQLConf.WINDOW_GROUP_LIMIT_THRESHOLD.key -> "100") {
+ rankLikeFunctions.foreach { rankLikeFunction =>
+ val result = df.withColumn("rn",
rankLikeFunction.over(window)).where($"rn" === 1)
+ checkAnswer(result, expected)
+ assert(countWindowGroupLimits(result) === (if (bypass) 1 else 2))
+ }
+ }
+ }
+
+ // Unpartitioned window: the final WindowGroupLimit requires AllTuples, so
the shuffle funnels
+ // the whole input into a single reducer. The partial pass bounds each
input partition to
+ // `limit` rank groups before that shuffle, so the bypass is gated off for
an empty partition
+ // spec and both the partial and the final WindowGroupLimit remain even
when the flag is on.
+ // Note row_number() is excluded: with an empty partition spec and a limit
below
+ // topKSortFallbackThreshold, InferWindowGroupLimit rewrites it to a Limit
+ Sort (top-n)
+ // rather than a WindowGroupLimit, so no WindowGroupLimit node is produced.
+ val unpartitionedWindow = Window.orderBy($"order")
+ val unpartitionedExpected = Seq(Row("c", 2, "a", 1))
+
+ Seq(true, false).foreach { bypass =>
+ withSQLConf(
+ SQLConf.BYPASS_PARTIAL_WINDOW_GROUP_LIMIT.key -> bypass.toString,
+ SQLConf.WINDOW_GROUP_LIMIT_THRESHOLD.key -> "100") {
+ Seq(rank(), dense_rank()).foreach { rankLikeFunction =>
+ val result =
+ df.withColumn("rn",
rankLikeFunction.over(unpartitionedWindow)).where($"rn" === 1)
+ checkAnswer(result, unpartitionedExpected)
+ assert(countWindowGroupLimits(result) === 2)
+ }
+ }
+ }
+ }
+
test("SPARK-42525: collapse two adjacent windows with the same
partition/order in subquery") {
withTempView("t1") {
Seq((1, 1), (2, 2)).toDF("a", "b").createOrReplaceTempView("t1")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]