This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.3 by this push:
     new 185f0da829c0 [SPARK-58404][SQL] Support bypassing partial 
WindowGroupLimit
185f0da829c0 is described below

commit 185f0da829c063d68efc424e0f79dbc3dae9a114
Author: Xiduo You <[email protected]>
AuthorDate: Mon Aug 3 22:09:35 2026 -0700

    [SPARK-58404][SQL] Support bypassing partial WindowGroupLimit
    
    ### What changes were proposed in this pull request?
    
    This PR adds a new config 
`spark.sql.execution.bypassPartialWindowGroupLimit` (default `false`). When 
enabled, the `WindowGroupLimit` planner strategy emits only the final 
`WindowGroupLimitExec` after the shuffle and skips constructing the pre-shuffle 
partial `WindowGroupLimitExec`, provided the window has a non-empty partition 
spec.
    
    The final node still declares its required child distribution 
(`ClusteredDistribution(partitionSpec)`), so `EnsureRequirements` inserts the 
shuffle regardless; dropping the partial node removes both the pre-shuffle 
local top-k filter and the local `SortExec` that the partial node's 
`requiredChildOrdering` forces via `EnsureRequirements`. Skipping that local 
sort is a good part of the win. This mirrors the shape of the existing 
partial/final split for window group limit and is analogou [...]
    
    The bypass is gated on a non-empty partition spec. With an empty partition 
spec the final node requires `AllTuples`, so the shuffle funnels the whole 
input into a single reducer, while the partial pass we would drop bounds each 
input partition to `limit` rank groups before that shuffle. Bypassing it there 
would shuffle all raw rows to a single partition and is strictly worse than the 
normal Partial+Final path, so the partial is always kept. This mirrors the 
grouping-keys carve-out in  [...]
    
    ### Why are the changes needed?
    
    The pre-shuffle partial `WindowGroupLimitExec` only pays off when it 
actually reduces rows. When each pre-shuffle partition already has few rows per 
window-partition group (a low reduction ratio), the partial node adds 
sorting/iteration cost with little benefit. This config lets users bypass the 
partial phase for such workloads.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. A new config `spark.sql.execution.bypassPartialWindowGroupLimit` is 
added, defaulting to `false`, which preserves the existing behavior. It is also 
documented in the SQL performance tuning guide.
    
    ### How was this patch tested?
    
    Added a unit test in `DataFrameWindowFunctionsSuite` covering 
`row_number`/`rank`/`dense_rank`. For a partitioned window it asserts the 
executed plan contains a single `WindowGroupLimitExec` when the config is on 
and two when it is off; for an unpartitioned window it asserts both nodes 
remain regardless of the config, since the bypass is gated off there. Results 
are unchanged across all settings.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Opus 4.8)
    
    Closes #57602 from ulysses-you/bypass-partial-window-group-limit.
    
    Authored-by: Xiduo You <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 91c03e36f0f4135759bfe6a823e911f5446b34df)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 docs/sql-performance-tuning.md                     | 25 +++++++++
 .../org/apache/spark/sql/internal/SQLConf.scala    | 16 ++++++
 .../spark/sql/execution/SparkStrategies.scala      | 21 ++++++-
 .../spark/sql/DataFrameWindowFunctionsSuite.scala  | 64 +++++++++++++++++++++-
 4 files changed, 122 insertions(+), 4 deletions(-)

diff --git a/docs/sql-performance-tuning.md b/docs/sql-performance-tuning.md
index 3ba3a6c749ca..6acb36413d93 100644
--- a/docs/sql-performance-tuning.md
+++ b/docs/sql-performance-tuning.md
@@ -168,6 +168,31 @@ SELECT /*+ REBALANCE_BY_SIZE('128m', c) */ * FROM t;
 
 For more details please refer to the documentation of [Partitioning 
Hints](sql-ref-syntax-qry-select-hints.html#partitioning-hints).
 
+## Tuning Window Functions
+
+Top-k window queries (such as filtering on <code>row_number</code>, 
<code>rank</code> or
+<code>dense_rank</code>) plan a window group limit on both sides of the 
shuffle: a partial one
+before the shuffle to reduce the data feeding it, and a final one after. The 
following configuration
+tunes that behavior.
+
+<table class="spark-config">
+  <thead><tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since 
Version</th></tr></thead>
+  <tr>
+    <td><code>spark.sql.execution.bypassPartialWindowGroupLimit</code></td>
+    <td>false</td>
+    <td>
+      When true, skips the pre-shuffle partial window group limit for 
partitioned top-k window
+      queries and runs only a single window group limit after the shuffle. 
Bypassing the partial
+      window group limit can improve performance when the pre-shuffle 
reduction ratio is low. The
+      bypass only applies to windows with a non-empty partition spec; for an 
unpartitioned window
+      the partial pass is always kept, since for a multi-partition input the 
shuffle funnels the
+      whole input into a single reducer. When false, a partial window group 
limit runs before the
+      shuffle and a final one runs after it.
+    </td>
+    <td>4.3.0</td>
+  </tr>
+</table>
+
 ## Leveraging Statistics
 Apache Spark's ability to choose the best execution plan among many possible 
options is determined in part by its estimates of how many rows will be output 
by every node in the execution plan (read, filter, join, etc.). Those estimates 
in turn are based on statistics that are made available to Spark in one of 
several ways:
 
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 403de67e56e7..64d732172960 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -4689,6 +4689,20 @@ object SQLConf {
         "The threshold of window group limit must be -1, 0 or positive 
integer.")
       .createWithDefault(1000)
 
+  val BYPASS_PARTIAL_WINDOW_GROUP_LIMIT =
+    buildConf("spark.sql.execution.bypassPartialWindowGroupLimit")
+      .doc("When true, skips the pre-shuffle partial WindowGroupLimit and runs 
only a single " +
+        "WindowGroupLimit after the shuffle. Bypassing the partial window 
group limit can " +
+        "improve performance when the pre-shuffle reduction ratio is low. This 
only applies to " +
+        "windows with a non-empty partition spec; an unpartitioned window 
always keeps the " +
+        "partial pass, since for a multi-partition input the shuffle funnels 
the whole input " +
+        "into a single reducer. When false (default), a partial 
WindowGroupLimit runs before " +
+        "the shuffle and a final one runs after it.")
+      .version("4.3.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+      .booleanConf
+      .createWithDefault(false)
+
   val WINDOW_SEGMENT_TREE_ENABLED =
     buildConf("spark.sql.window.segmentTree.enabled")
       .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
@@ -9109,6 +9123,8 @@ class SQLConf extends Serializable with Logging with 
SqlApiConf {
 
   def windowGroupLimitThreshold: Int = getConf(WINDOW_GROUP_LIMIT_THRESHOLD)
 
+  def bypassPartialWindowGroupLimit: Boolean = 
getConf(BYPASS_PARTIAL_WINDOW_GROUP_LIMIT)
+
   def windowSegmentTreeEnabled: Boolean = getConf(WINDOW_SEGMENT_TREE_ENABLED)
 
   def windowSegmentTreeMinPartitionRows: Int = 
getConf(WINDOW_SEGMENT_TREE_MIN_PARTITION_ROWS)
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
index 996ecac61b72..1d0917266ea8 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
@@ -802,10 +802,25 @@ abstract class SparkStrategies extends 
QueryPlanner[SparkPlan] {
   object WindowGroupLimit extends Strategy {
     def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
       case logical.WindowGroupLimit(partitionSpec, orderSpec, 
rankLikeFunction, limit, child) =>
-        val partialWindowGroupLimit = 
execution.window.WindowGroupLimitExec(partitionSpec,
-          orderSpec, rankLikeFunction, limit, execution.window.Partial, 
planLater(child))
+        // When the partial window group limit is bypassed, skip the 
pre-shuffle partial
+        // WindowGroupLimit and run only a single WindowGroupLimit after the 
shuffle. This can
+        // improve performance when the pre-shuffle reduction ratio is low.
+        //
+        // The bypass is only gated on a non-empty partitionSpec. With an 
empty partitionSpec the
+        // final WindowGroupLimit requires AllTuples, so for a multi-partition 
input the shuffle
+        // funnels everything into a single reducer; the partial pass we would 
drop bounds each
+        // input partition to `limit` rank groups before that shuffle, so 
keeping it is a sensible
+        // default (an already-single-partition child needs no shuffle, so the 
partial cannot reduce
+        // the shuffle input there and only adds another pass). This mirrors 
the grouping-keys
+        // carve-out in bypassPartialAggregation (see 
AggUtils.planAggregateWithoutDistinct).
+        val finalChild = if (conf.bypassPartialWindowGroupLimit && 
partitionSpec.nonEmpty) {
+          planLater(child)
+        } else {
+          execution.window.WindowGroupLimitExec(partitionSpec,
+            orderSpec, rankLikeFunction, limit, execution.window.Partial, 
planLater(child))
+        }
         val finalWindowGroupLimit = 
execution.window.WindowGroupLimitExec(partitionSpec, orderSpec,
-          rankLikeFunction, limit, execution.window.Final, 
partialWindowGroupLimit)
+          rankLikeFunction, limit, execution.window.Final, finalChild)
         finalWindowGroupLimit :: Nil
       case _ => Nil
     }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFunctionsSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFunctionsSuite.scala
index c1d1f13915a6..cf73633e74ed 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFunctionsSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFunctionsSuite.scala
@@ -27,7 +27,7 @@ import 
org.apache.spark.sql.catalyst.plans.physical.HashPartitioning
 import org.apache.spark.sql.catalyst.trees.UnaryLike
 import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
 import org.apache.spark.sql.execution.exchange.{ENSURE_REQUIREMENTS, Exchange, 
ShuffleExchangeExec}
-import org.apache.spark.sql.execution.window.WindowExec
+import org.apache.spark.sql.execution.window.{WindowExec, WindowGroupLimitExec}
 import org.apache.spark.sql.expressions.{Aggregator, MutableAggregationBuffer, 
UserDefinedAggregateFunction, Window}
 import org.apache.spark.sql.functions._
 import org.apache.spark.sql.internal.SQLConf
@@ -1682,6 +1682,68 @@ class DataFrameWindowFunctionsSuite extends 
SharedSparkSession
     }
   }
 
+  test("SPARK-58404: bypass partial WindowGroupLimit") {
+    val df = Seq(
+      ("a", 0, "c"),
+      ("a", 1, "x"),
+      ("a", 2, "y"),
+      ("b", 1, "h"),
+      ("b", 1, "n"),
+      ("c", 1, "z"),
+      ("c", 2, "a")).toDF("key", "value", "order")
+
+    def countWindowGroupLimits(df: DataFrame): Int = {
+      collect(df.queryExecution.executedPlan) {
+        case w: WindowGroupLimitExec => w
+      }.size
+    }
+
+    val rankLikeFunctions = Seq(row_number(), rank(), dense_rank())
+
+    // Partitioned window: a shuffle is required, so without the bypass both 
the partial and the
+    // final WindowGroupLimit are present; with the bypass only the final one 
remains.
+    val window = Window.partitionBy($"key").orderBy($"order")
+    val expected = Seq(
+      Row("a", 0, "c", 1),
+      Row("b", 1, "h", 1),
+      Row("c", 2, "a", 1))
+
+    Seq(true, false).foreach { bypass =>
+      withSQLConf(
+        SQLConf.BYPASS_PARTIAL_WINDOW_GROUP_LIMIT.key -> bypass.toString,
+        SQLConf.WINDOW_GROUP_LIMIT_THRESHOLD.key -> "100") {
+        rankLikeFunctions.foreach { rankLikeFunction =>
+          val result = df.withColumn("rn", 
rankLikeFunction.over(window)).where($"rn" === 1)
+          checkAnswer(result, expected)
+          assert(countWindowGroupLimits(result) === (if (bypass) 1 else 2))
+        }
+      }
+    }
+
+    // Unpartitioned window: the final WindowGroupLimit requires AllTuples, so 
the shuffle funnels
+    // the whole input into a single reducer. The partial pass bounds each 
input partition to
+    // `limit` rank groups before that shuffle, so the bypass is gated off for 
an empty partition
+    // spec and both the partial and the final WindowGroupLimit remain even 
when the flag is on.
+    // Note row_number() is excluded: with an empty partition spec and a limit 
below
+    // topKSortFallbackThreshold, InferWindowGroupLimit rewrites it to a Limit 
+ Sort (top-n)
+    // rather than a WindowGroupLimit, so no WindowGroupLimit node is produced.
+    val unpartitionedWindow = Window.orderBy($"order")
+    val unpartitionedExpected = Seq(Row("c", 2, "a", 1))
+
+    Seq(true, false).foreach { bypass =>
+      withSQLConf(
+        SQLConf.BYPASS_PARTIAL_WINDOW_GROUP_LIMIT.key -> bypass.toString,
+        SQLConf.WINDOW_GROUP_LIMIT_THRESHOLD.key -> "100") {
+        Seq(rank(), dense_rank()).foreach { rankLikeFunction =>
+          val result =
+            df.withColumn("rn", 
rankLikeFunction.over(unpartitionedWindow)).where($"rn" === 1)
+          checkAnswer(result, unpartitionedExpected)
+          assert(countWindowGroupLimits(result) === 2)
+        }
+      }
+    }
+  }
+
   test("SPARK-42525: collapse two adjacent windows with the same 
partition/order in subquery") {
     withTempView("t1") {
       Seq((1, 1), (2, 2)).toDF("a", "b").createOrReplaceTempView("t1")


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to