ulysses-you opened a new pull request, #57602: URL: https://github.com/apache/spark/pull/57602
### What changes were proposed in this pull request? This PR adds a new session-level config `spark.sql.execution.bypassPartialWindowGroupLimit` (default `false`). When enabled, the `WindowGroupLimit` planner strategy emits only the final `WindowGroupLimitExec` after the shuffle and skips constructing the pre-shuffle partial `WindowGroupLimitExec`. The final node still declares its required child distribution (`ClusteredDistribution(partitionSpec)`, or `AllTuples` when there is no partition spec), so `EnsureRequirements` inserts the shuffle regardless; dropping the partial node simply removes the pre-shuffle local top-k filter. This mirrors the shape of the existing partial/final split for window group limit and is analogous to bypassing partial aggregation. ### Why are the changes needed? The pre-shuffle partial `WindowGroupLimitExec` only pays off when it actually reduces rows. When each pre-shuffle partition already has few rows per window-partition group (a low reduction ratio), the partial node adds sorting/iteration cost with little benefit. This config lets users bypass the partial phase for such workloads. ### Does this PR introduce _any_ user-facing change? Yes. A new config `spark.sql.execution.bypassPartialWindowGroupLimit` is added, defaulting to `false`, which preserves the existing behavior. It is also documented in the SQL performance tuning guide. ### How was this patch tested? Added a unit test in `DataFrameWindowFunctionsSuite` that asserts the executed plan contains a single `WindowGroupLimitExec` when the config is on and two when it is off, and that results are unchanged between the two settings. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
