ulysses-you opened a new pull request, #57986:
URL: https://github.com/apache/spark/pull/57986
### What changes were proposed in this pull request?
Currently, `CollapseWindow` collapses two adjacent `Window` operators only
when their partition specs and order specs are identical. This PR relaxes it to
also merge two windows with the same partition spec when **one of them has an
empty order spec**, as long as every window expression of the empty-order
window is order-insensitive.
A window expression is treated as order-insensitive when its frame is the
whole partition (`ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING`):
such a frame always covers every row of the partition regardless of the
ordering, so aggregates like `count`/`sum`/`min`/`max` give the same value
under any ordering, and functions whose result does depend on the row order
(e.g. `collect_list`, `first`) are non-deterministic when the order spec is
empty, so evaluating them under any ordering yields a valid result. Windows
with a bounded frame (e.g. `ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`)
are order-sensitive and are never merged.
The merged window keeps the non-empty order spec of the other window.
### Why are the changes needed?
For queries that mix an ordered window function with an unordered aggregate
over the same partition, e.g.:
```sql
SELECT c2, c1,
row_number() OVER (PARTITION BY c1 ORDER BY c2) AS rk,
count(1) OVER (PARTITION BY c1)
FROM t3
```
the current rule keeps two `Window` operators even though they share
`PARTITION BY c1`. After this change they are collapsed into a single window
operator, saving one `WindowExec` pass (the `[c1, c2]` sort is already shared
in both plans). A local benchmark on 4M rows showed roughly 16% faster runtime
in the non-spill case and 20% in the spill case.
### Does this PR introduce _any_ user-facing change?
The query result is unchanged for the common shape (the empty-order window
written after the ordered one), where the merge does not change the input order
of any window expression. When the empty-order window appears before an ordered
sibling in the SELECT list, the merge evaluates its expressions under the
sibling's order; for functions documented as non-deterministic without an order
(`first`, `last`, `collect_list`) the value may differ, which is already
allowed by their contract. FP `sum`/`avg` may also differ at the bit level,
consistent with Spark's existing treatment of FP aggregation as order-sensitive
(`EliminateSorts.isOrderIrrelevantAggs`).
### How was this patch tested?
Added tests to `CollapseWindowSuite` covering:
- collapse when the empty-order window has a whole-partition frame
(`row_number` + `count`), including the case where it is the inner window;
- collapse when the empty-order window has multiple window expressions;
- collapse when the empty-order window has `first` over the whole partition;
- the SPARK-34565 shape with a `Project` between the windows;
- no collapse when the empty-order window has a bounded frame.
Ran `CollapseWindowSuite` (13 tests) and `TransposeWindowSuite` (8 tests),
all pass.
### Was this patch authored or co-authored using generative AI tooling?
Yes, developed with assistance from Claude Code.
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
