ulysses-you opened a new pull request, #57603:
URL: https://github.com/apache/spark/pull/57603
### What changes were proposed in this pull request?
This PR adds a new logical optimizer rule `PullUpProjectAliasThroughWindow`.
When a window function's partition/order key is an alias of an underlying
column, the rename lives in a `Project` *below* the window, while the
analyzer-inserted `Project` *above* the window references the renamed column
as a bare `Attribute` (empty alias map). Since `Window.outputPartitioning`
/`outputOrdering` are pure pass-throughs of the child, the upper project
cannot translate the child's `HashPartitioning(k)`/`SortOrder(k)` into the
renamed attribute a downstream operator requires.
The new rule pulls such aliases up from the bottom `Project` into the top
`Project`, across a chain of one or more adjacent `Window` operators
(`Project - Window+ - Project`), reusing the same exprId. After the rewrite,
the top project's alias map maps `k -> a`, so the existing
`PartitioningPreservingUnaryExecNode`/`OrderPreservingUnaryExecNode`
machinery that `ProjectExec` mixes in projects the window's partitioning and
ordering up through the alias. The `Window` operators themselves are left
untouched.
Example:
```sql
SELECT userid, count(*), sum(size(vset))
FROM (
SELECT key AS userid, collect_set(value) OVER (PARTITION BY key) AS vset
FROM testData
) u
GROUP BY 1
```
Before: the window shuffles on `key`, then a second, redundant shuffle on
`userid` is inserted for the aggregate. After: a single shuffle.
An entry of the bottom project is pulled up only when: it is an `Alias`; no
window in the chain references it; all of its input attributes remain
produced by the pruned bottom project; and the top project consumes its
output solely as a bare pass-through attribute.
Besides removing the redundant shuffle/sort, this also narrows the data
crossing the window's shuffle: the alias no longer flows through the window
as a separate column and is recomputed cheaply above the exchange.
### Why are the changes needed?
To avoid a redundant shuffle (and possibly a redundant sort) whenever a
downstream consumer (aggregate, repartition, chained window, ...) requires
distribution/ordering on an alias of a window partition/order key. The window
has already shuffled/sorted on the underlying key; the second exchange is
pure overhead.
### Does this PR introduce _any_ user-facing change?
No. It is a plan optimization; results are unchanged.
### How was this patch tested?
New unit tests:
- `PullUpProjectAliasThroughWindowSuite` (catalyst): 14 tests covering
pull-up
of partition/order key renames, multi-key windows, computed aliases whose
inputs are all window keys, window chains, and the negative/idempotency/
schema-preservation cases.
- `ProjectedOrderingAndPartitioningSuite` (sql/core): 4 end-to-end tests
asserting a single shuffle (and single sort where applicable) plus
`checkAnswer` correctness, for aggregate/repartition/stacked-window/
window-chain consumers.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
