cloud-fan commented on PR #58144:
URL: https://github.com/apache/spark/pull/58144#issuecomment-6197684443
Thanks for the update. I rechecked the current head. The pin-ridable last
group, the refsAbove two-link test, the bounded fallback for rejected stacks,
per-group ordering ranks, and the Linux JDK 17/21/25 benchmark results are in
place. Three of the items still miss their goal:
1. Now that derived links cut chains, a Window over a derived Project over a
Window splits into one-window runs that never reach the pairwise fallback.
Flag-on therefore skips the Window-Project-Window swap that flag-off applies.
For example, `SELECT *, sum(v) OVER (PARTITION BY k1) FROM (SELECT *, sum(v)
OVER (PARTITION BY k1, k2) * 2 AS d FROM t)` keeps two exchanges instead of
one, which contradicts the 'never worse than leaving it off' wording.
2. The pin merge check probes only the window directly under the pin.
CollapseWindow merges bottom-up, so a same-spec, same-order sum that the chosen
order puts right below the safe tail (e.g. S [k1,k2], X [k3], R = row_number
[k1,k2], Y [k4] under rn [k1,k2]) merges into the tail and then into the pin,
and WindowGroupLimit is lost.
3. The pin's predicate normalization covers ConstantFolding and UnwrapCast
but not OptimizeIn (`rn IN (1)`) or BooleanSimplification (`NOT (rn > 1)`). A
missed pin buries the rank window and later iterations cannot recover it.
Pinning conservatively whenever the Filter references the top window's rank
output may be more robust than emulating each later rule.
<!-- SPARK_DEV_REVIEW_REPLY
{"feedback_id":"conversation:6172792091","thread_id":"conversation:6172792091","verdict_sha256":"cc60598f2a006ac90ad75d09ef9bd83fc49755f003c445115793deb88057a394"}
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
