ulysses-you opened a new pull request, #57221: URL: https://github.com/apache/spark/pull/57221
### What changes were proposed in this pull request? This PR enhances `RemoveRedundantSorts` to remove a *dangling* local sort -- a local `SortExec` sitting directly below a shuffle that neither consumes nor exposes an ordering (both its `requiredChildOrdering` and `outputOrdering` are empty). Such a shuffle destroys the child ordering, so the sort has no effect and is dead. To make this effective for the skew-join case, `RemoveRedundantSorts` is moved after `OptimizeSkewedJoin` in the AQE `queryStagePreparationRules` (`AdaptiveSparkPlanExec`), and the same reorder is applied to the non-AQE `QueryExecution.preparations` to keep the two rule chains consistent. ### Why are the changes needed? Consider a stage with a `ShuffledHashJoin` feeding a `SortMergeJoin` on the same key, where the SHJ output partitioning already satisfies the SMJ requirement (no exchange in between, only the SMJ's local sort). When the SHJ is skewed and skew-join optimization is applied, an extra shuffle is inserted between the two joins. `EnsureRequirements` wraps that shuffle on top of the existing `Sort -> SHJ`, and adds a new sort above the shuffle for the SMJ. The original local sort is then left dangling right below the newly inserted shuffle, computed in the SHJ's stage for nothing. The current `RemoveRedundantSorts` cannot remove it: it only strips a sort whose child already satisfies the ordering, whereas this sort is dead because it sits directly under an ordering-destroying shuffle -- a different kind of redundancy. Running before `OptimizeSkewedJoin` also means it never sees the added shuffle. ### Does this PR introduce _any_ user-facing change? No. It only removes a redundant sort operator from the physical plan. ### How was this patch tested? New UT `SPARK-58099` in `AdaptiveQueryExecSuite` reproduces the SHJ->SMJ skew-join scenario: with the rule disabled it asserts a local sort dangles below the added shuffle, and with the rule enabled it asserts the sort is removed while both joins remain skew joins and results are unchanged. Also verified `RemoveRedundantSortsSuite` (both AE / non-AE variants) and the full `AdaptiveQueryExecSuite` pass. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
