ulysses-you commented on PR #59863: URL: https://github.com/apache/spark/pull/59863#issuecomment-6569130193
Thanks @peter-toth and @dongjoon-hyun for the reviews. We initially considered changing `ConvertSortMergeJoinToShuffledHashJoin` to look through an identity `GroupPartitionsExec`. We chose to remove the redundant node instead: besides blocking SHJ conversion, it retains grouping and partition-key information in the driver plan and can be misleading in `EXPLAIN` when it performs no regrouping. The node is constructed before the skip is decided, so this reduces retention in the resulting plan, not the cost of constructing it. To preserve the scan-rebuild guard, the skip is now restricted to alignments whose child has only `ShuffleQueryStageExec` leaves and whose output partitioning equals the child\x27s. A child with any non-stage leaf keeps its node, so a scan rebuilt with different partitioning is checked against the planned layout. The node is also retained when a replicate side is chosen, including when the join type declines that choice. The tests cover the stage-leaf and non-identity cases, a controlled scan rebuild, and replicate-side selection across an `EnsureRequirements` re-run. The existing tests that required scan-side alignment nodes have been restored. The four affected suites pass (509 tests), as do both scalastyle checks. The changes are in 5346f9f5880. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
