peter-toth commented on PR #58814: URL: https://github.com/apache/spark/pull/58814#issuecomment-5681657748
@ulysses-you thank you for the approve. One thing you should see before this merges: it landed at `10dd169b413`, and I pushed a commit after it. A second review round found the same hazard on the other site that inserts a grouping node after deciding a child needs no shuffle, the one-side-shuffle `withJoinKeyPositions` call. Measured with the pairing gate already in place: a marked `[3, 4, 1, 2]` against a non-keyed side leaves a `GroupPartitions` reporting `UnknownPartitioning(4)` and a plan `ValidateRequirements` rejects. A grouped but unsorted marked layout is what a one-side shuffle onto a union's key order leaves behind, so it is the shape your own `SPARK-59050` test builds. I fixed it above both sites, in `KeyedPartitioning.createShuffleSpec`, which now never projects a marked layout rather than refusing only a narrowing projection. The storage-partitioned join survives it: the other child is laid out on the marked side's own key order instead of a sorted one. The description and the sweep test are updated, and the sweep gained the two dimensions that had hidden the site. Please re-approve or push back, since your approve predates that commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
