peter-toth commented on PR #58351: URL: https://github.com/apache/spark/pull/58351#issuecomment-5464623712
Rebased and pushed. SPARK-58974 merged, so this is no longer stacked and the PR is one commit against master. The description is regenerated from it. The flag's contract and every producer are as reviewed, and two things around them changed shape. **`groupedSatisfies` split in two.** Its `!isGrouped` term was never a property of the partitioning, it was a caller discriminator: `satisfies0` only ever called the method with `isGrouped = true`, and `EnsureRequirements` only ever with `false`. So there are now `keysSatisfy` (the key matching, private) and `mayGroupToSatisfy` (`keysSatisfy` plus the permission to coalesce), and `EnsureRequirements` asks the second one of its non-grouped list. The permission is checked first, as it was before the split, so the gated case pays nothing for the key matching. **One projection.** `KeyedPartitioning.project` applies a projection and decides both `isGrouped` and `isCollapsed`, and `PartitioningPreservingUnaryExecNode` and `createShuffleSpec` both go through it. `UnionExec`'s merge rule moved to `KeyedPartitioning.concat` for the same reason, so the four rules for how the flag travels are now all on the type: from scratch, projection, grouping, concatenation. Sidenote: both point the same way as item 5 of the follow-up list on #58262. The remaining half is that *which* positions are operation keys is still derived in more than one place, and a comment in `keysSatisfy` records it. Two decisions from your review are unchanged and stated in the threads: `ReuseExchange` keeps the flag in equality, and `LeftExistence` keeps inheriting it. The commit message no longer carries the `Co-authored-by` line, to match what you did to the description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
