ulysses-you commented on PR #58942: URL: https://github.com/apache/spark/pull/58942#issuecomment-5793320348
Thanks @peter-toth, @dongjoon-hyun and @cloud-fan! All of the review's findings are in f95e44a9f42. - The pairing is owed by every multi-child clustered operator again, so a cogroup keeps the mutual check it had on the base, and only the ungrouped waiver is producer-scoped. - The operator kinds are one predicate now, `ShuffledJoin.partiallyClusteredJoinType`, read by the producer's dispatch and by the validator, rather than a copy in each file. - The ungrouped waiver asks the collapse permission the planner applies, so a member whose keys were collapsed is admitted ungrouped only where the planner would agree to group it. - A side is judged on the key the operation clusters on, as it reports it: the partition expressions carrying no cluster key are left out, and the count and the key order are the member's own. That takes in the subset shape without reading a projection the plan does not hold. - The producer states the invariant the waiver rests on, where it is decided: one side spreads its splits and the other repeats the whole group, since `replicateRightSide` is the negation of `replicateLeftSide` and the branch is reached only when the picked side may replicate. That is what makes pairing the two index by index sound, and the pairing cannot tell that from two sides that split the key between them. The tests are the suites the change touches: `ShuffleSpecSuite` and `DistributionSuite` (52), and `ValidateRequirementsSuite`, `EnsureRequirementsSuite`, `GroupPartitionsExecSuite`, `KeyGroupedPartitioningSuite` and `ProjectedOrderingAndPartitioningSuite` (362), with `dev/lint-scala` clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
