LuciferYang commented on PR #58527: URL: https://github.com/apache/spark/pull/58527#issuecomment-5548585122
`createKeyedShuffleSpec` still resolves a `PartitioningCollection` with `collectFirst`, taking the first satisfying member, while this PR makes the per-child branch resolve the finest member both sides agreed on. The default `requireAllClusterKeysForCoPartition=true` keeps partial-coverage members out of the candidates, so this only shows up with `requireAllClusterKeysForCoPartition=false` plus `allowKeysSubsetOfPartitionKeys=true`: when both sides report alias cross-products with a coarse member first, SPJ pairs the coarse members and joins on them, leaving the finer pairing unused; with coarse first on one side and fine first on the other, the picks fail to pair and the per-child branch takes over, grouping each side on its finest matching member, so what is skipped is the SPJ partition-value pushdown, not a shuffle. Results stay correct. If you align it, the choice cannot be an independent per-side finest (a side with only coarse members would then fail to pair); it has to pair like `bestMemberOpt` does, by finest member compatible with some member of the other side. Either fixing it here or a follow-up works for me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
