peter-toth commented on PR #59165: URL: https://github.com/apache/spark/pull/59165#issuecomment-5971975711
@dongjoon-hyun, I looked into the corner case. With `spark.sql.sources.v2.bucketing.shuffle.enabled`, a join on two keys whose sides pair only through a transform of an expression gets one more shuffle with this PR. I added it to the costs in the description. I'd intentionally not fix it here, since restoring master's plan is questionable. Both plans co-partition on only one of the two join keys, which `spark.sql.requireAllClusterKeysForCoPartition` is meant to prevent. I'll file a separate ticket for that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
