ulysses-you commented on PR #38176: URL: https://github.com/apache/spark/pull/38176#issuecomment-1273154006
`CoalesceShufflePartitions` will make each partition size close to `ADVISORY_PARTITION_SIZE_IN_BYTES` unless some partitions are skewed. So it has no meaning to compare with `AUTO_BROADCASTJOIN_THRESHOLD`... I'm thinking to add a new rule to optimize smj to shj by spliting the bigger partition to smaller (like skew join did), then we can build hash relation safely. But it only works for inner join, since we need split build side. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
