wankunde commented on PR #38176: URL: https://github.com/apache/spark/pull/38176#issuecomment-1273438360
> `CoalesceShufflePartitions` will make each partition size close to `ADVISORY_PARTITION_SIZE_IN_BYTES` unless some partitions are skewed. So it has no meaning to compare with `AUTO_BROADCASTJOIN_THRESHOLD`... > > I'm thinking to add a new rule to optimize smj to shj by spliting the bigger partition to smaller (like skew join did), then we can build hash relation safely. But it only works for inner join, since we need split build side. Hi, @ulysses-you Thanks for your review. * For example, if `ADVISORY_PARTITION_SIZE_IN_BYTES` is 64M, the two sides of join are : [30M, 30M, 30M, 30M, 30M] and [100M, 100M, 100M, 100M, 100M], `CoalesceShufflePartitions` will transform `ShuffleQueryStageExec` to `CustomShuffleReaderExec(stage, ShufflePartitionSpec[130M, 130M, 130M, 130M, 130M])` , this query should be able to convert to ShuffledHashJoin (not support now). * If we enable `BROADCASTJOIN`, I think the data that will be loaded into memory and build HashRelation should also less than `AUTO_BROADCASTJOIN_THRESHOLD`, i.e. `ADAPTIVE_MAX_SHUFFLE_HASH_JOIN_LOCAL_MAP_THRESHOLD <= AUTO_BROADCASTJOIN_THRESHOLD` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
