imback82 commented on pull request #29473: URL: https://github.com/apache/spark/pull/29473#issuecomment-682212843
> Just a question; we cannot set a simpler rule to select which strategy (reapportioning or coalescing) we use when reading buckets? I think it is a bit annoying to set true to `repartitionBucketsInJoin.enabled` or `coalesceBucketsInJoin.enabled` data-by-data. Good point. One use case for repartition over coalesce is when there are enough cores available in the cluster, not to reduce the parallelism by coalescing. @c21, did you observe any patterns or heuristics on your workloads where repartition is preferred? > For example, if one has too small buckets (this case is extreme though), repartitioning for higher cardinality might not be able to help for better performance. This is still guarded by `spark.sql.bucketing.coalesceOrRepartitionBucketsInJoin.maxBucketRatio`, so this scenario is a little less concerning? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
