cloud-fan commented on pull request #28123: URL: https://github.com/apache/spark/pull/28123#issuecomment-630627303
The general idea LGTM. Two high-level questions ： 1. It's not possible to have an accurate cost model to guide this optimization. Do we have a heuristic? like coalescing 10000 buckets to 2 is very likely to cause regression as parallelism is reduced too much. 2. Are we going to support joins with more than 2 tables? e.g. "100 buckets table" join "50 buckets table" join "10 buckets table". ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
