wankunde commented on PR #41786: URL: https://github.com/apache/spark/pull/41786#issuecomment-1618666379
Thanks @mridulm @maryannxue for your review. If a query contains shuffle A and shuffle B, there are many PartialReducerPartitions after OptimizeSkewedJoin optimization, and shuffle B is a local read shuffle. Enable `spark.shuffle.reduceLocality.enabled` may takes some extra time to get the preferred locations. But there is a limit on the number of PartialReducerPartitions in our production environment, so it is okay for me not to make this change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
