c21 commented on pull request #35552: URL: https://github.com/apache/spark/pull/35552#issuecomment-1046151139
> I looked into https://github.com/apache/spark/pull/35574, and it looks good in overall. I'll close this. Thanks for making the proposal be better! Thank you @HeartSaVioR for proposing the fix in the first place, and leading the discussion! > I even doubt we have to be adaptive for this case, unless the condition of being adaptive can be determined without requiring actual execution. @HeartSaVioR - The config of adaptive of partial aggregate skip, is similar to Spark AQE, which can be enabled by default. For our company production environment, we actually enable the feature by default. So end user could potentially not notice the data skew at all, as Spark resolves it adaptively under the hood. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
