mridulm commented on pull request #34158: URL: https://github.com/apache/spark/pull/34158#issuecomment-934019383
> If the approach described above was easy to implement then I'd prefer it since I think it's easier to reason about: the approach taken in this PR requires me to reason about push-based shuffle internals to argue that it's okay for the executor and driver to have divergent effective configurations in certain places (as discussed upthread). > @rmcyang Can you file a jira and link it to the improvements jira ? Given the correctness, we can get it addressed as follow up after release (this PR is blocking RC). Does that sound fine @JoshRosen ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
