mridulm commented on pull request #34158: URL: https://github.com/apache/spark/pull/34158#issuecomment-934472550
> Stepping back a bit, I have a more fundamental conceptual question about the push-based shuffle configurations: could the decision of whether to use push-based shuffle be performed on a per-ShuffleDependency basis rather than a per-app basis? Agree, we should relook at this. Probably mirror more closely what batch fetch does. This will require a closer look to make sure we make decisions such that, for a shuffle dependency: a) We dont push at mapper side if reducers cant read it. b) Dont wait for push finalization at driver. c) Enable push shuffle reads only if shuffleDep.serializer supports relocation. d) Relook at how system is initialized - as currently it is based on a global state. Current state is a more strict enforcement, which can be relaxed as we evolve the functionality. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
