otterc commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-717737324
I have updated the PR with these changes: - Added support for subdirs under merge directory. - Removed the yarn local directory path pattern. - Removed the `registerApplication` API from `MergedShuffleFileManager` and updated the `registerExecutor` API of `MergedShuffleFileManager`. - Addressed few other comments. These are some pending tasks: - Server side configuration which is a class that provides the implementation of push-based shuffle service. Suggested by @tgravescs - Changes to the message `PushBlockStream` as suggested by @attilapiros . This is a bigger change that will touch upon `OneForOneBlockPusher` and `ExternalShuffleBlockClient`. - Other review comments from @mridulm and @attilapiros @jiangxb1987 @Ngone51 Just checking to see if you have additional feedback on this PR? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
