hiboyang commented on pull request #34864: URL: https://github.com/apache/spark/pull/34864#issuecomment-992832070
Hi Dongjoon, you got some misunderstandings here. I am writing a design doc for this PR. Hope that will help you to understand more and address your questions. > You are completely wrong because you already know the worker decommission feature. > > > but it will not work well when there is shuffle data distributed on many executors (those executors cannot be released). > > You should mention this in the PR description explicitly instead of misleading the users. > > > The work here (storing shuffle data on S3) does not conflict with worker decommission feature. The eventual goal is to store shuffle data on S3 or other external storage directly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
