dongjoon-hyun commented on pull request #34864: URL: https://github.com/apache/spark/pull/34864#issuecomment-992740381
You are completely wrong because you already know the worker decommission feature. > but it will not work well when there is shuffle data distributed on many executors (those executors cannot be released). You should mention this in the PR description explicitly instead of misleading the users. > The work here (storing shuffle data on S3) does not conflict with worker decommission feature. The eventual goal is to store shuffle data on S3 or other external storage directly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
