Deependra-Patel opened a new pull request, #42536: URL: https://github.com/apache/spark/pull/42536
### What changes were proposed in this pull request? This will prevent driver from scheduling new tasks on these executors, they will also migrate any shuffle data they wrote. Migration of shuffle data written by executors which are removed by Spark because they are no longer needed is not handled. ### Why are the changes needed? This removes containers from DECOMMISSIONING YARN nodes hence node can be removed from cluster. We are re-using existing Spark graceful decommissioning mechanism that migrates the shuffle data. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Tested manually by running on a cluster with YARN and manually putting a node in DECOMMISSIONING state. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
