SaurabhChawla100 edited a comment on pull request #27636: URL: https://github.com/apache/spark/pull/27636#issuecomment-642102323
@agrawaldevesh - Thanks for checking this PR, Sure will check this PR https://github.com/apache/spark/pull/27864 and tried to use this in the current PR Regarding this > `I also don't fully understand the relationship b/w this PR and the original decommissioning PR https://github.com/apache/spark/pull/27864 I am trying to get a sense of the end state with all these multiple decommissioning PR's trying to stretch the framework in different ways.` So https://github.com/apache/spark/pull/27864 this is based on executor decommission where in k8s/ standalone mode the pods/worker received the SIGPWR and stop scheduling the task on the new task to executors on the same worker and creating new executors on worker. Its more of decommissioning of Executors Where as in this current PR we are using the node information from the Yarn and using the information related to Node (spot node , Preemptible VM in GCP) to take an action. So on receiving the information related to node decommissioning we take the following action as described in the PR description https://github.com/apache/spark/pull/27636#issue-377085357. Its more of Node Decommissioning where driver is responsible for executor decommission and shuffle decommission. In this PR we are maintaining a life cycle how node decommissioned is handled This is the life cycle of the nodes which is decommissioned DECOMMISSIONING -> EXECUTOR_DECOMMISSIONED -> SHUFFLEDATA_DECOMMISSIONED -> TERMINATED. Also we can create another PR for making common parts for both PR once this PR is merged , the current PR and https://github.com/apache/spark/pull/27864 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
