abhishekd0907 commented on PR #35683: URL: https://github.com/apache/spark/pull/35683#issuecomment-1152971009
> Thanks for @abhishekd0907 to submit this PR. I think this is a good improvement for the Spark jobs' stability, especially in the scenario where we implement elastic Yarn NMs on K8s. > > > we do not want to move shuffle blocks around if decomissioning is enabled when external shuffle service is in use. > > @mridulm But in our internal cluster, we deployed the ess on Yarn nodemangers, and sometimes nm will be gracefully decommissioned, it means that the ess will be invalid on these nms. > > So why not allow trigger decommission event to executors when ess is in use? @zuston I will raise a separate PR for this scenario shortly after this one is merged so it should cover things exhaustively. This is the associated sub-task: [SPARK-39024](https://issues.apache.org/jira/browse/SPARK-39024) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
