LucaCanali opened a new pull request, #39127: URL: https://github.com/apache/spark/pull/39127
### What changes were proposed in this pull request? The Spark exclude node functionality for Spark on YARN, introduced in [SPARK-26688](https://issues.apache.org/jira/browse/SPARK-26688), allows users to specify a list of node names that are excluded from resource allocation. This is done using the configuration parameter: `spark.yarn.exclude.nodes` The feature currently works only for executors allocated via dynamic allocation. To use the feature on Spark 3.3.1, for example, one may need also to configure `spark.dynamicAllocation.minExecutors=0` and `spark.executor.instances=0`, therefore relying on executor resource allocation only via dynamic allocation. ### Why are the changes needed? This proposes to extend the use of Spark exclude node functionality for YARN beyond dynamic allocation, which I believe makes it more consistent also with what the documentation reports for this feature/configuration parameter. ### Does this PR introduce _any_ user-facing change? Yes, this allows using the executor exclude nodes feature for Spark on YARN also when not using dynamical allocation. ### How was this patch tested? Existing tests and manual tests on a YARN cluster -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
