[ https://issues.apache.org/jira/browse/SPARK-41585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves resolved SPARK-41585. ----------------------------------- Fix Version/s: 3.5.0 Target Version/s: 3.5.0 Assignee: Luca Canali Resolution: Fixed > The Spark exclude node functionality for YARN should work independently of > dynamic allocation > --------------------------------------------------------------------------------------------- > > Key: SPARK-41585 > URL: https://issues.apache.org/jira/browse/SPARK-41585 > Project: Spark > Issue Type: Improvement > Components: YARN > Affects Versions: 3.0.3, 3.1.3, 3.2.2, 3.3.1 > Reporter: Luca Canali > Assignee: Luca Canali > Priority: Minor > Fix For: 3.5.0 > > > The Spark exclude node functionality for Spark on YARN, introduced in > SPARK-26688, allows users to specify a list of node names that are excluded > from resource allocation. This is done using the configuration parameter: > {{spark.yarn.exclude.nodes}} > The feature currently works only for executors allocated via dynamic > allocation. To use the feature on Spark 3.3.1, for example, one may set the > configurations {{{}spark.dynamicAllocation.enabled{}}}=true, > spark.dynamicAllocation.minExecutors=0 and spark.executor.instances=0, thus > making Spark spawning executors only via dynamic allocation. > This proposes to document this behavior for the current Spark release and > also proposes an improvement of this feature by extending the scope of Spark > exclude node functionality for YARN beyond dynamic allocation, which I > believe makes it more generally useful. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org