[
https://issues.apache.org/jira/browse/SPARK-41585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves resolved SPARK-41585.
-----------------------------------
Fix Version/s: 3.5.0
Target Version/s: 3.5.0
Assignee: Luca Canali
Resolution: Fixed
> The Spark exclude node functionality for YARN should work independently of
> dynamic allocation
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-41585
> URL: https://issues.apache.org/jira/browse/SPARK-41585
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 3.0.3, 3.1.3, 3.2.2, 3.3.1
> Reporter: Luca Canali
> Assignee: Luca Canali
> Priority: Minor
> Fix For: 3.5.0
>
>
> The Spark exclude node functionality for Spark on YARN, introduced in
> SPARK-26688, allows users to specify a list of node names that are excluded
> from resource allocation. This is done using the configuration parameter:
> {{spark.yarn.exclude.nodes}}
> The feature currently works only for executors allocated via dynamic
> allocation. To use the feature on Spark 3.3.1, for example, one may set the
> configurations {{{}spark.dynamicAllocation.enabled{}}}=true,
> spark.dynamicAllocation.minExecutors=0 and spark.executor.instances=0, thus
> making Spark spawning executors only via dynamic allocation.
> This proposes to document this behavior for the current Spark release and
> also proposes an improvement of this feature by extending the scope of Spark
> exclude node functionality for YARN beyond dynamic allocation, which I
> believe makes it more generally useful.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
