Luca Canali created SPARK-41585:
-----------------------------------
Summary: The Spark exclude node functionality for YARN should work
independently of dynamic allocation
Key: SPARK-41585
URL: https://issues.apache.org/jira/browse/SPARK-41585
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 3.3.1
Reporter: Luca Canali
The Spark exclude node functionality for YARN, introduced in SPARK-26688,
allows users to specify a list of node names that are excluded from resource
allocation. This is done using the configuration parameter:
{{spark.yarn.exclude.nodes}}
The feature currently works only for executors allocated via dynamic
allocation. To use the feature on Spark 3.3.1, for eaxmple, one needs to
configure spark.dynamicAllocation.minExecutors=0 and
spark.executor.instances=0, therefore relying on executor resource allocation
only via dynamic allocation.
This proposes to extend the use of Spark exclude node functionality for YARN
beyond dynamic allocation, which I believe makes it more consistent also with
what the documentation reports for this feature/configuration parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
