AngersZhuuuu opened a new pull request #32804: URL: https://github.com/apache/spark/pull/32804
### What changes were proposed in this pull request? Support YARN Placement Constraint, In this pr: - Add `LocalityPreferredSchedulingRequestContainerPlacementStrategy` for compute locality for SchedulingRequest - Add `ContainerImpl` for UT, since the interface Container's common API can't support `setAllocationRequestId` and `getAllocationRequestId` - Add `YarnSchedulingRequestAllocator ` as the implement of allocator when use Placement Constraint. ### Why are the changes needed? Spark can allow users to configure the Placement Constraint so that users will have more control on where the executors will get placed. For example: 1. Spark job wants to be run on machines where Python version is x or Java version is y (Node Attributes) 2. Spark job needs / does not need executors to be placed on machine where Hbase RegionServer / Zookeeper / Or any other Service is running. (Affinity / Anti Affinity) 3. Spark job wants no more than 2 of it's executors on same node (Cardinality) 4. Spark Job A executors wants / does not want to be run on where Spark Job / Any Other Job B containers runs (Application_Tag NameSpace) ### Does this PR introduce _any_ user-facing change? Use can set `spark.yarn.schedulingRequestEnabled` to use YARN placement constraint and set `spark.yarn.executor.nodeAttributes` to set constraint. ### How was this patch tested? Added UT and manuel tested this in yarn cluser: **set node attribute** ``` /usr/share/yarn-3/bin/yarn nodeattributes -attributestonodes -attributes attr1 Hostname Attribute-value rm.yarn.io/attr1 :ip-xxxx 1 /usr/share/yarn-3/bin/yarn nodeattributes -attributestonodes -attributes attr2 Hostname Attribute-value rm.yarn.io/attr2 :ip-xxx spark command export SPARK_CONF_DIR=/tmp/spark-conf-3.1.1 export SPARK_HOME=/tmp/spark-3.1.1-sdi-002-bin-3.2.0-sdi-001 /tmp/spark-3.1.1-sdi-002-bin-3.2.0-sdi-001/bin/spark-sql --queue infra \ --executor-memory 1g --executor-cores 1 \ --conf spark.yarn.schedulingRequestEnabled=true \ --conf spark.executor.instances=1 \ --conf spark.dynamicAllocation.enabled=true \ --conf spark.dynamicAllocation.initialExecutors=1 \ --conf spark.dynamicAllocation.maxExecutors=2 \ --conf spark.dynamicAllocation.minExecutors=0 \ --conf spark.dynamicAllocation.executorIdleTimeout=10s \ --conf spark.yarn.executor.nodeAttribute='attr1=1' ``` Under dynamic allocation, can work as expect, all allocated container is under NM `ip-xxx`, if we set node attribute as `attr1=2`, we can't allocate any executor container. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
