rafaelfc-olx opened a new pull request #26505: [SPARK-24353][K8S][2.4] Support for specifying multiple custom tolerations to driver and executor pods URL: https://github.com/apache/spark/pull/26505 ### What changes were proposed in this pull request? The PR allows a user to specify custom [k8s tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) when running Spark on kubernetes for both driver and executor pods. The user can now set the **key**, **effect**, **operator** and **value** properties when running spark-submit, as long as they're prefixed by: `spark.kubernetes.driver.tolerations.n` or `spark.kubernetes.executor.tolerations.n` where n = a number starting from 0 that is incremented of 1. #### Why not using the key field, as per original design doc attached to the JIRA issue? As per [kubernetes toleration v1 spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/#toleration-v1-core), the field key can be unset, thus disqualifying it as a unique identifier. Some examples: ``` # driver pods tolerate the taint group:spark-driver registered to a specialised worker group spark.kubernetes.driver.tolerations.0.key group spark.kubernetes.driver.tolerations.0.operator Equal spark.kubernetes.driver.tolerations.0.effect NoSchedule spark.kubernetes.driver.tolerations.0.value spark-driver # executor pods tolerate the taint group:spark-executor registered to another specialised worker group spark.kubernetes.executor.tolerations.0.key group spark.kubernetes.executor.tolerations.0.operator Equal spark.kubernetes.executor.tolerations.0.effect NoSchedule spark.kubernetes.executor.tolerations.0.value spark-executor # driver pods tolerate when the key spot exists spark.kubernetes.executor.tolerations.1.key spot spark.kubernetes.executor.tolerations.1.operator Exists ``` ### Why are the changes needed? Spark already supports using [node selectors](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector), which is helpful to pin driver/executor pods to a specific worker group. However, without tainting the workers and setting at least one toleration to the pods, the specialised worker group cannot ensure repelling non-spark pods, which could potentially lead to undesired, potentially unsuitable pods being scheduled in the specialised worker group. This can be very important for many use cases, such as running spark jobs in [AWS spot instances](https://aws.amazon.com/ec2/spot/), which can be reclaimed at anytime by Amazon thus making it not suitable for every kind of workload. ### Does this PR introduce any user-facing change? No, it's a new feature. ### How was this patch tested? added both unit tests and k8s integration tests
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
