This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 5aa4f7f [SPARK-38189][K8S][DOC] Add `Priority scheduling` doc for
Spark on K8S
5aa4f7f is described below
commit 5aa4f7f2017585ef75e577974cb382d1409189a9
Author: Yikun Jiang <[email protected]>
AuthorDate: Fri Mar 4 19:28:14 2022 -0800
[SPARK-38189][K8S][DOC] Add `Priority scheduling` doc for Spark on K8S
### What changes were proposed in this pull request?
Document how to set the priority class with the pod template.
### Why are the changes needed?
Currently, we didn't have a certain doc to help user enable priority
scheduling
Related: https://github.com/apache/spark/pull/35716
https://github.com/apache/spark/pull/35639#issuecomment-1055847723
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Closes #35728 from Yikun/SPARK-38189-doc.
Authored-by: Yikun Jiang <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 727f044612c0a71097aa0d29cb3f24a53b93fc1f)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/running-on-kubernetes.md | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index 058dddb..8c9f858 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -1574,6 +1574,30 @@ Spark automatically handles translating the Spark
configs <code>spark.{driver/ex
Kubernetes does not tell Spark the addresses of the resources allocated to
each container. For that reason, the user must specify a discovery script that
gets run by the executor on startup to discover what resources are available to
that executor. You can find an example scripts in
`examples/src/main/scripts/getGpusResources.sh`. The script must have execute
permissions set and the user should setup permissions to not allow malicious
users to modify it. The script should write to STDOUT [...]
+### Resource Level Scheduling Overview
+
+There are several resource level scheduling features supported by Spark on
Kubernetes.
+
+#### Priority Scheduling
+
+Kubernetes supports [Pod
priority](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption)
by default.
+
+Spark on Kubernetes allows defining the priority of jobs by [Pod
template](#pod-template). The user can specify the
<code>priorityClassName</code> in driver or executor Pod template
<code>spec</code> section. Below is an example to show how to specify it:
+
+```
+apiVersion: v1
+Kind: Pod
+metadata:
+ labels:
+ template-label-key: driver-template-label-value
+spec:
+ # Specify the priority in here
+ priorityClassName: system-node-critical
+ containers:
+ - name: test-driver-container
+ image: will-be-overwritten
+```
+
### Stage Level Scheduling Overview
Stage level scheduling is supported on Kubernetes when dynamic allocation is
enabled. This also requires
<code>spark.dynamicAllocation.shuffleTracking.enabled</code> to be enabled
since Kubernetes doesn't support an external shuffle service at this time. The
order in which containers for different profiles is requested from Kubernetes
is not guaranteed. Note that since dynamic allocation on Kubernetes requires
the shuffle tracking feature, this means that executors from previous stages t
[...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
