This is an automated email from the ASF dual-hosted git repository.
chia7712 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git
The following commit(s) were added to refs/heads/master by this push:
new 05bc6472 [YUNIKORN-2131] Add ready-to-use example for peemption (#723)
05bc6472 is described below
commit 05bc6472c01d9e623c0a002f801b65e056b08c5a
Author: 陳昱霖(Yu-Lin Chen) <[email protected]>
AuthorDate: Sat Dec 30 00:03:22 2023 +0800
[YUNIKORN-2131] Add ready-to-use example for peemption (#723)
Reviewers: PoAn Yang <[email protected]>, Chia-Ping Tsai
<[email protected]>
---
deployments/examples/README.md | 10 ++-
deployments/examples/preemption/README.md | 79 ++++++++++++++++++++++
.../examples/preemption/high-priority-job.yaml | 39 +++++++++++
.../preemption/low-priority-job-outside-fence.yaml | 39 +++++++++++
.../examples/preemption/low-priority-job.yaml | 39 +++++++++++
...rmal-priority-job-with-9999-priority-class.yaml | 50 ++++++++++++++
.../examples/preemption/normal-priority-job.yaml | 39 +++++++++++
.../examples/preemption/yunikorn-configs.yaml | 63 +++++++++++++++++
8 files changed, 357 insertions(+), 1 deletion(-)
diff --git a/deployments/examples/README.md b/deployments/examples/README.md
index afa634c0..63ee731c 100644
--- a/deployments/examples/README.md
+++ b/deployments/examples/README.md
@@ -102,7 +102,7 @@ The namespace example uses a placement rule and special
queue configuration. The
* run the sleep pod in the production namespace which creates a new
`production` queue using the local
[sleeppod_prod.yaml](namespace/sleeppod_prod.yaml): `kubectl create -f
namespaces.yaml`.
The pod spec does not specify a queue just a namespace but the application
will be run in the newly created `root.production` queue. This queue does not
exist in the queue configuration.
-### placements
+## placements
The placements' rules are described in [Yunikorn
website](https://yunikorn.apache.org/docs/user_guide/placement_rules).
App placements rules in Yunikorn contains `Provided Rule`, `User Name Rule`,
`Fixed Rule`, `Tag Rule`.
Every placement example includes a example yaml file and a config yaml file.
@@ -113,3 +113,11 @@ Before deploying the pods, the configuration field in
yunikorn-release/helm/yuni
* [User Name Rule](./placements/username)
* [Fixed Rule](./placements/fixed)
* [Tag Rule](./placements/tag)
+
+## preemption
+This example demonstrates how to set up priority queues and initiate jobs to
trigger preemption. Follow the steps in [README.md](./preemption/README.md) to
understand how preemption works between queues.
+
+More documents for preemption:
+* [App & Queue
Priorities](https://yunikorn.apache.org/docs/user_guide/priorities)
+* [Preemption](https://yunikorn.apache.org/docs/next/design/preemption/)
+* [Use Case: Preemption & Priority scheduling with
fencing](https://yunikorn.apache.org/docs/user_guide/use_cases#preemption--priority-scheduling-with-fencing)
diff --git a/deployments/examples/preemption/README.md
b/deployments/examples/preemption/README.md
new file mode 100644
index 00000000..1478d1b7
--- /dev/null
+++ b/deployments/examples/preemption/README.md
@@ -0,0 +1,79 @@
+<!--
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+-->
+
+# A simple example to demo preemption
+
+## Description
+This example includes yamls to
+1. Create priority queue with preemption fence.
([yunikorn-configs.yaml](./yunikorn-configs.yaml))
+2. Create application with PriorityClass.
([normal-priority-job-with-9999-priority-class.yaml](./normal-priority-job-with-9999-priority-class.yaml))
+
+By following the steps below, you will be able to see the below behaviors of
preemption
+1. A task cannot preempt tasks outside its preemption fence.
+2. Preemption is aiming to ensure that the queue's resource usage reaches at
least the guaranteed amount of resources.
+3. Preemption can never leave a queue lower than its guaranteed capacity.
+
+## Run the example workloads
+ ```shell script
+ # Update YuniKorn config
+ kubectl apply -f yunikorn-configs.yaml
+
+ # Create low-priority jobs
+ kubectl apply -f low-priority-job-outside-fence.yaml
+ kubectl apply -f low-priority-job.yaml
+ kubectl get pods | grep Running
+ # Should have 10 pods in Running.
+ # Expected: 5 outside fence, 5 low-priority.
+
+ # Create normal-priority job
+ kubectl apply -f normal-priority-job.yaml
+ kubectl get pods | grep Running
+ # Preemption is triggered after a specified delay, with the aim of ensuring
that each queue's resource usage reaches at least the guaranteed amount of
resources. A task cannot preempt tasks outside its preemption fence.
+ # Wait 1~2 minutes for preemption to occur.
+ # Expected: 5 outside fence, 2 low-priority, 3 normal-priority.
+
+ # Remove low-priority job
+ kubectl delete -f low-priority-job.yaml
+ kubectl get pods | grep Running
+ # Expected: 5 outside fence, 5 normal-priority.
+
+ # Create high-priority job
+ kubectl apply -f high-priority-job.yaml
+ kubectl get pods | grep Running
+ # Preemption can never leave a queue lower than its guaranteed capacity.
+ # Wait 1~2 minutes for preemption to occur.
+ # Expected: 5 outside fence, 3 normal-priority, 2 high-priority.
+
+ # Remove normal-priority job
+ kubectl delete -f normal-priority-job.yaml
+ kubectl get pods | grep Running
+ # Expected: 5 outside fence, 5 high-priority.
+
+ # Create normal-priority job with PriorityClass
+ kubectl apply -f normal-priority-job-with-9999-priority-class.yaml
+ kubectl get pods | grep Running
+ # After applying priority.offset in the queue, the job has the highest
priority.
+ # Wait 1~2 minutes for preemption to occur.
+ # Expected: 5 outside fence, 3 high-priority, 2
normal-priority-job-with-9999-priority-class.
+
+ #cleanup
+ kubectl delete -f normal-priority-job-with-9999-priority-class.yaml
+ kubectl delete -f high-priority-job.yaml
+ kubectl delete -f low-priority-job-outside-fence.yaml
+ ```
+
diff --git a/deployments/examples/preemption/high-priority-job.yaml
b/deployments/examples/preemption/high-priority-job.yaml
new file mode 100644
index 00000000..db963b37
--- /dev/null
+++ b/deployments/examples/preemption/high-priority-job.yaml
@@ -0,0 +1,39 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: batch/v1
+kind: Job
+metadata:
+ name: high-priority-job
+spec:
+ completions: 10
+ parallelism: 10
+ template:
+ metadata:
+ labels:
+ applicationId: high-priority-job
+ queue: root.sandbox.tenants.tenant-high
+ spec:
+ schedulerName: yunikorn
+ containers:
+ - name: pause
+ image: registry.k8s.io/pause:3.7
+ resources:
+ requests:
+ cpu: "100m"
+ memory: "100Mi"
+ restartPolicy: Never
diff --git
a/deployments/examples/preemption/low-priority-job-outside-fence.yaml
b/deployments/examples/preemption/low-priority-job-outside-fence.yaml
new file mode 100644
index 00000000..647ec4b6
--- /dev/null
+++ b/deployments/examples/preemption/low-priority-job-outside-fence.yaml
@@ -0,0 +1,39 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: batch/v1
+kind: Job
+metadata:
+ name: low-priority-job-outside-fence
+spec:
+ completions: 5
+ parallelism: 5
+ template:
+ metadata:
+ labels:
+ applicationId: low-priority-job-outside-fence
+ queue: root.sandbox.system.system-low
+ spec:
+ schedulerName: yunikorn
+ containers:
+ - name: pause
+ image: registry.k8s.io/pause:3.7
+ resources:
+ requests:
+ cpu: "100m"
+ memory: "100Mi"
+ restartPolicy: Never
diff --git a/deployments/examples/preemption/low-priority-job.yaml
b/deployments/examples/preemption/low-priority-job.yaml
new file mode 100644
index 00000000..64de7acf
--- /dev/null
+++ b/deployments/examples/preemption/low-priority-job.yaml
@@ -0,0 +1,39 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: batch/v1
+kind: Job
+metadata:
+ name: low-priority-job
+spec:
+ completions: 5
+ parallelism: 5
+ template:
+ metadata:
+ labels:
+ applicationId: low-priority-job
+ queue: root.sandbox.tenants.tenant-low
+ spec:
+ schedulerName: yunikorn
+ containers:
+ - name: pause
+ image: registry.k8s.io/pause:3.7
+ resources:
+ requests:
+ cpu: "100m"
+ memory: "100Mi"
+ restartPolicy: Never
diff --git
a/deployments/examples/preemption/normal-priority-job-with-9999-priority-class.yaml
b/deployments/examples/preemption/normal-priority-job-with-9999-priority-class.yaml
new file mode 100644
index 00000000..da7d8a32
--- /dev/null
+++
b/deployments/examples/preemption/normal-priority-job-with-9999-priority-class.yaml
@@ -0,0 +1,50 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: scheduling.k8s.io/v1
+kind: PriorityClass
+metadata:
+ name: add-9999-priority
+ annotations:
+ yunikorn.apache.org/allow-preemption: "true"
+value: 9999
+globalDefault: false
+
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+ name: normal-priority-job-with-9999-priority-class
+spec:
+ completions: 10
+ parallelism: 10
+ template:
+ metadata:
+ labels:
+ applicationId: normal-priority-job-with-9999-priority-class
+ queue: root.sandbox.tenants.tenant-normal
+ spec:
+ schedulerName: yunikorn
+ containers:
+ - name: pause
+ image: registry.k8s.io/pause:3.7
+ resources:
+ requests:
+ cpu: "100m"
+ memory: "100Mi"
+ restartPolicy: Never
+ priorityClassName: add-9999-priority
diff --git a/deployments/examples/preemption/normal-priority-job.yaml
b/deployments/examples/preemption/normal-priority-job.yaml
new file mode 100644
index 00000000..f82068bc
--- /dev/null
+++ b/deployments/examples/preemption/normal-priority-job.yaml
@@ -0,0 +1,39 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: batch/v1
+kind: Job
+metadata:
+ name: normal-priority-job
+spec:
+ completions: 10
+ parallelism: 10
+ template:
+ metadata:
+ labels:
+ applicationId: normal-priority-job
+ queue: root.sandbox.tenants.tenant-normal
+ spec:
+ schedulerName: yunikorn
+ containers:
+ - name: pause
+ image: registry.k8s.io/pause:3.7
+ resources:
+ requests:
+ cpu: "100m"
+ memory: "100Mi"
+ restartPolicy: Never
diff --git a/deployments/examples/preemption/yunikorn-configs.yaml
b/deployments/examples/preemption/yunikorn-configs.yaml
new file mode 100644
index 00000000..b71e1661
--- /dev/null
+++ b/deployments/examples/preemption/yunikorn-configs.yaml
@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: yunikorn-configs
+ namespace: yunikorn
+data:
+ queues.yaml: |
+ partitions:
+ - name: default
+ queues:
+ - name: root
+ queues:
+ - name: sandbox
+ resources:
+ max:
+ {memory: 1000Mi, vcore: 1000m}
+ queues:
+ - name: tenants
+ properties:
+ preemption.policy: fence
+ preemption.delay: 10s
+ queues:
+ - name: tenant-normal
+ submitacl: '*'
+ resources:
+ guaranteed:
+ {memory: 300Mi, vcore: 300m}
+ properties:
+ priority.offset: "0"
+ - name: tenant-high
+ submitacl: '*'
+ resources:
+ guaranteed:
+ {memory: 300Mi, vcore: 300m}
+ properties:
+ priority.offset: "1000"
+ - name: tenant-low
+ submitacl: '*'
+ properties:
+ priority.offset: "-1000"
+ - name: system
+ queues:
+ - name: system-low
+ submitacl: '*'
+ properties:
+ priority.offset: "-9999"
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
