This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-yunikorn-site.git
The following commit(s) were added to refs/heads/master by this push:
new 533110a [YUNIKORN-308] Document queue quota in hierarchy (#27)
533110a is described below
commit 533110ad5c3d5aac5e1182bb01ff832ed3778207
Author: Wilfred Spiegelenburg <[email protected]>
AuthorDate: Fri Dec 11 16:52:58 2020 +1100
[YUNIKORN-308] Document queue quota in hierarchy (#27)
Documentation on how quotas in a queue hierarchy interact and work.
Additional updates to the examples to explain the same points in each
example.
---
docs/assets/namespace-mapping.png | Bin 0 -> 327547 bytes
docs/assets/queue-resource-quotas.png | Bin 0 -> 283689 bytes
docs/user_guide/resource_quota_mgmt.md | 231 +++++++++++++++++++++------------
3 files changed, 149 insertions(+), 82 deletions(-)
diff --git a/docs/assets/namespace-mapping.png
b/docs/assets/namespace-mapping.png
new file mode 100644
index 0000000..9ad07da
Binary files /dev/null and b/docs/assets/namespace-mapping.png differ
diff --git a/docs/assets/queue-resource-quotas.png
b/docs/assets/queue-resource-quotas.png
new file mode 100644
index 0000000..fb72138
Binary files /dev/null and b/docs/assets/queue-resource-quotas.png differ
diff --git a/docs/user_guide/resource_quota_mgmt.md
b/docs/user_guide/resource_quota_mgmt.md
index 7cbfa01..1d3e280 100644
--- a/docs/user_guide/resource_quota_mgmt.md
+++ b/docs/user_guide/resource_quota_mgmt.md
@@ -22,39 +22,75 @@ specific language governing permissions and limitations
under the License.
-->
-YuniKorn can offer more fine-grained resource quota management comparing to
simply
-using namespace resource quota. Here are some how-to documents about setting up
-resource quota management with YuniKorn queues.
+## Quota configuration and rules
+YuniKorn can offer a finer grained resource quota management setup compared to
the simple namespace resource quota provided by Kubernetes.
-## Option 1) Static queues
+On Kubernetes a pod must fit into the namespace quota when the pod is
submitted.
+If the pod does not fit in the namespace quota the pod is rejected.
+The client must implement a retry-mechanism and re-submit the pod if it needs
the pod to be scheduled.
-### Goal
+Contrary to quotas in Kubernetes YuniKorn does not enforce quotas on
submission but only on actively consumed resources.
+To explain the difference: when using YuniKorn for quota enforcement a new pod
submitted to Kubernetes is always accepted.
+Yunikorn will queue the pod without counting the queued pod's resources
towards the consumed quota.
+When YuniKorn tries to schedule the pod it checks at scheduling time if the
pod fits in the quota configured for the queue the pod is assigned to.
+If at that point the pod does not fit in the quota the pod is skipped and not
counted in the resource consumption.
+This means that until a scheduling attempt of a pod is successful a pod it is
not consuming resources in the YuniKorn quota system.
-Pre-setup a hierarchy of queues with min/max capacity, users can only submit
-jobs to the leaf queues. This approach fully manages the resource capacity for
-each of the queues, which is suitable to the scenarios that queues do not
change
-too often.
+Resource quotas in YuniKorn are linked to the queue and its place in the queue
hierarchy.
+The base of the queue structure, the `root` queue, does not allow setting a
quota as it reflects the current size of the cluster.
+Node additions and removals update the `root` queue quota automatically.
-### Configuration
+Beside the `root` queue the quotas can be set, and is enforced, at any point
in the hierarchy.
+Every queue can have a quota set. The quota is enforced recursively throughout
the hierarchy.
+This means that a child queue can never use more resources than the
**configured** quota of the parent queue.
+Setting a quota on a child queue larger than its parent queue's quota would
thus not have any effect and is handled as a configuration error.
+
+In the hierarchy there are some further rules that need to be considered.
+If a parent queue has multiple children the sum of the **usage** of all
children combined can never exceed the quota **configured** on the parent.
+However, from a configuration perspective this does not mean that the sum of
the **configured** quotas for all children must be smaller than the parent
quota.
+
+
+
+As an example the `root.parent` queue has a quota of 900.
+It contains three child queues, two with a quota set.
+The `root.parent.child1` has no quota set and will thus be limited to the
`root.parent` quota.
+The two other queues `root.parent.child2` and `root.parent.child3` each have a
quota of 750 set.
+During normal operation the total usage of the 3 child queues together will be
900.
+The applications running in each child queue have a demand of more than 1000
each.
+
+Distribution in that case could be any of:
+* all 900 used by just the `child1` queue
+* spread out evenly over the 3 queues (300 by each)
+* `child2` maxed out using 750, and the left over 150 used by `child3`
+
+The exact distribution between the queues will fluctuate and is dependent on
the scheduling policies.
:::note
-The following configuration is an example to demonstrate the format,
-you need to setup the queue hierarchy based on your own structure and capacity,
+The following configuration examples are just to demonstrate the format needed
+to create a queue hierarchy with quotas set.
:::
-Apply the following configuration to YuniKorn's configmap:
+## Static queue definition
+
+### Goal
+A preconfigured hierarchy of queues with a maximum and guaranteed capacity.
+The users can only submit applications to the leaf queues.
+This approach manages the resource capacity for each of the queues, which is
suitable to the scenarios that queues do not change too often.
+
+### Configuration
+Apply the following configuration to YuniKorn's configmap to:
+* setup 3 queues under `root`
+* each queue has a specific guaranteed and maximum capacity
+* anyone can submit to any queue
```yaml
partitions:
- -
- name: default
+ - name: default
queues:
- -
- name: root
+ - name: root
submitacl: '*'
queues:
- -
- name: advertisement
+ - name: advertisement
resources:
guaranteed:
memory: 500000
@@ -62,8 +98,7 @@ partitions:
max:
memory: 800000
vcore: 80000
- -
- name: search
+ - name: search
resources:
guaranteed:
memory: 400000
@@ -71,8 +106,7 @@ partitions:
max:
memory: 600000
vcore: 60000
- -
- name: sandbox
+ - name: sandbox
resources:
guaranteed:
memory: 100000
@@ -82,36 +116,33 @@ partitions:
vcore: 10000
```
-in this example, we are going to setup 3 queues under root, and each of them
has
-a specific min/max capacity set up.
-
-### Run workloads
-
-In order to run jobs in specific queues, you will need to set the following
label in all pods' spec:
+### Run a workload
+In order to run applications in specific queues, you will need to set the
following labels in all pod specs.
+All pods with the same `applicationID` label are considered ti be one
application.
+In the below example the application `my-test-app` will run in the queue
`root.sandbox`:
```yaml
labels:
app: my-test-app
- applicationId: " my-test-app-01"
+ applicationId: "my-test-app-01"
queue: root.sandbox
```
-## Option 2) 1:1 mapping from namespaces to queues
+## Namespace to queue mapping
### Goal
-
-User just needs to setup namespaces, YuniKorn automatically maps each
namespace to an internal resource queue (AKA dynamical queue).
-There is no additional steps to create YuniKorn queues, all queues will be
created dynamically,
-resource allocation and quotas will be managed by YuniKorn instead of the
namespace resource quota.
+Automatically map a Kubernetes `namespace` to a queue in YuniKorn.
+The user creates the required namespaces in Kubernetes.
+The YuniKorn k8s shim and core scheduler automatically pass the required
information and map the namespace to a queue, creating the queue if it does not
exist.
+The resource quota will be managed by YuniKorn instead of using the Kubernetes
namespace quota.
+This does require the namespaces to be setup without Kubernetes quota
enforcement and tags as per the [setup](#Namespace-quota) below.
### Configuration
-
Apply the following configuration to YuniKorn's configmap:
```yaml
partitions:
- -
- name: default
+ - name: default
placementrules:
- name: tag
value: namespace
@@ -121,80 +152,116 @@ partitions:
submitacl: '*'
properties:
application.sort.policy: stateaware
-
```
-Note, the property `application.sort.policy` in this configuration is set to
-`stateaware`. This is a simple app sorting policy applicable for batch jobs,
you
-can find more document [here](sorting_policies.md#StateAwarePolicy).
+This configuration places an application based on the `tag` rule.
+The tag selected is the `namespace` tag which is automatically added by the
k8s shim to all applications that get created.
+The `create` flag is set to true which will trigger the creation of the queue
with the same name as the namespace if it does not exist.
-You can do this during the installation by overwriting the configuration in the
-[helm chart
template](https://github.com/apache/incubator-yunikorn-release/blob/724ec82d0d548598e170cc6d5ca6aaae00f8286c/helm-charts/yunikorn/values.yaml#L71-L81).
+Applications within the automatically created child queues will be sorted
based sorting policy set on the parent queue.
+In this case the property `application.sort.policy` is in this configuration
set to `stateaware`.
+This is a simple app sorting policy applicable for batch jobs, you can find
more document [here](sorting_policies.md#StateAwarePolicy).
-### Set up namespaces
+You can change the configuration using the helm charts during the installation
by overwriting the configuration in the
+[helm chart
template](https://github.com/apache/incubator-yunikorn-release/blob/master/helm-charts/yunikorn/values.yaml#L71-L81).
-Continue to create namespaces like before, do not create namespace quota
anymore.
-Instead, set the following annotation in the namespace object:
+### Namespace quota
+Namespaces in Kubernetes contain the quota information.
+If a quota is set on a namespace Kubernetes will automatically enforce the
quota.
+In the case that YuniKorn is used for quota enforcement no quota must be set
on the namespace.
+To allow specifying a quota on the namespace the following annotations should
be set in the namespace object:
```yaml
yunikorn.apache.org/namespace.max.cpu: "64"
yunikorn.apache.org/namespace.max.memory: "100Gi"
```
+YuniKorn will parse these annotations and set the maximum capacity of the
queue mapped to this namespace.
+The values specified follow the standard Kubernetes formatting and unit
specification.
+Currently, we only support mapping memory and cpu not other resource types.
-YuniKorn will parse the annotation and set the max capacity of the dynamical
queue
-that mapped to this namespace to 64 CPU and 100GB memory.
+The example above will limit the queue mapped to the annotated namespace to 64
CPUs and 100GB memory.
-### Run workloads
+### Run a workload
-Jobs continue to be submitted to namespaces, based on the `Placementrule` used
-in the configuration. YuniKorn will automatically run the job and all its pods
in
-the corresponding queue. For example, if a job is submitted to namespace
`development`,
-then you will see the job is running in `root.development` queue.
+Applications, and the pods that are part of the application, can be submitted
without specific labels.
+YuniKorn will automatically add the required tags.
+The configured placement rule will create the queue, if required, and add the
application to the queue.
+
+For example, if an application is submitted to namespace `development`, then
the application will run in the `root.development` queue.
-## Option 3) Hierarchy queues with dynamical leaves that mapped to namespaces
+## Parent queue mapping for namespaces
### Goal
-Though the tag placement rule using the `namespace` tag is capable of putting
applications based on the name of the namespace, sometimes more dynamic
placement is required.
+Though the tag placement rule using the `namespace` tag is capable of placing
an application in a queue this might not be enough in all setups.
+In some cases, multi tenancy for example, namespaces need to be grouped
together.
+Administrators could annotate namespaces which allows dynamic placement of
applications based on multiple annotations if placement rules were setup.
+YuniKorn cannot and does not just add all annotations from a namespace to an
application.
-Users can annotate namespaces which allows dynamic placement of applications
based on the annotation value if proper placement rules are present.
+To help support this grouping case a parent queue can be tagged on a
namespace.
### Configuration
-Apply the following configuration to YuniKorn's configmap:
+The configuration for this functionality consists of two pieces:
+1. the mapping rule
+1. the namespace annotation
-```yaml
-placementrules:
- - name: tag
- value: namespace
- create: true
- parent:
- - name: tag
- value: namespace.parentqueue
-queues:
- - name: root
- queues:
- - name: production
- - name: development
+First we set the following configuration to YuniKorn's configmap:
+```yaml
+partitions:
+ - name: default
+ placementrules:
+ - name: tag
+ value: namespace
+ create: true
+ parent:
+ - name: tag
+ value: namespace.parentqueue
+ queues:
+ - name: root
+ queues:
+ - name: production
+ - name: development
```
-The `namespace.parentqueue` tag is provided by the shim (see next section).
+The configuration used for the namespace to queue mapping is the same as
[above](#Namespace-to-queue-mapping).
+As an extension to the placement rule a `parent` rule is added to support the
grouping.
+The parent rule is used to generate the parent, or the queue above, in the
hierarchy.
+The rule uses the tag `namespace.parentqueue` from the application to generate
the parent queue name.
+The `namespace.parentqueue` tag is automatically added by the Kubernetes shim
but does require a namespace annotation (see below).
+
+In the example rule configuration given the `create` flag is not set on the
parent rule.
+This means that the parent queue must exist in the configuration otherwise the
application submit will fail.
+For the example configuration this means supported values for the parent are
thus limited to `production` and `development`.
-You can do this during the installation by overwriting the configuration in the
-[helm chart
template](https://github.com/apache/incubator-yunikorn-release/blob/724ec82d0d548598e170cc6d5ca6aaae00f8286c/helm-charts/yunikorn/values.yaml#L71-L81).
+Quotas cannot be set on the parent queue using any of these mappings.
+The quota linked to the namespace is set on the namespace queue not the parent
as per the namespace mapping provided earlier.
-### Set up namespaces
-You can configure the following annotation for a Kubernetes namespace in your
cluster: `yunikorn.apache.org/parentqueue`.
+Parent queue quotas must always be set directly in the configuration.
+This requires the `create` flag to be set to `false` on the parent rule.
-E.g. `finance` namespace can be annotated with the following annotation:
+### Namespace parent queue
+Contrary to the namespace name itself, and inline with the quota settings, the
namespaces need to be annotated to use the parent queue mapping.
+Namespace names must be unique in Kubernetes which is not affected by this
annotation.
+The same annotation value may be used for multiple namespaces:
```yaml
yunikorn.apache.org/parentqueue: root.production
```
-Each pod (allocation) created in the namespace will be passed to the scheduler
with the value of that annotation under the `namespace.parentqueue` tag.
+The example annotation above will map the parent queue to the existing
`root.production` queue.
+Note that the rule will fully qualify the name if needed, you can thus omit
the `root.` part in the annotation.
+If the annotation starts with `root.` the system assumes it is a fully
qualified queue name.
+
+To complete the picture here is an image that shows the mapping from
Kubernetes namespaces to queues in YuniKorn.
+It uses the annotations on the namespaces in Kubernetes as described, and the
example configuration for the mapping rules.
+The `finance` and `sales` namespaces become queues grouped under the parent
queue `production`.
+The namespaces `dev` and `test` are placed under the `development` parent
queue.
-This tag can be further used in the tag placement rule to provide the desired
mapping as seen above in the Configuration section.
+
-### Run workloads
-After the configmap has been configured and the namespace has been annotated,
users simply submit applications to the namespace and the placement rule will
take effect.
+### Run a workload
+Applications, and the pods that are part of the application, can be submitted
without specific labels or changes.
+YuniKorn will add the tags, the placement rules will do the rest.
+The configured placement rule will create the queues, if required, and add the
application to the queue.
-Let's say a user submits an application to the `finance` namespace. The
application will be placed onto `root.production.finance` based on the configs
above.
+Since the namespace `finance` is annotated with the example value, and the
rules are in place.
+Applications in the `finance` namespace will run in the
`root.production.finance` queue that is created dynamically.
