You’re right. I sat down yesterday and tested few scenarios. Test results same
as you quoted below.
What are we trying to do?
1. Offer free projects to clients --> no paid or no cost associated. They
can request these projects and create POC apps, test apps, prove ideas etc. We
still schedule these apps on same cluster without any dedicated nodes. we want
to offer best effort QOS and restrict them to create other QO pods/apps
Working solution (need your feedback)
1. Create a best effort quota with # PODS
2. Create a non-best-effort quota with pods = 0 with some limit values
Result: am able to create best-effort pods but no other QOS type pods.
2. Offer non-best-effort projects: Client paying depndign on how much
quota he is asking. They can create guarnatd or burst pods. We also create a
limitrnage object for these projects and set limit defaults ( assume we using
overcomit plug-in so request defualt is no use as it calcualted autoaticaly) .
no best-efffort pods allowed
working solution : (need your feedback)
1. Create quota and limitrnage with limit defaults and max limit
2. Client can start with burst pods and eventully go guaranted if they
really want by putting request and limits same explicitely
3. Unable to create best-effort pods since limitrange with limits (
default and max)
Finally ….
(3) Offer a project with best effort and non-best effort : At this point we
are not sure why we should be abel to offer this type since above 2 satisfy our
requimrents. However exploring how to offer if we really want. Client wont pay
for non-best effort pods but will pay non-best effort quota.
Working solution ( need your feedback)
1. Create a best-effort quota with # PODS
2. Create a non-best-efffort quota with hard limits
3. No limitrange object
In this case, client has to explicitely specific resources{limits}. If
not, that is considered best-effort-pod. If he explicitely specificy then that
could be guaranteed or burst depending request and limits numbers. In this case
we wont’ be abel to enforce limitrange
in all above scanariso we want to leverage overcommit plug-in.
--
Srinivas Kotaru
From: Derek Carr <[email protected]>
Date: Wednesday, November 2, 2016 at 10:14 AM
To: Srinivas Naga Kotaru <[email protected]>
Cc: "[email protected]" <[email protected]>, dev
<[email protected]>
Subject: Re: Quota Policies
Srinivas,
1. Can best-effort quota be extended to all object counts?
No. A quota with scope=BestEffort can only count items that match that scope
per documentation. You can have multiple quotas in a single project/namespace,
so you can have an object-count quota that controls counts on various
resources, and a slack-resources (scope=BestEffort) quota that controls the
number of pods that run BestEffort.
2. If you define only best effort quota with --scope=NotTerminating,BestEffort
with no LimitRange, can client still create non-best effort pods in same
project?
Yes. Clients can create pods that are BestEffort and Terminating, and clients
can create pods that have Burstable/Guaranteed quality of service by setting
requests/limits themselves.
3. If we define only non-best effort quota with
scope=NotBestEffort,NotTerminating with no LimitRange or LimitRange without
defaults, can client deploy BestEffort pods?
with no LimitRange:
The client is able to create BestEffort pods.
with a LimitRange (but no defaults):
Assuming you set min/max constraints for CPU and memory on the LimitRange, you
are not able to create BestEffort pods.
4. If we define both BestEffort and NotBestEffort quota on same project
(without defaults on LimitRange object and use only max), can client deploy
both types of workloads?
No. Clients will be prevented from creating BestEffort pods because the max
constraint from the LimitRange will check will enforce a value to be set. I
appreciate what you are trying to do in this scenario, and I think it's worth
opening an RFE to allow for the option to have a LimitRange be used to enforce
the shape of Burstable/Guaranteed pods, but still let BestEffort pods into the
project. I think we would need that to be an option on the LimitRange resource
in order to preserve backwards compatibility on the existing behavior.
5. What happens if client only mention requests, but not limits in overcommit
scenarios:
without the plugin:
the scheduler will schedule to the request value. on the physical node, the
cpu request will be enforced via CFS shares, but no quota will cap the amount
of CPU that that pod can consume absent contention. with contention, CFS
shares will apportion relative CPU time relative to the request value as
normal. there will be no memory limit enforced on the node via cgroups, so the
pod can burst up to the node capacity. if you enable the node to run
eviction-thresholds for memory.available, the node when observing
MemoryPressure will attempt to evict the pod that is consuming the most memory
relative to the request. If the out of resource monitor in the node is unable
to act in time, the node will OOM, and the oom_killer will target the workload
that is consuming the most memory relative to its request.
with the plugin:
the requests should not be modified, and the limits will remain unbounded as
above.
Thanks,
Derek
On Tue, Nov 1, 2016 at 2:24 PM, Srinivas Naga Kotaru (skotaru)
<[email protected]<mailto:[email protected]>> wrote:
Derek,
Can you shed some light on below questions?
We’re trying to come out with a policy, which satisfy all workloads from a same
project rather creating separate projects for each type of work load. We also
liked overcommit cluster configuration where client don’t have to worry about
requests and focus on what he want at maximum.
1. Can best effort quota can be extended to all objects counts? As per
documentation, it says only applies “pods”. I knew best effort quota definition
doesn’t need compute resources and limitrange object.
https://docs.openshift.com/container-platform/3.3/admin_guide/quota.html#quota-scopes
2. If we define only best effort quota with –scope non-terminated,
best-effort (no limitrange object), can client still would be able to create
non best effort pods on same project? If Yes, that is backdoor to create
non-BestEffort pods by client by explicitly specifying request & limits and
bypass quota he intend?
3. if we define only non-best effort quota with scope – non-best-effort,
non-terminated, without limit range object or with limit range object without
defaults, can client would be able to deploy best effort pods ? if yes that is
backdoor again to run multiple non best effort pods although he intended to
run just guaranteed pods.
4. if we define both best-effort and non-effort quota on same project
(without defaults on limitrange object and use only max), can client would be
able to deploy 2 different workloads and pods? Answer is yes as per our
previous discussion but here from any ramifications.
5. What happens if client only mention requests but not limits in over
commit plug-in model (or without plug-in scenario (assume we use overcommit
cluster plug-in)
--
Srinivas Kotaru
From: Derek Carr <[email protected]<mailto:[email protected]>>
Date: Thursday, October 27, 2016 at 2:45 PM
To: Srinivas Naga Kotaru <[email protected]<mailto:[email protected]>>
Cc: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>, dev
<[email protected]<mailto:[email protected]>>
Subject: Re: Quota Policies
Your understanding is correct, but one caveat.
This config doesn’t alter or increase limit numbers put the developers
This is true UNLESS you set limitCPUToMemoryPercent. In that case, the only
value a user sets is memory limits.
In a nutshell, the idea behind the cluster resource override is users should
only think about the limits for cpu/memory and not think about the request at
all (since the operator is taking that responsibility).
Thanks,
Derek
On Thu, Oct 27, 2016 at 5:13 PM, Srinivas Naga Kotaru (skotaru)
<[email protected]<mailto:[email protected]>> wrote:
Derek
We have separate project for non-prod & prod.
I fully understood the example you quoted. It Is very clear. Would be nice if
someone paste this explanation with example to the overcommit documentation.
In summary:
This config only applicable to pods which have explicit request or limit or
both (via using limitrange/defaults)
This overcommit ratio apply to entire cluster/projects who satisfy above
requirement
This is cluster administrator explicitly controlling the overcommit and
overriding what development teams put on request #
This config doesn’t alter or increase limit numbers put the developers
Is above my understanding is correct?
--
Srinivas Kotaru
From: Derek Carr <[email protected]<mailto:[email protected]>>
Date: Thursday, October 27, 2016 at 1:07 PM
To: Srinivas Naga Kotaru <[email protected]<mailto:[email protected]>>
Cc: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>, dev
<[email protected]<mailto:[email protected]>>
Subject: Re: Quota Policies
Do you plan to manage non-prod apps in the same project(s) as prod-apps?
I will describe the ClusterResourceOverride behavior via an example, but it is
basically a giant hammer you can enable on the cluster that lets an
administrator set a cluster-wide over-commit target which projects may
opt-in/out from being utilized via annotation.
If a project opts into the behavior, all incoming pods will be modified based
on the configuration.
Sample Scenario: A project opts into the ClusterResourceOverride and it has no
LimitRange defined
$ kubectl run best-effort-pods --image=nginx
The resulting pod will still have no resource requirements made (the plug-in
has no impact).
$ kubectl run pods-with-resources --image=nginx --limits=cpu=1,memory=1Gi
Traditionally, this pod would have Guaranteed quality of service and both the
request and limit value would be cpu=1 and memory=1Gi.
But let's see what happens if you enable the overriding behavior on this
project using the following config:
memoryRequestToLimitPercent: 25
cpuRequestToLimitPercent: 25
limitCPUToMemoryPercent: 200
The pod ends up with the following:
requests.cpu=500m
limits.cpu=2
requests.memory=256Mi
limits.memory=1Gi
As you can see, the only value that had meaning from the end-user was the
memory limit, but all other values were tuned relative to that value. The
memory request was tuned down to 25% of the the limit. The cpu limit was tuned
up to 2 cores because it was set to 200% of the memory limit where 1Gi =1 core
in that conversion. Finally, the cpu request was tuned down to 25% of the
limit to 500m.
If we remove the limitCPUToMemoryPercent setting, and use the following
configuration:
memoryRequestToLimitPercent: 25
cpuRequestToLimitPercent: 25
The pod ends up with the following:
requests.cpu=250m
limits.cpu=1
requests.memory=256Mi
limits.memory=1Gi
In this case, you can see the limit was respected from the user, but the
requests were tuned down to meet the desired overcommit. In effect, it is only
possible to run BestEffort/Burstable pods but not Guaranteed pods with this
configuration on in a project.
Thanks,
Derek
On Thu, Oct 27, 2016 at 2:32 PM, Srinivas Naga Kotaru (skotaru)
<[email protected]<mailto:[email protected]>> wrote:
Derek
Thanks for helping so far. It is not clear how quota & QOS works. We are
planning ot use BestEffort for non-prod apps and non-BestEffort for prod
applications. This has some side effect and app teams might complain that their
application experience is not same as non-prod behaves different then prod when
they testing release and monitoring performances. We need to think about it how
to mitigate these challenges
I was reading below link and this is pretty good.
https://docs.openshift.com/container-platform/3.3/admin_guide/overcommit.html
didn’t understand Configuring Masters for Overcommitment and its example. Can
you breif how this overcommitment works in the scanarios we talked about?
BestEffort, Burst, and Guarnted ..
memoryRequestToLimitPercent: 25
cpuRequestToLimitPercent: 25
limitCPUToMemoryPercent: 200
would be glad if you explain with simple examples… I’m trying to understand how
this overcommit helps platform admisn to tune better.
--
Srinivas Kotaru
From: Derek Carr <[email protected]<mailto:[email protected]>>
Date: Wednesday, October 26, 2016 at 1:23 PM
To: Srinivas Naga Kotaru <[email protected]<mailto:[email protected]>>
Cc: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>, dev
<[email protected]<mailto:[email protected]>>
Subject: Re: Quota Policies
A BestEffort pod is a pod whose pod.spec.containers[x].resources.requests and
pod.spec.containers[x].resources.limits are empty so your understanding is
correct.
If you want to have a project that supports both BestEffort and NotBestEffort
pods together, you can do that and control usage via ResourceQuota using the
examples I provided.
If you want to have a project that supports both BestEffort and NotBestEffort
pods together, and use LimitRange to enforce min/max constraints and default
resource requirements, you will encounter problems.
1. The LimitRange will assign default resources to each BestEffort pod you
submit (making them no longer BestEffort) or
2. It will require that each pod have a cpu or memory value specified as
part of its validation (if you configured it as such)
Thanks,
Derek
On Wed, Oct 26, 2016 at 2:54 PM, Srinivas Naga Kotaru (skotaru)
<[email protected]<mailto:[email protected]>> wrote:
Can u answer this question? Trying to understand how do we call BestEffort pods
in terms of quota/limtrange/pod definitions perceptive?
My understand is, a pod is called besteffort pod, it it does not have any quota
defination without compute resources ( limit or request) and it doesn’t have
any explicit request and limit in pod defiantion. Is It my understanding is
correct?
--
Srinivas Kotaru
From: Srinivas Naga Kotaru <[email protected]<mailto:[email protected]>>
Date: Tuesday, October 25, 2016 at 3:42 PM
To: Derek Carr <[email protected]<mailto:[email protected]>>
Cc: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>, dev
<[email protected]<mailto:[email protected]>>
Subject: Re: Quota Policies
This is good. I’m getting enough details to craft my policies.
In case of 1st example (BestEffort), we don’t have to create any limitrange
with default request and limits? Or quota definition without having any
request.cpu, request.memory, limit.cpu and limit.memory?
Am trying to understand what exactly it means by BestEffort when it comes to
quota, limitrange, pod definitions perceptive. Is it just an arbitrary word or
a pod is called as BestEffort if it doesn’t have request, limits in its
definition?
--
Srinivas Kotaru
From: Derek Carr <[email protected]<mailto:[email protected]>>
Date: Tuesday, October 25, 2016 at 2:26 PM
To: Srinivas Naga Kotaru <[email protected]<mailto:[email protected]>>
Cc: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>, dev
<[email protected]<mailto:[email protected]>>
Subject: Re: Quota Policies
Sorry, the command is the following (missed scopes on second):
$ kubectl create quota best-effort-not-terminating --hard=pods=5
--scopes=NotTerminating,BestEffort
$ kubectl create quota not-best-effort-not-terminating
--hard=requests.cpu=5,requests.memory=10Gi,limits.cpu=10,limits.memory=20Gi
--scopes=NotTerminating,NotBestEffort
On Tue, Oct 25, 2016 at 5:25 PM, Derek Carr
<[email protected]<mailto:[email protected]>> wrote:
If you only want to quota pods that have a more permanent footprint on the
node, then create a quota that only matches on the NotTerminating scope.
If you want to allow usage of slack resources (i.e. run BestEffort pods), and
define a quota that controls otherwise, create 2 quotas.
$ kubectl create quota best-effort-not-terminating --hard=pods=5
--scopes=NotTerminating,BestEffort
$ kubectl create quota not-best-effort-not-terminating
--hard=requests.cpu=5,requests.memory=10Gi,limits.cpu=10,limits.memory=20Gi
So in this example:
1. the user is able to create 5 long running pods that make no resource request
(i.e. no cpu, memory specified)
2. the user to request up to 5 cpu cores and 10Gi memory for scheduling
purposes, and the node will work to ensure is available
3. are able to burst up to 10 cpu cores, and 20Gi memory based on node-local
conditions
Thanks,
Derek
On Tue, Oct 25, 2016 at 5:14 PM, Srinivas Naga Kotaru (skotaru)
<[email protected]<mailto:[email protected]>> wrote:
Derek/Clayton
I saw this link yesterday. It was really good and helpful; I didn’t understand
the last advanced section. Let me spend some time again.
@Clayton: Do we need to create separate quota policies for both terminated and
non-terminated ? or just creating a single policy for non-terminated would be
enough? Want to be simple but at same time, don’t want non-terminated short
lived pods don’t create any issues to regular working pods.
--
Srinivas Kotaru
From: Derek Carr <[email protected]<mailto:[email protected]>>
Date: Tuesday, October 25, 2016 at 1:09 PM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Cc: Srinivas Naga Kotaru <[email protected]<mailto:[email protected]>>, dev
<[email protected]<mailto:[email protected]>>
Subject: Re: Quota Policies
You may find this document useful:
http://kubernetes.io/docs/admin/resourcequota/walkthrough/
>BestEffort or NotBestEffort are used to explain the concept or can Pod
>definition can have these words?
This refers to the quality of service for a pod. If a container in a pod makes
no request/limit for compute resources, it is BestEffort. If it makes a
request for any resource, its NotBestEffort.
You can apply a quota to control the number of BestEffort pods you can create
separate from the number of NotBestEffort pods.
See step 5 in the above linked example for a walkthrough.
Thanks,
Derek
On Tue, Oct 25, 2016 at 4:02 PM, Clayton Coleman
<[email protected]<mailto:[email protected]>> wrote:
On Tue, Oct 25, 2016 at 3:55 PM, Srinivas Naga Kotaru (skotaru)
<[email protected]<mailto:[email protected]>> wrote:
Hi
I’m trying to frame a policy for best usage of compute resources for our
environment. I stared reading documentation on this topic. Although
documentation is pretty limited on this topic with working examples, now I have
some better understanding on quota and limtrange objects.
We are planning to enforce quota and limtrange on every project as part of
project provision. Client can increase these limits by going to modify screen
on our system and pay the cost accordingly. Goal is to have high efficient
cluster resource usage and minimal client disturbance.
Have few questions around implementation?
Can we exclude build, deploy like short time span pods from quota restrictions?
There are two quotas - one for terminating pods (pods that are guaranteed to
finish in a certain time period) and one for non-terminating pods.
Quotas enforced only running pods or dead pods, pending status, succeeded?
Once a pod terminates (failed, succeeded) it is not counted for quota. Pods
that are pending deletion are still counted for quota.
What is the meaning of scopes: Terminating or scopes: NotTerminating in quota
definition? It is bit confusing to understand.
Terminating means "will finish in bounded time", i.e. does not have
RestartAlways and also has activeDeadlineSeconds. NonTerminating is everything
else.
BestEffort or NotBestEffort are used to explain the concept or can Pod
definition can have these words?
We don't have quota per QoS class yet today, but it would be useful.
Any good documentation with examples would help in documentation.
I thought Derek had some good write ups of this.
Srinivas Kotaru
--
Srinivas Kotaru
_______________________________________________
dev mailing list
[email protected]<mailto:[email protected]>
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
_______________________________________________
dev mailing list
[email protected]<mailto:[email protected]>
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
_______________________________________________
dev mailing list
[email protected]
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
