Srinivas, Thanks for the details, let me offer my comments for each use case.
1. Offer free projects to clients at no paid or no cost associated. They can request projects and create POC apps, test apps, etc. Restrict to BestEffort pods only. $ kubectl create quota slack-resources --scope=BestEffort --hard=pods=X $ kubectl create quota qos-resources --scope=NotBestEffort --hard=pods=0,requests.cpu=0,requests.memory=0,... The above should work, I would share the following caveats: - what percentage of your workload does this represent? - what is your target node utilization for memory? - strongly encourage you to take advantage of out of resource handling on the node for memory based on your utilization target ( https://docs.openshift.org/latest/admin_guide/out_of_resource_handling.html ) - absent utilization aware scheduling in the future, your scheduling decisions for these pods by their nature may not be 'perfect' by definition. - I assume your handling storage as a separate concern? Or are you restricting PersistentVolumeClaims as well? 2. Offer projects that run pods with Burstable/Guaranteed quality of service. I think this has a problem, but it could be there was a misunderstanding on my part. The project in this scenario: - has a LimitRange that specifies - default resource requests? - default resource limits? - both of the above? - enforces a max constraint - has the ClusterResourceOverride plug-in active with your cluster-wide configuration By default, this would mean clients send pods with limits, and they are placed in the Burstable tier as defined in the product by default since request < limits. If you do not have default resource limits on your LimitRange, then a client could send pods with requests specified and leave limits unbounded. The ClusterResourceOverride plug-in would not modify the pod resource requirements at all. The pod by definition of the OpenShift product would have Burstable QOS, the pod's local memory limit is not enforced via a cgroup, but the eviction monitor on the node (if enabled) would not evict that pod as long as it does not exceed its scheduling request in active memory usage. That would be the strongest guarantee you could provide, but it is slightly different than Guaranteed as defined in Kubernetes/OpenShift since the cgroup memory limit in that context is enforced that prevents users from exceeding their usage. In this environment, I will still recommend you run with eviction thresholds enabled on your nodes. 3. Offer a project with both style workloads Your summary looks good to me. Thanks, Derek On Wed, Nov 2, 2016 at 3:11 PM, Srinivas Naga Kotaru (skotaru) < [email protected]> wrote: > You’re right. I sat down yesterday and tested few scenarios. Test results > same as you quoted below. > > > > What are we trying to do? > > > > 1. Offer free projects to clients à no paid or no cost associated. > They can request these projects and create POC apps, test apps, prove ideas > etc. We still schedule these apps on same cluster without any dedicated > nodes. we want to offer best effort QOS and restrict them to create other > QO pods/apps > > > > Working solution (need your feedback) > > 1. Create a best effort quota with # PODS > > 2. Create a non-best-effort quota with pods = 0 with some limit > values > > Result: am able to create best-effort pods but no other QOS type > pods. > > > > 2. Offer non-best-effort projects: Client paying depndign on how > much quota he is asking. They can create guarnatd or burst pods. We also > create a limitrnage object for these projects and set limit defaults ( > assume we using overcomit plug-in so request defualt is no use as it > calcualted autoaticaly) . no best-efffort pods allowed > > working solution : (need your feedback) > > 1. Create quota and limitrnage with limit defaults and max limit > > 2. Client can start with burst pods and eventully go guaranted if > they really want by putting request and limits same explicitely > > 3. Unable to create best-effort pods since limitrange with limits ( > default and max) > > > > > > Finally …. > > > > (3) Offer a project with best effort and non-best effort : At this > point we are not sure why we should be abel to offer this type since above > 2 satisfy our requimrents. However exploring how to offer if we really > want. Client wont pay for non-best effort pods but will pay non-best effort > quota. > > > > Working solution ( need your feedback) > > > > 1. Create a best-effort quota with # PODS > > 2. Create a non-best-efffort quota with hard limits > > 3. No limitrange object > > > > In this case, client has to explicitely specific > resources{limits}. If not, that is considered best-effort-pod. If he > explicitely specificy then that could be guaranteed or burst depending > request and limits numbers. In this case we wont’ be abel to enforce > limitrange > > > > in all above scanariso we want to leverage overcommit plug-in. > > > > > > -- > > *Srinivas Kotaru* > > > > *From: *Derek Carr <[email protected]> > *Date: *Wednesday, November 2, 2016 at 10:14 AM > > *To: *Srinivas Naga Kotaru <[email protected]> > *Cc: *"[email protected]" <[email protected]>, dev < > [email protected]> > *Subject: *Re: Quota Policies > > > > Srinivas, > > 1. Can best-effort quota be extended to all object counts? > > > No. A quota with scope=BestEffort can only count items that match that > scope per documentation. You can have multiple quotas in a single > project/namespace, so you can have an object-count quota that controls > counts on various resources, and a slack-resources (scope=BestEffort) quota > that controls the number of pods that run BestEffort. > > 2. If you define only best effort quota with > --scope=NotTerminating,BestEffort > with no LimitRange, can client still create non-best effort pods in same > project? > > Yes. Clients can create pods that are BestEffort and Terminating, and > clients can create pods that have Burstable/Guaranteed quality of service > by setting requests/limits themselves. > > 3. If we define only non-best effort quota with > scope=NotBestEffort,NotTerminating > with no LimitRange or LimitRange without defaults, can client deploy > BestEffort pods? > > with no LimitRange: > > The client is able to create BestEffort pods. > > with a LimitRange (but no defaults): > > Assuming you set min/max constraints for CPU and memory on the LimitRange, > you are not able to create BestEffort pods. > > > > 4. If we define both BestEffort and NotBestEffort quota on same project > (without defaults on LimitRange object and use only max), can client deploy > both types of workloads? > > No. Clients will be prevented from creating BestEffort pods because the > max constraint from the LimitRange will check will enforce a value to be > set. I appreciate what you are trying to do in this scenario, and I think > it's worth opening an RFE to allow for the option to have a LimitRange be > used to enforce the shape of Burstable/Guaranteed pods, but still let > BestEffort pods into the project. I think we would need that to be an > option on the LimitRange resource in order to preserve backwards > compatibility on the existing behavior. > > 5. What happens if client only mention requests, but not limits in > overcommit scenarios: > > without the plugin: > > the scheduler will schedule to the request value. on the physical node, > the cpu request will be enforced via CFS shares, but no quota will cap the > amount of CPU that that pod can consume absent contention. with > contention, CFS shares will apportion relative CPU time relative to the > request value as normal. there will be no memory limit enforced on the > node via cgroups, so the pod can burst up to the node capacity. if you > enable the node to run eviction-thresholds for memory.available, the node > when observing MemoryPressure will attempt to evict the pod that is > consuming the most memory relative to the request. If the out of resource > monitor in the node is unable to act in time, the node will OOM, and the > oom_killer will target the workload that is consuming the most memory > relative to its request. > > with the plugin: > > the requests should not be modified, and the limits will remain unbounded > as above. > > > > Thanks, > > Derek > > > > On Tue, Nov 1, 2016 at 2:24 PM, Srinivas Naga Kotaru (skotaru) < > [email protected]> wrote: > > Derek, > > > > Can you shed some light on below questions? > > > > We’re trying to come out with a policy, which satisfy all workloads from a > same project rather creating separate projects for each type of work load. > We also liked overcommit cluster configuration where client don’t have to > worry about requests and focus on what he want at maximum. > > > > 1. Can best effort quota can be extended to all objects counts? As > per documentation, it says only applies “pods”. I knew best effort quota > definition doesn’t need compute resources and limitrange object. > > https://docs.openshift.com/container-platform/3.3/admin_ > guide/quota.html#quota-scopes > > > > 2. If we define only best effort quota with –scope non-terminated, > best-effort (no limitrange object), can client still would be able to > create non best effort pods on same project? If Yes, that is backdoor to > create non-BestEffort pods by client by explicitly specifying request & > limits and bypass quota he intend? > > 3. if we define only non-best effort quota with scope – > non-best-effort, non-terminated, without limit range object or with limit > range object without defaults, can client would be able to deploy best > effort pods ? if yes that is backdoor again to run multiple non best > effort pods although he intended to run just guaranteed pods. > > 4. if we define both best-effort and non-effort quota on same > project (without defaults on limitrange object and use only max), can > client would be able to deploy 2 different workloads and pods? Answer is > yes as per our previous discussion but here from any ramifications. > > 5. What happens if client only mention requests but not limits in > over commit plug-in model (or without plug-in scenario (assume we use > overcommit cluster plug-in) > > > > -- > > *Srinivas Kotaru* > > > > *From: *Derek Carr <[email protected]> > *Date: *Thursday, October 27, 2016 at 2:45 PM > > > *To: *Srinivas Naga Kotaru <[email protected]> > *Cc: *"[email protected]" <[email protected]>, dev < > [email protected]> > *Subject: *Re: Quota Policies > > > > Your understanding is correct, but one caveat. > > This config doesn’t alter or increase limit numbers put the developers > > > > This is true UNLESS you set limitCPUToMemoryPercent. In that case, the > only value a user sets is memory limits. > > In a nutshell, the idea behind the cluster resource override is users > should only think about the limits for cpu/memory and not think about the > request at all (since the operator is taking that responsibility). > > > > Thanks, > > Derek > > > > > > On Thu, Oct 27, 2016 at 5:13 PM, Srinivas Naga Kotaru (skotaru) < > [email protected]> wrote: > > Derek > > > > We have separate project for non-prod & prod. > > > > I fully understood the example you quoted. It Is very clear. Would be nice > if someone paste this explanation with example to the overcommit > documentation. > > > > In summary: > > > > This config only applicable to pods which have explicit request or limit > or both (via using limitrange/defaults) > > This overcommit ratio apply to entire cluster/projects who satisfy above > requirement > > This is cluster administrator explicitly controlling the overcommit and > overriding what development teams put on request # > > This config doesn’t alter or increase limit numbers put the developers > > > > Is above my understanding is correct? > > > > -- > > *Srinivas Kotaru* > > > > *From: *Derek Carr <[email protected]> > *Date: *Thursday, October 27, 2016 at 1:07 PM > > > *To: *Srinivas Naga Kotaru <[email protected]> > *Cc: *"[email protected]" <[email protected]>, dev < > [email protected]> > *Subject: *Re: Quota Policies > > > > Do you plan to manage non-prod apps in the same project(s) as prod-apps? > > I will describe the ClusterResourceOverride behavior via an example, but > it is basically a giant hammer you can enable on the cluster that lets an > administrator set a cluster-wide over-commit target which projects may > opt-in/out from being utilized via annotation. > > If a project opts into the behavior, all incoming pods will be modified > based on the configuration. > > Sample Scenario: A project opts into the ClusterResourceOverride and it > has no LimitRange defined > > > $ kubectl run best-effort-pods --image=nginx > > The resulting pod will still have no resource requirements made (the > plug-in has no impact). > > $ kubectl run pods-with-resources --image=nginx --limits=cpu=1,memory=1Gi > > Traditionally, this pod would have Guaranteed quality of service and both > the request and limit value would be cpu=1 and memory=1Gi. > > But let's see what happens if you enable the overriding behavior on this > project using the following config: > > memoryRequestToLimitPercent: 25 > > cpuRequestToLimitPercent: 25 > > limitCPUToMemoryPercent: 200 > > The pod ends up with the following: > > > > requests.cpu=500m > > limits.cpu=2 > > requests.memory=256Mi > > limits.memory=1Gi > > As you can see, the only value that had meaning from the end-user was the > memory limit, but all other values were tuned relative to that value. The > memory request was tuned down to 25% of the the limit. The cpu limit was > tuned up to 2 cores because it was set to 200% of the memory limit where > 1Gi =1 core in that conversion. Finally, the cpu request was tuned down to > 25% of the limit to 500m. > > If we remove the limitCPUToMemoryPercent setting, and use the following > configuration: > > > > memoryRequestToLimitPercent: 25 > > cpuRequestToLimitPercent: 25 > > > > The pod ends up with the following: > > > > requests.cpu=250m > > limits.cpu=1 > > requests.memory=256Mi > > limits.memory=1Gi > > In this case, you can see the limit was respected from the user, but the > requests were tuned down to meet the desired overcommit. In effect, it is > only possible to run BestEffort/Burstable pods but not Guaranteed pods with > this configuration on in a project. > > > > Thanks, > > Derek > > > > > > > > > > > > > > > > > > > > On Thu, Oct 27, 2016 at 2:32 PM, Srinivas Naga Kotaru (skotaru) < > [email protected]> wrote: > > Derek > > > > Thanks for helping so far. It is not clear how quota & QOS works. We are > planning ot use BestEffort for non-prod apps and non-BestEffort for prod > applications. This has some side effect and app teams might complain that > their application experience is not same as non-prod behaves different then > prod when they testing release and monitoring performances. We need to > think about it how to mitigate these challenges > > > > I was reading below link and this is pretty good. > > > > https://docs.openshift.com/container-platform/3.3/admin_ > guide/overcommit.html > > > > didn’t understand Configuring Masters for Overcommitment and its example. > Can you breif how this overcommitment works in the scanarios we talked > about? BestEffort, Burst, and Guarnted .. > > > > memoryRequestToLimitPercent: 25 > > cpuRequestToLimitPercent: 25 > > limitCPUToMemoryPercent: 200 > > > > would be glad if you explain with simple examples… I’m trying to > understand how this overcommit helps platform admisn to tune better. > > > > -- > > *Srinivas Kotaru* > > > > *From: *Derek Carr <[email protected]> > *Date: *Wednesday, October 26, 2016 at 1:23 PM > > > *To: *Srinivas Naga Kotaru <[email protected]> > *Cc: *"[email protected]" <[email protected]>, dev < > [email protected]> > *Subject: *Re: Quota Policies > > > > A BestEffort pod is a pod whose pod.spec.containers[x].resources.requests > and pod.spec.containers[x].resources.limits are empty so your > understanding is correct. > > If you want to have a project that supports both BestEffort and > NotBestEffort pods together, you can do that and control usage via > ResourceQuota using the examples I provided. > > If you want to have a project that supports both BestEffort and > NotBestEffort pods together, and use LimitRange to enforce min/max > constraints and default resource requirements, you will encounter problems. > > 1. The LimitRange will assign default resources to each BestEffort pod > you submit (making them no longer BestEffort) or > 2. It will require that each pod have a cpu or memory value specified > as part of its validation (if you configured it as such) > > Thanks, > > Derek > > > > > > > > On Wed, Oct 26, 2016 at 2:54 PM, Srinivas Naga Kotaru (skotaru) < > [email protected]> wrote: > > Can u answer this question? Trying to understand how do we call BestEffort > pods in terms of quota/limtrange/pod definitions perceptive? > > > > My understand is, a pod is called besteffort pod, it it does not have any > quota defination without compute resources ( limit or request) and it > doesn’t have any explicit request and limit in pod defiantion. Is It my > understanding is correct? > > > > -- > > *Srinivas Kotaru* > > > > *From: *Srinivas Naga Kotaru <[email protected]> > *Date: *Tuesday, October 25, 2016 at 3:42 PM > *To: *Derek Carr <[email protected]> > > > *Cc: *"[email protected]" <[email protected]>, dev < > [email protected]> > *Subject: *Re: Quota Policies > > > > This is good. I’m getting enough details to craft my policies. > > > > In case of 1st example (BestEffort), we don’t have to create any > limitrange with default request and limits? Or quota definition without > having any request.cpu, request.memory, limit.cpu and limit.memory? > > > > Am trying to understand what exactly it means by BestEffort when it comes > to quota, limitrange, pod definitions perceptive. Is it just an arbitrary > word or a pod is called as BestEffort if it doesn’t have request, limits in > its definition? > > > > -- > > *Srinivas Kotaru* > > > > *From: *Derek Carr <[email protected]> > *Date: *Tuesday, October 25, 2016 at 2:26 PM > *To: *Srinivas Naga Kotaru <[email protected]> > *Cc: *"[email protected]" <[email protected]>, dev < > [email protected]> > *Subject: *Re: Quota Policies > > > > Sorry, the command is the following (missed scopes on second): > > $ kubectl create quota best-effort-not-terminating --hard=pods=5 > --scopes=NotTerminating,BestEffort > $ kubectl create quota not-best-effort-not-terminating > --hard=requests.cpu=5,requests.memory=10Gi,limits.cpu=10,limits.memory=20Gi > --scopes=NotTerminating,NotBestEffort > > > > On Tue, Oct 25, 2016 at 5:25 PM, Derek Carr <[email protected]> wrote: > > If you only want to quota pods that have a more permanent footprint on the > node, then create a quota that only matches on the NotTerminating scope. > > If you want to allow usage of slack resources (i.e. run BestEffort pods), > and define a quota that controls otherwise, create 2 quotas. > > $ kubectl create quota best-effort-not-terminating --hard=pods=5 > --scopes=NotTerminating,BestEffort > > $ kubectl create quota not-best-effort-not-terminating > --hard=requests.cpu=5,requests.memory=10Gi,limits. > cpu=10,limits.memory=20Gi > > So in this example: > > > > 1. the user is able to create 5 long running pods that make no resource > request (i.e. no cpu, memory specified) > > 2. the user to request up to 5 cpu cores and 10Gi memory for scheduling > purposes, and the node will work to ensure is available > > 3. are able to burst up to 10 cpu cores, and 20Gi memory based on > node-local conditions > > > > Thanks, > > Derek > > > > On Tue, Oct 25, 2016 at 5:14 PM, Srinivas Naga Kotaru (skotaru) < > [email protected]> wrote: > > Derek/Clayton > > > > I saw this link yesterday. It was really good and helpful; I didn’t > understand the last advanced section. Let me spend some time again. > > > > @Clayton: Do we need to create separate quota policies for both terminated > and non-terminated ? or just creating a single policy for non-terminated > would be enough? Want to be simple but at same time, don’t want > non-terminated short lived pods don’t create any issues to regular working > pods. > > > > -- > > *Srinivas Kotaru* > > > > *From: *Derek Carr <[email protected]> > *Date: *Tuesday, October 25, 2016 at 1:09 PM > *To: *"[email protected]" <[email protected]> > *Cc: *Srinivas Naga Kotaru <[email protected]>, dev < > [email protected]> > *Subject: *Re: Quota Policies > > > > You may find this document useful: > http://kubernetes.io/docs/admin/resourcequota/walkthrough/ > > > >BestEffort or NotBestEffort are used to explain the concept or can Pod > definition can have these words? > > This refers to the quality of service for a pod. If a container in a pod > makes no request/limit for compute resources, it is BestEffort. If it > makes a request for any resource, its NotBestEffort. > > You can apply a quota to control the number of BestEffort pods you can > create separate from the number of NotBestEffort pods. > > See step 5 in the above linked example for a walkthrough. > > Thanks, > > Derek > > > > > > > > > > On Tue, Oct 25, 2016 at 4:02 PM, Clayton Coleman <[email protected]> > wrote: > > > > > > On Tue, Oct 25, 2016 at 3:55 PM, Srinivas Naga Kotaru (skotaru) < > [email protected]> wrote: > > Hi > > > > I’m trying to frame a policy for best usage of compute resources for our > environment. I stared reading documentation on this topic. Although > documentation is pretty limited on this topic with working examples, now I > have some better understanding on quota and limtrange objects. > > > > We are planning to enforce quota and limtrange on every project as part of > project provision. Client can increase these limits by going to modify > screen on our system and pay the cost accordingly. Goal is to have high > efficient cluster resource usage and minimal client disturbance. > > > > Have few questions around implementation? > > > > Can we exclude build, deploy like short time span pods from quota > restrictions? > > > > There are two quotas - one for terminating pods (pods that are guaranteed > to finish in a certain time period) and one for non-terminating pods. > > > > Quotas enforced only running pods or dead pods, pending status, succeeded? > > > > Once a pod terminates (failed, succeeded) it is not counted for quota. > Pods that are pending deletion are still counted for quota. > > > > What is the meaning of scopes: Terminating or scopes: NotTerminating in > quota definition? It is bit confusing to understand. > > > > Terminating means "will finish in bounded time", i.e. does not have > RestartAlways and also has activeDeadlineSeconds. NonTerminating is > everything else. > > > > BestEffort or NotBestEffort are used to explain the concept or can Pod > definition can have these words? > > > > We don't have quota per QoS class yet today, but it would be useful. > > > > > > Any good documentation with examples would help in documentation. > > > > I thought Derek had some good write ups of this. > > > > > > Srinivas Kotaru > > > > > > -- > > *Srinivas Kotaru* > > > _______________________________________________ > dev mailing list > [email protected] > http://lists.openshift.redhat.com/openshiftmm/listinfo/dev > > > > > _______________________________________________ > dev mailing list > [email protected] > http://lists.openshift.redhat.com/openshiftmm/listinfo/dev > > > > > > > > > > > > > > >
_______________________________________________ dev mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
