The quota controller observes pod deletions, and queues the pertinent 
quotas for processing in response.  The replenishment should be relatively 
quick now.

We actually exercise this behavior in e2e runs.

   1. What version of Kubernetes are you running?
   2. How many quotas?
   3. How many pods?

My guess is that there is a conflict of some kind during the quota sync, 
and the 5 minutes you are seeing is a full re-sync interval for the quota 
controller.

It would be good to try to determine what the conflict is so we can fix.

If you have a quick set of steps I can run to reproduce that would be great.

I recommend opening an issue with the above details and cc @derekwaynecarr

Thanks,
Derek

On Wednesday, October 12, 2016 at 3:18:07 PM UTC-4, David Eads wrote:
>
> Quota reconcilation of pods was supposed to run at a faster rate.  I'm not 
> sure if 5 minutes is that faster rate.  
>
> *Derek:* Do you recall where and how the fast pod replenishment is?
>
>
> On Wednesday, October 12, 2016 at 1:28:45 PM UTC-4, Daniel Smith wrote:
>>
>> Ah, you're blocked because the quota check reconciles slowly. The quick 
>> fix is probably to just get more quota.
>>
>> +David who may know of an already existing issue about this.
>>
>> On Wed, Oct 12, 2016 at 5:39 AM, Diego Rodríguez Rodríguez <
>> diego...@gmail.com> wrote:
>>
>>> I have already configured my cluster to test what you have stated. What 
>>> I have done so far is to create a ResourceQuota which takes care that there 
>>> will not be more than 4 pods running. Then I ask for say 20 jobs to be 
>>> completed.
>>>
>>> What happens in reality is that the first 4 jobs are completed and then, 
>>> even though the pods are completed and therefore resources are available, 
>>> it takes some minutes (around 4-5 minutes) to have another 4 jobs being 
>>> completed. Indeed, what you said is true, however it is no practical 
>>> because a delay of minutes can not be assumed.
>>> The waiting jobs events look like this:
>>> `FailedCreate    Error creating: pods 
>>> "d59fa9b2-6c6b-4dc5-b149-7f89b35421bf-10-" is forbidden: Exceeded quota: 
>>> compute-resources, requested: pods=1, used: pods=4, limited: pods=4`
>>> So, it fails because there are no resources but it is trying it again 
>>> only after some minutes. This behavior is far from the desired one, which 
>>> is relaying on Kubernetes for the execution of a set of tasks no matter the 
>>> resources available, just getting them executed as soon as possible.
>>>
>>> thanks
>>>
>>> On Monday, 10 October 2016 19:24:46 UTC+2, Daniel Smith wrote:
>>>>
>>>> If the system lacks resources, Pods will remain "Pending" until 
>>>> resources become available. Cluster scalers may use pending pods as a 
>>>> signal that the cluster size should be increased.
>>>>
>>>> On Mon, Oct 10, 2016 at 5:58 AM, Diego Rodríguez Rodríguez <
>>>> diego...@gmail.com> wrote:
>>>>
>>>>> Hello, I have a doubt about how Kubernetes' jobs are handled.
>>>>>
>>>>> I have a queue to submit certain amount of incoming tasks (Celery and 
>>>>> RabbitMQ take care of this). Each one of this tasks is, in fact, a 
>>>>> *workflow* which will be executed in a worker (Celery worker with a 
>>>>> DAG executor inside). Each step of the *workflow* is a Docker image 
>>>>> with an input file and an output file.
>>>>>
>>>>> My question is, if I submit jobs from the workflow engine directly to 
>>>>> the Kubernetes API, what happens I at some point there are no more 
>>>>> resources? Will the remaining tasks be kept or will they be lost? My goal 
>>>>> is to treat Kubernetes' jobs as a black box to submit works to. This 
>>>>> works 
>>>>> are of a very different and heterogeneous nature and I wouldn't need to 
>>>>> bother with what is inside them because they are dockerized and executed 
>>>>> by 
>>>>> Kubernetes at some point.
>>>>>
>>>>> To sum up, I already have the layer of Celery workers with a DAG 
>>>>> executor inside which knows the right order of the tasks and knows how to 
>>>>> manage everything concerning the *workflow*. These components will 
>>>>> submit jobs (through Kubernetes API) and will wait for them to be 
>>>>> executed 
>>>>> and then continue with the remaining tasks asking Kubernetes to run them 
>>>>> until the *workflow* ends.
>>>>>
>>>>> I have read about a somehow related issue in Github:
>>>>> https://github.com/kubernetes/kubernetes/pull/17787
>>>>> https://github.com/kubernetes/kubernetes/pull/18827
>>>>>
>>>>> I couldn't determine if this is closed or it is coming in a future 
>>>>> release.
>>>>>
>>>>>
>>>>> Thanks in advance!
>>>>>
>>>>> -- 
>>>>> You received this message because you are subscribed to the Google 
>>>>> Groups "Kubernetes user discussion and Q&A" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send 
>>>>> an email to kubernetes-use...@googlegroups.com.
>>>>> To post to this group, send email to kubernet...@googlegroups.com.
>>>>> Visit this group at https://groups.google.com/group/kubernetes-users.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Kubernetes user discussion and Q&A" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to kubernetes-use...@googlegroups.com.
>>> To post to this group, send email to kubernet...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/kubernetes-users.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.
  • [kubernetes-user... Diego Rodríguez Rodríguez
    • Re: [kubern... 'Daniel Smith' via Kubernetes user discussion and Q&A
      • Re: [ku... Diego Rodríguez Rodríguez
        • Re:... 'Daniel Smith' via Kubernetes user discussion and Q&A
          • ... David Eads
            • ... Derek Carr
              • ... Diego Rodríguez Rodríguez
                • ... Rodrigo Campos
                • ... Diego Rodríguez Rodríguez
                • ... 'Daniel Smith' via Kubernetes user discussion and Q&A
                • ... Diego Rodríguez Rodríguez
                • ... Diego Rodríguez Rodríguez
                • ... 'Daniel Smith' via Kubernetes user discussion and Q&A
                • ... Diego Rodríguez Rodríguez
                • ... 'Daniel Smith' via Kubernetes user discussion and Q&A
                • ... ravi prasad l r

Reply via email to