Quota reconcilation of pods was supposed to run at a faster rate. I'm not sure if 5 minutes is that faster rate.
*Derek:* Do you recall where and how the fast pod replenishment is? On Wednesday, October 12, 2016 at 1:28:45 PM UTC-4, Daniel Smith wrote: > > Ah, you're blocked because the quota check reconciles slowly. The quick > fix is probably to just get more quota. > > +David who may know of an already existing issue about this. > > On Wed, Oct 12, 2016 at 5:39 AM, Diego Rodríguez Rodríguez < > diego...@gmail.com <javascript:>> wrote: > >> I have already configured my cluster to test what you have stated. What I >> have done so far is to create a ResourceQuota which takes care that there >> will not be more than 4 pods running. Then I ask for say 20 jobs to be >> completed. >> >> What happens in reality is that the first 4 jobs are completed and then, >> even though the pods are completed and therefore resources are available, >> it takes some minutes (around 4-5 minutes) to have another 4 jobs being >> completed. Indeed, what you said is true, however it is no practical >> because a delay of minutes can not be assumed. >> The waiting jobs events look like this: >> `FailedCreate Error creating: pods >> "d59fa9b2-6c6b-4dc5-b149-7f89b35421bf-10-" is forbidden: Exceeded quota: >> compute-resources, requested: pods=1, used: pods=4, limited: pods=4` >> So, it fails because there are no resources but it is trying it again >> only after some minutes. This behavior is far from the desired one, which >> is relaying on Kubernetes for the execution of a set of tasks no matter the >> resources available, just getting them executed as soon as possible. >> >> thanks >> >> On Monday, 10 October 2016 19:24:46 UTC+2, Daniel Smith wrote: >>> >>> If the system lacks resources, Pods will remain "Pending" until >>> resources become available. Cluster scalers may use pending pods as a >>> signal that the cluster size should be increased. >>> >>> On Mon, Oct 10, 2016 at 5:58 AM, Diego Rodríguez Rodríguez < >>> diego...@gmail.com> wrote: >>> >>>> Hello, I have a doubt about how Kubernetes' jobs are handled. >>>> >>>> I have a queue to submit certain amount of incoming tasks (Celery and >>>> RabbitMQ take care of this). Each one of this tasks is, in fact, a >>>> *workflow* which will be executed in a worker (Celery worker with a >>>> DAG executor inside). Each step of the *workflow* is a Docker image >>>> with an input file and an output file. >>>> >>>> My question is, if I submit jobs from the workflow engine directly to >>>> the Kubernetes API, what happens I at some point there are no more >>>> resources? Will the remaining tasks be kept or will they be lost? My goal >>>> is to treat Kubernetes' jobs as a black box to submit works to. This works >>>> are of a very different and heterogeneous nature and I wouldn't need to >>>> bother with what is inside them because they are dockerized and executed >>>> by >>>> Kubernetes at some point. >>>> >>>> To sum up, I already have the layer of Celery workers with a DAG >>>> executor inside which knows the right order of the tasks and knows how to >>>> manage everything concerning the *workflow*. These components will >>>> submit jobs (through Kubernetes API) and will wait for them to be executed >>>> and then continue with the remaining tasks asking Kubernetes to run them >>>> until the *workflow* ends. >>>> >>>> I have read about a somehow related issue in Github: >>>> https://github.com/kubernetes/kubernetes/pull/17787 >>>> https://github.com/kubernetes/kubernetes/pull/18827 >>>> >>>> I couldn't determine if this is closed or it is coming in a future >>>> release. >>>> >>>> >>>> Thanks in advance! >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Kubernetes user discussion and Q&A" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to kubernetes-use...@googlegroups.com. >>>> To post to this group, send email to kubernet...@googlegroups.com. >>>> Visit this group at https://groups.google.com/group/kubernetes-users. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Kubernetes user discussion and Q&A" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to kubernetes-use...@googlegroups.com <javascript:>. >> To post to this group, send email to kubernet...@googlegroups.com >> <javascript:>. >> Visit this group at https://groups.google.com/group/kubernetes-users. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
