Ah, you're blocked because the quota check reconciles slowly. The quick fix
is probably to just get more quota.

+David who may know of an already existing issue about this.

On Wed, Oct 12, 2016 at 5:39 AM, Diego Rodríguez Rodríguez <
diegorgz...@gmail.com> wrote:

> I have already configured my cluster to test what you have stated. What I
> have done so far is to create a ResourceQuota which takes care that there
> will not be more than 4 pods running. Then I ask for say 20 jobs to be
> completed.
>
> What happens in reality is that the first 4 jobs are completed and then,
> even though the pods are completed and therefore resources are available,
> it takes some minutes (around 4-5 minutes) to have another 4 jobs being
> completed. Indeed, what you said is true, however it is no practical
> because a delay of minutes can not be assumed.
> The waiting jobs events look like this:
> `FailedCreate    Error creating: pods 
> "d59fa9b2-6c6b-4dc5-b149-7f89b35421bf-10-"
> is forbidden: Exceeded quota: compute-resources, requested: pods=1, used:
> pods=4, limited: pods=4`
> So, it fails because there are no resources but it is trying it again only
> after some minutes. This behavior is far from the desired one, which is
> relaying on Kubernetes for the execution of a set of tasks no matter the
> resources available, just getting them executed as soon as possible.
>
> thanks
>
> On Monday, 10 October 2016 19:24:46 UTC+2, Daniel Smith wrote:
>>
>> If the system lacks resources, Pods will remain "Pending" until resources
>> become available. Cluster scalers may use pending pods as a signal that the
>> cluster size should be increased.
>>
>> On Mon, Oct 10, 2016 at 5:58 AM, Diego Rodríguez Rodríguez <
>> diego...@gmail.com> wrote:
>>
>>> Hello, I have a doubt about how Kubernetes' jobs are handled.
>>>
>>> I have a queue to submit certain amount of incoming tasks (Celery and
>>> RabbitMQ take care of this). Each one of this tasks is, in fact, a
>>> *workflow* which will be executed in a worker (Celery worker with a DAG
>>> executor inside). Each step of the *workflow* is a Docker image with an
>>> input file and an output file.
>>>
>>> My question is, if I submit jobs from the workflow engine directly to
>>> the Kubernetes API, what happens I at some point there are no more
>>> resources? Will the remaining tasks be kept or will they be lost? My goal
>>> is to treat Kubernetes' jobs as a black box to submit works to. This works
>>> are of a very different and heterogeneous nature and I wouldn't need to
>>> bother with what is inside them because they are dockerized and executed by
>>> Kubernetes at some point.
>>>
>>> To sum up, I already have the layer of Celery workers with a DAG
>>> executor inside which knows the right order of the tasks and knows how to
>>> manage everything concerning the *workflow*. These components will
>>> submit jobs (through Kubernetes API) and will wait for them to be executed
>>> and then continue with the remaining tasks asking Kubernetes to run them
>>> until the *workflow* ends.
>>>
>>> I have read about a somehow related issue in Github:
>>> https://github.com/kubernetes/kubernetes/pull/17787
>>> https://github.com/kubernetes/kubernetes/pull/18827
>>>
>>> I couldn't determine if this is closed or it is coming in a future
>>> release.
>>>
>>>
>>> Thanks in advance!
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Kubernetes user discussion and Q&A" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to kubernetes-use...@googlegroups.com.
>>> To post to this group, send email to kubernet...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/kubernetes-users.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-users+unsubscr...@googlegroups.com.
> To post to this group, send email to kubernetes-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.
  • [kubernetes-user... Diego Rodríguez Rodríguez
    • Re: [kubern... 'Daniel Smith' via Kubernetes user discussion and Q&A
      • Re: [ku... Diego Rodríguez Rodríguez
        • Re:... 'Daniel Smith' via Kubernetes user discussion and Q&A
          • ... David Eads
            • ... Derek Carr
              • ... Diego Rodríguez Rodríguez
                • ... Rodrigo Campos
                • ... Diego Rodríguez Rodríguez
                • ... 'Daniel Smith' via Kubernetes user discussion and Q&A
                • ... Diego Rodríguez Rodríguez
                • ... Diego Rodríguez Rodríguez
                • ... 'Daniel Smith' via Kubernetes user discussion and Q&A
                • ... Diego Rodríguez Rodríguez

Reply via email to