You can probably watch via the API and get notified, but Haven't used it.

But if you are building something not trivial, you may be better off
creating a third party resource that handles this and creates kubernetes
jobs under the hood. You will have more flexibility, etc.

On Thursday, October 13, 2016, Diego Rodríguez Rodríguez <
diegorgz...@gmail.com> wrote:

> I have already created an issue
> <https://github.com/kubernetes/kubernetes/issues/34715> in Github.
>
> Building on top of that, and closely related to
> https://github.com/kubernetes/kubernetes/issues/34363 and to
> https://github.com/kubernetes/kubernetes/pull/18827 I have the following
> question. Since I am planning to use Kubernetes as a executing backend for
> the workflow execution environment I am building, at some point we will
> need to know the current state of a job. Well, this time has come, and for
> now the only thing I am able to do is polling the API (
> kubernetes.io/docs/api-reference/extensions/v1beta1/operations/). Taking
> into account that this is unacceptable, and I only did it for testing
> purposes, what is the recommended way in Kubernetes for doing so? (Keep in
> mind that I should do it from inside the cluster, no kubectl command) If it
> is not possible right now, will it be possible soon? I am working on a
> quite big project and since it is starting, and the first problems we are
> facing are related to this issues, it is time to decide if I continue with
> Kubernetes or I have to start looking for alternatives.
>
> Thanks for your commitment
>
> On Wednesday, 12 October 2016 23:54:16 UTC+2, Derek Carr wrote:
>>
>> The quota controller observes pod deletions, and queues the pertinent
>> quotas for processing in response.  The replenishment should be relatively
>> quick now.
>>
>> We actually exercise this behavior in e2e runs.
>>
>>    1. What version of Kubernetes are you running?
>>    2. How many quotas?
>>    3. How many pods?
>>
>> My guess is that there is a conflict of some kind during the quota sync,
>> and the 5 minutes you are seeing is a full re-sync interval for the quota
>> controller.
>>
>> It would be good to try to determine what the conflict is so we can fix.
>>
>> If you have a quick set of steps I can run to reproduce that would be
>> great.
>>
>> I recommend opening an issue with the above details and cc @derekwaynecarr
>>
>> Thanks,
>> Derek
>>
>> On Wednesday, October 12, 2016 at 3:18:07 PM UTC-4, David Eads wrote:
>>>
>>> Quota reconcilation of pods was supposed to run at a faster rate.  I'm
>>> not sure if 5 minutes is that faster rate.
>>>
>>> *Derek:* Do you recall where and how the fast pod replenishment is?
>>>
>>>
>>> On Wednesday, October 12, 2016 at 1:28:45 PM UTC-4, Daniel Smith wrote:
>>>>
>>>> Ah, you're blocked because the quota check reconciles slowly. The quick
>>>> fix is probably to just get more quota.
>>>>
>>>> +David who may know of an already existing issue about this.
>>>>
>>>> On Wed, Oct 12, 2016 at 5:39 AM, Diego Rodríguez Rodríguez <
>>>> diego...@gmail.com> wrote:
>>>>
>>>>> I have already configured my cluster to test what you have stated.
>>>>> What I have done so far is to create a ResourceQuota which takes care that
>>>>> there will not be more than 4 pods running. Then I ask for say 20 jobs to
>>>>> be completed.
>>>>>
>>>>> What happens in reality is that the first 4 jobs are completed and
>>>>> then, even though the pods are completed and therefore resources are
>>>>> available, it takes some minutes (around 4-5 minutes) to have another 4
>>>>> jobs being completed. Indeed, what you said is true, however it is no
>>>>> practical because a delay of minutes can not be assumed.
>>>>> The waiting jobs events look like this:
>>>>> `FailedCreate    Error creating: pods 
>>>>> "d59fa9b2-6c6b-4dc5-b149-7f89b35421bf-10-"
>>>>> is forbidden: Exceeded quota: compute-resources, requested: pods=1, used:
>>>>> pods=4, limited: pods=4`
>>>>> So, it fails because there are no resources but it is trying it again
>>>>> only after some minutes. This behavior is far from the desired one, which
>>>>> is relaying on Kubernetes for the execution of a set of tasks no matter 
>>>>> the
>>>>> resources available, just getting them executed as soon as possible.
>>>>>
>>>>> thanks
>>>>>
>>>>> On Monday, 10 October 2016 19:24:46 UTC+2, Daniel Smith wrote:
>>>>>>
>>>>>> If the system lacks resources, Pods will remain "Pending" until
>>>>>> resources become available. Cluster scalers may use pending pods as a
>>>>>> signal that the cluster size should be increased.
>>>>>>
>>>>>> On Mon, Oct 10, 2016 at 5:58 AM, Diego Rodríguez Rodríguez <
>>>>>> diego...@gmail.com> wrote:
>>>>>>
>>>>>>> Hello, I have a doubt about how Kubernetes' jobs are handled.
>>>>>>>
>>>>>>> I have a queue to submit certain amount of incoming tasks (Celery
>>>>>>> and RabbitMQ take care of this). Each one of this tasks is, in fact, a
>>>>>>> *workflow* which will be executed in a worker (Celery worker with a
>>>>>>> DAG executor inside). Each step of the *workflow* is a Docker image
>>>>>>> with an input file and an output file.
>>>>>>>
>>>>>>> My question is, if I submit jobs from the workflow engine directly
>>>>>>> to the Kubernetes API, what happens I at some point there are no more
>>>>>>> resources? Will the remaining tasks be kept or will they be lost? My 
>>>>>>> goal
>>>>>>> is to treat Kubernetes' jobs as a black box to submit works to. This 
>>>>>>> works
>>>>>>> are of a very different and heterogeneous nature and I wouldn't need to
>>>>>>> bother with what is inside them because they are dockerized and 
>>>>>>> executed by
>>>>>>> Kubernetes at some point.
>>>>>>>
>>>>>>> To sum up, I already have the layer of Celery workers with a DAG
>>>>>>> executor inside which knows the right order of the tasks and knows how 
>>>>>>> to
>>>>>>> manage everything concerning the *workflow*. These components will
>>>>>>> submit jobs (through Kubernetes API) and will wait for them to be 
>>>>>>> executed
>>>>>>> and then continue with the remaining tasks asking Kubernetes to run them
>>>>>>> until the *workflow* ends.
>>>>>>>
>>>>>>> I have read about a somehow related issue in Github:
>>>>>>> https://github.com/kubernetes/kubernetes/pull/17787
>>>>>>> https://github.com/kubernetes/kubernetes/pull/18827
>>>>>>>
>>>>>>> I couldn't determine if this is closed or it is coming in a future
>>>>>>> release.
>>>>>>>
>>>>>>>
>>>>>>> Thanks in advance!
>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "Kubernetes user discussion and Q&A" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to kubernetes-use...@googlegroups.com.
>>>>>>> To post to this group, send email to kubernet...@googlegroups.com.
>>>>>>> Visit this group at https://groups.google.com/group/kubernetes-users
>>>>>>> .
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>
>>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Kubernetes user discussion and Q&A" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to kubernetes-use...@googlegroups.com.
>>>>> To post to this group, send email to kubernet...@googlegroups.com.
>>>>> Visit this group at https://groups.google.com/group/kubernetes-users.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-users+unsubscr...@googlegroups.com
> <javascript:_e(%7B%7D,'cvml','kubernetes-users%2bunsubscr...@googlegroups.com');>
> .
> To post to this group, send email to kubernetes-users@googlegroups.com
> <javascript:_e(%7B%7D,'cvml','kubernetes-users@googlegroups.com');>.
> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.
  • [kubernetes-user... Diego Rodríguez Rodríguez
    • Re: [kubern... 'Daniel Smith' via Kubernetes user discussion and Q&A
      • Re: [ku... Diego Rodríguez Rodríguez
        • Re:... 'Daniel Smith' via Kubernetes user discussion and Q&A
          • ... David Eads
            • ... Derek Carr
              • ... Diego Rodríguez Rodríguez
                • ... Rodrigo Campos
                • ... Diego Rodríguez Rodríguez
                • ... 'Daniel Smith' via Kubernetes user discussion and Q&A
                • ... Diego Rodríguez Rodríguez
                • ... Diego Rodríguez Rodríguez
                • ... 'Daniel Smith' via Kubernetes user discussion and Q&A
                • ... Diego Rodríguez Rodríguez
                • ... 'Daniel Smith' via Kubernetes user discussion and Q&A
                • ... ravi prasad l r

Reply via email to