I have already created an issue 
<https://github.com/kubernetes/kubernetes/issues/34715> in Github.

Building on top of that, and closely related to 
https://github.com/kubernetes/kubernetes/issues/34363 and to 
https://github.com/kubernetes/kubernetes/pull/18827 I have the following 
question. Since I am planning to use Kubernetes as a executing backend for 
the workflow execution environment I am building, at some point we will 
need to know the current state of a job. Well, this time has come, and for 
now the only thing I am able to do is polling the API 
(kubernetes.io/docs/api-reference/extensions/v1beta1/operations/). Taking 
into account that this is unacceptable, and I only did it for testing 
purposes, what is the recommended way in Kubernetes for doing so? (Keep in 
mind that I should do it from inside the cluster, no kubectl command) If it 
is not possible right now, will it be possible soon? I am working on a 
quite big project and since it is starting, and the first problems we are 
facing are related to this issues, it is time to decide if I continue with 
Kubernetes or I have to start looking for alternatives.

Thanks for your commitment

On Wednesday, 12 October 2016 23:54:16 UTC+2, Derek Carr wrote:
>
> The quota controller observes pod deletions, and queues the pertinent 
> quotas for processing in response.  The replenishment should be relatively 
> quick now.
>
> We actually exercise this behavior in e2e runs.
>
>    1. What version of Kubernetes are you running?
>    2. How many quotas?
>    3. How many pods?
>
> My guess is that there is a conflict of some kind during the quota sync, 
> and the 5 minutes you are seeing is a full re-sync interval for the quota 
> controller.
>
> It would be good to try to determine what the conflict is so we can fix.
>
> If you have a quick set of steps I can run to reproduce that would be 
> great.
>
> I recommend opening an issue with the above details and cc @derekwaynecarr
>
> Thanks,
> Derek
>
> On Wednesday, October 12, 2016 at 3:18:07 PM UTC-4, David Eads wrote:
>>
>> Quota reconcilation of pods was supposed to run at a faster rate.  I'm 
>> not sure if 5 minutes is that faster rate.  
>>
>> *Derek:* Do you recall where and how the fast pod replenishment is?
>>
>>
>> On Wednesday, October 12, 2016 at 1:28:45 PM UTC-4, Daniel Smith wrote:
>>>
>>> Ah, you're blocked because the quota check reconciles slowly. The quick 
>>> fix is probably to just get more quota.
>>>
>>> +David who may know of an already existing issue about this.
>>>
>>> On Wed, Oct 12, 2016 at 5:39 AM, Diego Rodríguez Rodríguez <
>>> diego...@gmail.com> wrote:
>>>
>>>> I have already configured my cluster to test what you have stated. What 
>>>> I have done so far is to create a ResourceQuota which takes care that 
>>>> there 
>>>> will not be more than 4 pods running. Then I ask for say 20 jobs to be 
>>>> completed.
>>>>
>>>> What happens in reality is that the first 4 jobs are completed and 
>>>> then, even though the pods are completed and therefore resources are 
>>>> available, it takes some minutes (around 4-5 minutes) to have another 4 
>>>> jobs being completed. Indeed, what you said is true, however it is no 
>>>> practical because a delay of minutes can not be assumed.
>>>> The waiting jobs events look like this:
>>>> `FailedCreate    Error creating: pods 
>>>> "d59fa9b2-6c6b-4dc5-b149-7f89b35421bf-10-" is forbidden: Exceeded quota: 
>>>> compute-resources, requested: pods=1, used: pods=4, limited: pods=4`
>>>> So, it fails because there are no resources but it is trying it again 
>>>> only after some minutes. This behavior is far from the desired one, which 
>>>> is relaying on Kubernetes for the execution of a set of tasks no matter 
>>>> the 
>>>> resources available, just getting them executed as soon as possible.
>>>>
>>>> thanks
>>>>
>>>> On Monday, 10 October 2016 19:24:46 UTC+2, Daniel Smith wrote:
>>>>>
>>>>> If the system lacks resources, Pods will remain "Pending" until 
>>>>> resources become available. Cluster scalers may use pending pods as a 
>>>>> signal that the cluster size should be increased.
>>>>>
>>>>> On Mon, Oct 10, 2016 at 5:58 AM, Diego Rodríguez Rodríguez <
>>>>> diego...@gmail.com> wrote:
>>>>>
>>>>>> Hello, I have a doubt about how Kubernetes' jobs are handled.
>>>>>>
>>>>>> I have a queue to submit certain amount of incoming tasks (Celery and 
>>>>>> RabbitMQ take care of this). Each one of this tasks is, in fact, a 
>>>>>> *workflow* which will be executed in a worker (Celery worker with a 
>>>>>> DAG executor inside). Each step of the *workflow* is a Docker image 
>>>>>> with an input file and an output file.
>>>>>>
>>>>>> My question is, if I submit jobs from the workflow engine directly to 
>>>>>> the Kubernetes API, what happens I at some point there are no more 
>>>>>> resources? Will the remaining tasks be kept or will they be lost? My 
>>>>>> goal 
>>>>>> is to treat Kubernetes' jobs as a black box to submit works to. This 
>>>>>> works 
>>>>>> are of a very different and heterogeneous nature and I wouldn't need to 
>>>>>> bother with what is inside them because they are dockerized and executed 
>>>>>> by 
>>>>>> Kubernetes at some point.
>>>>>>
>>>>>> To sum up, I already have the layer of Celery workers with a DAG 
>>>>>> executor inside which knows the right order of the tasks and knows how 
>>>>>> to 
>>>>>> manage everything concerning the *workflow*. These components will 
>>>>>> submit jobs (through Kubernetes API) and will wait for them to be 
>>>>>> executed 
>>>>>> and then continue with the remaining tasks asking Kubernetes to run them 
>>>>>> until the *workflow* ends.
>>>>>>
>>>>>> I have read about a somehow related issue in Github:
>>>>>> https://github.com/kubernetes/kubernetes/pull/17787
>>>>>> https://github.com/kubernetes/kubernetes/pull/18827
>>>>>>
>>>>>> I couldn't determine if this is closed or it is coming in a future 
>>>>>> release.
>>>>>>
>>>>>>
>>>>>> Thanks in advance!
>>>>>>
>>>>>> -- 
>>>>>> You received this message because you are subscribed to the Google 
>>>>>> Groups "Kubernetes user discussion and Q&A" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it, 
>>>>>> send an email to kubernetes-use...@googlegroups.com.
>>>>>> To post to this group, send email to kubernet...@googlegroups.com.
>>>>>> Visit this group at https://groups.google.com/group/kubernetes-users.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>
>>>>> -- 
>>>> You received this message because you are subscribed to the Google 
>>>> Groups "Kubernetes user discussion and Q&A" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send 
>>>> an email to kubernetes-use...@googlegroups.com.
>>>> To post to this group, send email to kubernet...@googlegroups.com.
>>>> Visit this group at https://groups.google.com/group/kubernetes-users.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.
  • [kubernetes-user... Diego Rodríguez Rodríguez
    • Re: [kubern... 'Daniel Smith' via Kubernetes user discussion and Q&A
      • Re: [ku... Diego Rodríguez Rodríguez
        • Re:... 'Daniel Smith' via Kubernetes user discussion and Q&A
          • ... David Eads
            • ... Derek Carr
              • ... Diego Rodríguez Rodríguez
                • ... Rodrigo Campos
                • ... Diego Rodríguez Rodríguez
                • ... 'Daniel Smith' via Kubernetes user discussion and Q&A
                • ... Diego Rodríguez Rodríguez
                • ... Diego Rodríguez Rodríguez
                • ... 'Daniel Smith' via Kubernetes user discussion and Q&A
                • ... Diego Rodríguez Rodríguez
                • ... 'Daniel Smith' via Kubernetes user discussion and Q&A
                • ... ravi prasad l r

Reply via email to