Re: [kubernetes-users] Job execution in Kubernetes as a black box

['Daniel Smith' via Kubernetes user discussion and Q]

I would follow https://github.com/mbohlool/, he'll probably be the one
making the repos.

On Fri, Oct 14, 2016 at 11:07 AM, Diego Rodríguez Rodríguez <
diegorgz...@gmail.com> wrote:

> It was just that, but I feel like I would need a library to rely on as
> soon as possible (I am doing everything by hand) because the actual ones
> are not mature enough. When do you plan to release it? I am very interested.
>
>
> On Friday, 14 October 2016, 'Daniel Smith' via Kubernetes user discussion
> and Q  wrote:
>
>> Glad you figured out what was going on (I was about to say, I don't know
>> python well but that code doesn't look like it could possibly work for a
>> streaming connection).
>>
>> We are working on generating a python client. I will make sure we try the
>> watch feature.
>>
>> On Fri, Oct 14, 2016 at 5:15 AM, Diego Rodríguez Rodríguez <
>> diegorgz...@gmail.com> wrote:
>>
>>> I have identified the problem. It has nothing to do with Kubernetes, it
>>> is about how Python's requests module handles this kind of connections.
>>> Further information can be found in:
>>>
>>> https://github.com/kennethreitz/requests/issues/2433   ---> I have
>>> tested it, and it does not work for me.
>>> https://cobe.io/blog/posts/kubernetes-watch-python/ ---> Talks
>>> about the problem and provides a link to the Python module they have
>>> created to wrap the k8s API calls
>>>
>>> The module is https://bitbucket.org/cobeio/kube/overview and it might
>>> be of help for someone who runs in that kind of troubles, for me does not
>>> work because as I could read in their docs, they do not support k8s jobs
>>> yet.
>>>
>>>
>>> On 14 October 2016 at 10:36, Diego Rodríguez Rodríguez <
>>> diegorgz...@gmail.com> wrote:
>>>
 This is what I am doing already

 def get_job_state(job_id):
> response = requests.get('{}{}/{}'.format('https://kubernetes',
>  ENDPOINT, job_id),
> headers={'Authorization': 'Bearer
> {}'.format(
> TOKEN
> )},
> verify=CA_CERT_PATH)
>
> if response.ok:
> res_dict = json.loads(response.text)
> resource_version = res_dict['metadata']['resourceVersion']
> while not (res_dict['status'].get('succeeded', False) or
>res_dict['status'].get('failed', False)):
> response = requests.get('{0}{1}/{2}?watch
> =true={3}'
> .format('https://kubernetes',
> ENDPOINT, job_id,
> resource_version),
> headers={'Authorization': 'Bearer
> {}'.
>  format(
>  TOKEN
>  )},
> verify=CA_CERT_PATH)
>
> if response.ok:
> res_dict = json.loads(response.text)
> else:
> # handle this
>
 But it is not working, it is doing just the same as when I was using it
 without watch and RV.



 On 13 October 2016 at 19:13, 'Daniel Smith' via Kubernetes user
 discussion and Q  wrote:

> Watch definitely should do the thing you want.
>
> In a loop:
> 1. List. Note the returned RV. This gives you the current state of the
> system.
> 2. Watch, passing the RV.
> 3. Process events until the watch closes or times out.
>
> On Thu, Oct 13, 2016 at 7:43 AM, Diego Rodríguez Rodríguez <
> diegorgz...@gmail.com> wrote:
>
>> Thanks for the advice but I have already tried the `watch` parameter
>> and as far as I understand it does not work as expected. It gives me the
>> same output as if I did not use the `watch` parameter even if I provide 
>> the
>> first `resourceVersion` in order to be sure that it is comparing with the
>> first state of the job.
>>
>> I am making reference to http://kubernetes.io/docs/api-
>> reference/extensions/v1beta1/operations/ on the 'watch changes to an
>> object of kind Job' section.
>>
>> To be honest I have never used something like this `watch` parameter
>> before, so I may be misunderstanding it, but I am afraid I am not.
>>
>> On 13 October 2016 at 16:22, Rodrigo Campos 
>> wrote:
>>
>>> You can probably watch via the API and get notified, but Haven't
>>> used it.
>>>
>>> But if you are building something not trivial, you may be better off
>>> creating a third party resource that handles this and creates kubernetes
>>> jobs under the hood. You 

Re: [kubernetes-users] Job execution in Kubernetes as a black box

[Diego Rodríguez Rodríguez]

It was just that, but I feel like I would need a library to rely on as soon
as possible (I am doing everything by hand) because the actual ones are not
mature enough. When do you plan to release it? I am very interested.

On Friday, 14 October 2016, 'Daniel Smith' via Kubernetes user discussion
and Q  wrote:

> Glad you figured out what was going on (I was about to say, I don't know
> python well but that code doesn't look like it could possibly work for a
> streaming connection).
>
> We are working on generating a python client. I will make sure we try the
> watch feature.
>
> On Fri, Oct 14, 2016 at 5:15 AM, Diego Rodríguez Rodríguez <
> diegorgz...@gmail.com
> > wrote:
>
>> I have identified the problem. It has nothing to do with Kubernetes, it
>> is about how Python's requests module handles this kind of connections.
>> Further information can be found in:
>>
>> https://github.com/kennethreitz/requests/issues/2433   ---> I have
>> tested it, and it does not work for me.
>> https://cobe.io/blog/posts/kubernetes-watch-python/ ---> Talks about
>> the problem and provides a link to the Python module they have created to
>> wrap the k8s API calls
>>
>> The module is https://bitbucket.org/cobeio/kube/overview and it might be
>> of help for someone who runs in that kind of troubles, for me does not work
>> because as I could read in their docs, they do not support k8s jobs yet.
>>
>>
>> On 14 October 2016 at 10:36, Diego Rodríguez Rodríguez <
>> diegorgz...@gmail.com
>> > wrote:
>>
>>> This is what I am doing already
>>>
>>> def get_job_state(job_id):
 response = requests.get('{}{}/{}'.format('https://kubernetes',
  ENDPOINT, job_id),
 headers={'Authorization': 'Bearer
 {}'.format(
 TOKEN
 )},
 verify=CA_CERT_PATH)

 if response.ok:
 res_dict = json.loads(response.text)
 resource_version = res_dict['metadata']['resourceVersion']
 while not (res_dict['status'].get('succeeded', False) or
res_dict['status'].get('failed', False)):
 response = requests.get('{0}{1}/{2}?watch
 =true={3}'
 .format('https://kubernetes',
 ENDPOINT, job_id,
 resource_version),
 headers={'Authorization': 'Bearer
 {}'.
  format(
  TOKEN
  )},
 verify=CA_CERT_PATH)

 if response.ok:
 res_dict = json.loads(response.text)
 else:
 # handle this

>>> But it is not working, it is doing just the same as when I was using it
>>> without watch and RV.
>>>
>>>
>>>
>>> On 13 October 2016 at 19:13, 'Daniel Smith' via Kubernetes user
>>> discussion and Q >> >
>>> wrote:
>>>
 Watch definitely should do the thing you want.

 In a loop:
 1. List. Note the returned RV. This gives you the current state of the
 system.
 2. Watch, passing the RV.
 3. Process events until the watch closes or times out.

 On Thu, Oct 13, 2016 at 7:43 AM, Diego Rodríguez Rodríguez <
 diegorgz...@gmail.com
 > wrote:

> Thanks for the advice but I have already tried the `watch` parameter
> and as far as I understand it does not work as expected. It gives me the
> same output as if I did not use the `watch` parameter even if I provide 
> the
> first `resourceVersion` in order to be sure that it is comparing with the
> first state of the job.
>
> I am making reference to http://kubernetes.io/docs/api-
> reference/extensions/v1beta1/operations/ on the 'watch changes to an
> object of kind Job' section.
>
> To be honest I have never used something like this `watch` parameter
> before, so I may be misunderstanding it, but I am afraid I am not.
>
> On 13 October 2016 at 16:22, Rodrigo Campos  > wrote:
>
>> You can probably watch via the API and get notified, but Haven't used
>> it.
>>
>> But if you are building something not trivial, you may be better off
>> creating a third party resource that handles this and creates kubernetes
>> jobs under the hood. 

Re: [kubernetes-users] Job execution in Kubernetes as a black box

['Daniel Smith' via Kubernetes user discussion and Q]

Watch definitely should do the thing you want.

In a loop:
1. List. Note the returned RV. This gives you the current state of the
system.
2. Watch, passing the RV.
3. Process events until the watch closes or times out.

On Thu, Oct 13, 2016 at 7:43 AM, Diego Rodríguez Rodríguez <
diegorgz...@gmail.com> wrote:

> Thanks for the advice but I have already tried the `watch` parameter and
> as far as I understand it does not work as expected. It gives me the same
> output as if I did not use the `watch` parameter even if I provide the
> first `resourceVersion` in order to be sure that it is comparing with the
> first state of the job.
>
> I am making reference to http://kubernetes.io/docs/api-
> reference/extensions/v1beta1/operations/ on the 'watch changes to an
> object of kind Job' section.
>
> To be honest I have never used something like this `watch` parameter
> before, so I may be misunderstanding it, but I am afraid I am not.
>
> On 13 October 2016 at 16:22, Rodrigo Campos  wrote:
>
>> You can probably watch via the API and get notified, but Haven't used it.
>>
>> But if you are building something not trivial, you may be better off
>> creating a third party resource that handles this and creates kubernetes
>> jobs under the hood. You will have more flexibility, etc.
>>
>>
>> On Thursday, October 13, 2016, Diego Rodríguez Rodríguez <
>> diegorgz...@gmail.com> wrote:
>>
>>> I have already created an issue
>>>  in Github.
>>>
>>> Building on top of that, and closely related to
>>> https://github.com/kubernetes/kubernetes/issues/34363 and to
>>> https://github.com/kubernetes/kubernetes/pull/18827 I have the
>>> following question. Since I am planning to use Kubernetes as a executing
>>> backend for the workflow execution environment I am building, at some point
>>> we will need to know the current state of a job. Well, this time has come,
>>> and for now the only thing I am able to do is polling the API (
>>> kubernetes.io/docs/api-reference/extensions/v1beta1/operations/).
>>> Taking into account that this is unacceptable, and I only did it for
>>> testing purposes, what is the recommended way in Kubernetes for doing so?
>>> (Keep in mind that I should do it from inside the cluster, no kubectl
>>> command) If it is not possible right now, will it be possible soon? I am
>>> working on a quite big project and since it is starting, and the first
>>> problems we are facing are related to this issues, it is time to decide if
>>> I continue with Kubernetes or I have to start looking for alternatives.
>>>
>>> Thanks for your commitment
>>>
>>> On Wednesday, 12 October 2016 23:54:16 UTC+2, Derek Carr wrote:

 The quota controller observes pod deletions, and queues the pertinent
 quotas for processing in response.  The replenishment should be relatively
 quick now.

 We actually exercise this behavior in e2e runs.

1. What version of Kubernetes are you running?
2. How many quotas?
3. How many pods?

 My guess is that there is a conflict of some kind during the quota
 sync, and the 5 minutes you are seeing is a full re-sync interval for the
 quota controller.

 It would be good to try to determine what the conflict is so we can fix.

 If you have a quick set of steps I can run to reproduce that would be
 great.

 I recommend opening an issue with the above details and cc
 @derekwaynecarr

 Thanks,
 Derek

 On Wednesday, October 12, 2016 at 3:18:07 PM UTC-4, David Eads wrote:
>
> Quota reconcilation of pods was supposed to run at a faster rate.  I'm
> not sure if 5 minutes is that faster rate.
>
> *Derek:* Do you recall where and how the fast pod replenishment is?
>
>
> On Wednesday, October 12, 2016 at 1:28:45 PM UTC-4, Daniel Smith wrote:
>>
>> Ah, you're blocked because the quota check reconciles slowly. The
>> quick fix is probably to just get more quota.
>>
>> +David who may know of an already existing issue about this.
>>
>> On Wed, Oct 12, 2016 at 5:39 AM, Diego Rodríguez Rodríguez <
>> diego...@gmail.com> wrote:
>>
>>> I have already configured my cluster to test what you have stated.
>>> What I have done so far is to create a ResourceQuota which takes care 
>>> that
>>> there will not be more than 4 pods running. Then I ask for say 20 jobs 
>>> to
>>> be completed.
>>>
>>> What happens in reality is that the first 4 jobs are completed and
>>> then, even though the pods are completed and therefore resources are
>>> available, it takes some minutes (around 4-5 minutes) to have another 4
>>> jobs being completed. Indeed, what you said is true, however it is no
>>> practical because a delay of minutes can not be assumed.
>>> The waiting jobs events look like this:
>>> `FailedCreateError 

Re: [kubernetes-users] Job execution in Kubernetes as a black box

[Diego Rodríguez Rodríguez]

Thanks for the advice but I have already tried the `watch` parameter and as
far as I understand it does not work as expected. It gives me the same
output as if I did not use the `watch` parameter even if I provide the
first `resourceVersion` in order to be sure that it is comparing with the
first state of the job.

I am making reference to
http://kubernetes.io/docs/api-reference/extensions/v1beta1/operations/ on
the 'watch changes to an object of kind Job' section.

To be honest I have never used something like this `watch` parameter
before, so I may be misunderstanding it, but I am afraid I am not.

On 13 October 2016 at 16:22, Rodrigo Campos  wrote:

> You can probably watch via the API and get notified, but Haven't used it.
>
> But if you are building something not trivial, you may be better off
> creating a third party resource that handles this and creates kubernetes
> jobs under the hood. You will have more flexibility, etc.
>
>
> On Thursday, October 13, 2016, Diego Rodríguez Rodríguez <
> diegorgz...@gmail.com> wrote:
>
>> I have already created an issue
>>  in Github.
>>
>> Building on top of that, and closely related to
>> https://github.com/kubernetes/kubernetes/issues/34363 and to
>> https://github.com/kubernetes/kubernetes/pull/18827 I have the following
>> question. Since I am planning to use Kubernetes as a executing backend for
>> the workflow execution environment I am building, at some point we will
>> need to know the current state of a job. Well, this time has come, and for
>> now the only thing I am able to do is polling the API (
>> kubernetes.io/docs/api-reference/extensions/v1beta1/operations/). Taking
>> into account that this is unacceptable, and I only did it for testing
>> purposes, what is the recommended way in Kubernetes for doing so? (Keep in
>> mind that I should do it from inside the cluster, no kubectl command) If it
>> is not possible right now, will it be possible soon? I am working on a
>> quite big project and since it is starting, and the first problems we are
>> facing are related to this issues, it is time to decide if I continue with
>> Kubernetes or I have to start looking for alternatives.
>>
>> Thanks for your commitment
>>
>> On Wednesday, 12 October 2016 23:54:16 UTC+2, Derek Carr wrote:
>>>
>>> The quota controller observes pod deletions, and queues the pertinent
>>> quotas for processing in response.  The replenishment should be relatively
>>> quick now.
>>>
>>> We actually exercise this behavior in e2e runs.
>>>
>>>1. What version of Kubernetes are you running?
>>>2. How many quotas?
>>>3. How many pods?
>>>
>>> My guess is that there is a conflict of some kind during the quota sync,
>>> and the 5 minutes you are seeing is a full re-sync interval for the quota
>>> controller.
>>>
>>> It would be good to try to determine what the conflict is so we can fix.
>>>
>>> If you have a quick set of steps I can run to reproduce that would be
>>> great.
>>>
>>> I recommend opening an issue with the above details and cc
>>> @derekwaynecarr
>>>
>>> Thanks,
>>> Derek
>>>
>>> On Wednesday, October 12, 2016 at 3:18:07 PM UTC-4, David Eads wrote:

 Quota reconcilation of pods was supposed to run at a faster rate.  I'm
 not sure if 5 minutes is that faster rate.

 *Derek:* Do you recall where and how the fast pod replenishment is?


 On Wednesday, October 12, 2016 at 1:28:45 PM UTC-4, Daniel Smith wrote:
>
> Ah, you're blocked because the quota check reconciles slowly. The
> quick fix is probably to just get more quota.
>
> +David who may know of an already existing issue about this.
>
> On Wed, Oct 12, 2016 at 5:39 AM, Diego Rodríguez Rodríguez <
> diego...@gmail.com> wrote:
>
>> I have already configured my cluster to test what you have stated.
>> What I have done so far is to create a ResourceQuota which takes care 
>> that
>> there will not be more than 4 pods running. Then I ask for say 20 jobs to
>> be completed.
>>
>> What happens in reality is that the first 4 jobs are completed and
>> then, even though the pods are completed and therefore resources are
>> available, it takes some minutes (around 4-5 minutes) to have another 4
>> jobs being completed. Indeed, what you said is true, however it is no
>> practical because a delay of minutes can not be assumed.
>> The waiting jobs events look like this:
>> `FailedCreateError creating: pods 
>> "d59fa9b2-6c6b-4dc5-b149-7f89b35421bf-10-"
>> is forbidden: Exceeded quota: compute-resources, requested: pods=1, used:
>> pods=4, limited: pods=4`
>> So, it fails because there are no resources but it is trying it again
>> only after some minutes. This behavior is far from the desired one, which
>> is relaying on Kubernetes for the execution of a set of tasks no matter 
>> the
>> resources 

Re: [kubernetes-users] Job execution in Kubernetes as a black box

[Diego Rodríguez Rodríguez]

I have already created an issue 
 in Github.

Building on top of that, and closely related to 
https://github.com/kubernetes/kubernetes/issues/34363 and to 
https://github.com/kubernetes/kubernetes/pull/18827 I have the following 
question. Since I am planning to use Kubernetes as a executing backend for 
the workflow execution environment I am building, at some point we will 
need to know the current state of a job. Well, this time has come, and for 
now the only thing I am able to do is polling the API 
(kubernetes.io/docs/api-reference/extensions/v1beta1/operations/). Taking 
into account that this is unacceptable, and I only did it for testing 
purposes, what is the recommended way in Kubernetes for doing so? (Keep in 
mind that I should do it from inside the cluster, no kubectl command) If it 
is not possible right now, will it be possible soon? I am working on a 
quite big project and since it is starting, and the first problems we are 
facing are related to this issues, it is time to decide if I continue with 
Kubernetes or I have to start looking for alternatives.

Thanks for your commitment

On Wednesday, 12 October 2016 23:54:16 UTC+2, Derek Carr wrote:
>
> The quota controller observes pod deletions, and queues the pertinent 
> quotas for processing in response.  The replenishment should be relatively 
> quick now.
>
> We actually exercise this behavior in e2e runs.
>
>1. What version of Kubernetes are you running?
>2. How many quotas?
>3. How many pods?
>
> My guess is that there is a conflict of some kind during the quota sync, 
> and the 5 minutes you are seeing is a full re-sync interval for the quota 
> controller.
>
> It would be good to try to determine what the conflict is so we can fix.
>
> If you have a quick set of steps I can run to reproduce that would be 
> great.
>
> I recommend opening an issue with the above details and cc @derekwaynecarr
>
> Thanks,
> Derek
>
> On Wednesday, October 12, 2016 at 3:18:07 PM UTC-4, David Eads wrote:
>>
>> Quota reconcilation of pods was supposed to run at a faster rate.  I'm 
>> not sure if 5 minutes is that faster rate.  
>>
>> *Derek:* Do you recall where and how the fast pod replenishment is?
>>
>>
>> On Wednesday, October 12, 2016 at 1:28:45 PM UTC-4, Daniel Smith wrote:
>>>
>>> Ah, you're blocked because the quota check reconciles slowly. The quick 
>>> fix is probably to just get more quota.
>>>
>>> +David who may know of an already existing issue about this.
>>>
>>> On Wed, Oct 12, 2016 at 5:39 AM, Diego Rodríguez Rodríguez <
>>> diego...@gmail.com> wrote:
>>>
 I have already configured my cluster to test what you have stated. What 
 I have done so far is to create a ResourceQuota which takes care that 
 there 
 will not be more than 4 pods running. Then I ask for say 20 jobs to be 
 completed.

 What happens in reality is that the first 4 jobs are completed and 
 then, even though the pods are completed and therefore resources are 
 available, it takes some minutes (around 4-5 minutes) to have another 4 
 jobs being completed. Indeed, what you said is true, however it is no 
 practical because a delay of minutes can not be assumed.
 The waiting jobs events look like this:
 `FailedCreateError creating: pods 
 "d59fa9b2-6c6b-4dc5-b149-7f89b35421bf-10-" is forbidden: Exceeded quota: 
 compute-resources, requested: pods=1, used: pods=4, limited: pods=4`
 So, it fails because there are no resources but it is trying it again 
 only after some minutes. This behavior is far from the desired one, which 
 is relaying on Kubernetes for the execution of a set of tasks no matter 
 the 
 resources available, just getting them executed as soon as possible.

 thanks

 On Monday, 10 October 2016 19:24:46 UTC+2, Daniel Smith wrote:
>
> If the system lacks resources, Pods will remain "Pending" until 
> resources become available. Cluster scalers may use pending pods as a 
> signal that the cluster size should be increased.
>
> On Mon, Oct 10, 2016 at 5:58 AM, Diego Rodríguez Rodríguez <
> diego...@gmail.com> wrote:
>
>> Hello, I have a doubt about how Kubernetes' jobs are handled.
>>
>> I have a queue to submit certain amount of incoming tasks (Celery and 
>> RabbitMQ take care of this). Each one of this tasks is, in fact, a 
>> *workflow* which will be executed in a worker (Celery worker with a 
>> DAG executor inside). Each step of the *workflow* is a Docker image 
>> with an input file and an output file.
>>
>> My question is, if I submit jobs from the workflow engine directly to 
>> the Kubernetes API, what happens I at some point there are no more 
>> resources? Will the remaining tasks be kept or will they be lost? My 
>> goal 
>> is to treat Kubernetes' jobs as a black box to submit works to. This 

Re: [kubernetes-users] Job execution in Kubernetes as a black box

[David Eads]

Quota reconcilation of pods was supposed to run at a faster rate.  I'm not 
sure if 5 minutes is that faster rate.  

*Derek:* Do you recall where and how the fast pod replenishment is?


On Wednesday, October 12, 2016 at 1:28:45 PM UTC-4, Daniel Smith wrote:
>
> Ah, you're blocked because the quota check reconciles slowly. The quick 
> fix is probably to just get more quota.
>
> +David who may know of an already existing issue about this.
>
> On Wed, Oct 12, 2016 at 5:39 AM, Diego Rodríguez Rodríguez <
> diego...@gmail.com > wrote:
>
>> I have already configured my cluster to test what you have stated. What I 
>> have done so far is to create a ResourceQuota which takes care that there 
>> will not be more than 4 pods running. Then I ask for say 20 jobs to be 
>> completed.
>>
>> What happens in reality is that the first 4 jobs are completed and then, 
>> even though the pods are completed and therefore resources are available, 
>> it takes some minutes (around 4-5 minutes) to have another 4 jobs being 
>> completed. Indeed, what you said is true, however it is no practical 
>> because a delay of minutes can not be assumed.
>> The waiting jobs events look like this:
>> `FailedCreateError creating: pods 
>> "d59fa9b2-6c6b-4dc5-b149-7f89b35421bf-10-" is forbidden: Exceeded quota: 
>> compute-resources, requested: pods=1, used: pods=4, limited: pods=4`
>> So, it fails because there are no resources but it is trying it again 
>> only after some minutes. This behavior is far from the desired one, which 
>> is relaying on Kubernetes for the execution of a set of tasks no matter the 
>> resources available, just getting them executed as soon as possible.
>>
>> thanks
>>
>> On Monday, 10 October 2016 19:24:46 UTC+2, Daniel Smith wrote:
>>>
>>> If the system lacks resources, Pods will remain "Pending" until 
>>> resources become available. Cluster scalers may use pending pods as a 
>>> signal that the cluster size should be increased.
>>>
>>> On Mon, Oct 10, 2016 at 5:58 AM, Diego Rodríguez Rodríguez <
>>> diego...@gmail.com> wrote:
>>>
 Hello, I have a doubt about how Kubernetes' jobs are handled.

 I have a queue to submit certain amount of incoming tasks (Celery and 
 RabbitMQ take care of this). Each one of this tasks is, in fact, a 
 *workflow* which will be executed in a worker (Celery worker with a 
 DAG executor inside). Each step of the *workflow* is a Docker image 
 with an input file and an output file.

 My question is, if I submit jobs from the workflow engine directly to 
 the Kubernetes API, what happens I at some point there are no more 
 resources? Will the remaining tasks be kept or will they be lost? My goal 
 is to treat Kubernetes' jobs as a black box to submit works to. This works 
 are of a very different and heterogeneous nature and I wouldn't need to 
 bother with what is inside them because they are dockerized and executed 
 by 
 Kubernetes at some point.

 To sum up, I already have the layer of Celery workers with a DAG 
 executor inside which knows the right order of the tasks and knows how to 
 manage everything concerning the *workflow*. These components will 
 submit jobs (through Kubernetes API) and will wait for them to be executed 
 and then continue with the remaining tasks asking Kubernetes to run them 
 until the *workflow* ends.

 I have read about a somehow related issue in Github:
 https://github.com/kubernetes/kubernetes/pull/17787
 https://github.com/kubernetes/kubernetes/pull/18827

 I couldn't determine if this is closed or it is coming in a future 
 release.


 Thanks in advance!

 -- 
 You received this message because you are subscribed to the Google 
 Groups "Kubernetes user discussion and Q" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to kubernetes-use...@googlegroups.com.
 To post to this group, send email to kubernet...@googlegroups.com.
 Visit this group at https://groups.google.com/group/kubernetes-users.
 For more options, visit https://groups.google.com/d/optout.

>>>
>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Kubernetes user discussion and Q" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to kubernetes-use...@googlegroups.com .
>> To post to this group, send email to kubernet...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/kubernetes-users.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Kubernetes user discussion and Q" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to 

Re: [kubernetes-users] Job execution in Kubernetes as a black box

['Daniel Smith' via Kubernetes user discussion and Q]

If the system lacks resources, Pods will remain "Pending" until resources
become available. Cluster scalers may use pending pods as a signal that the
cluster size should be increased.

On Mon, Oct 10, 2016 at 5:58 AM, Diego Rodríguez Rodríguez <
diegorgz...@gmail.com> wrote:

> Hello, I have a doubt about how Kubernetes' jobs are handled.
>
> I have a queue to submit certain amount of incoming tasks (Celery and
> RabbitMQ take care of this). Each one of this tasks is, in fact, a
> *workflow* which will be executed in a worker (Celery worker with a DAG
> executor inside). Each step of the *workflow* is a Docker image with an
> input file and an output file.
>
> My question is, if I submit jobs from the workflow engine directly to the
> Kubernetes API, what happens I at some point there are no more resources?
> Will the remaining tasks be kept or will they be lost? My goal is to treat
> Kubernetes' jobs as a black box to submit works to. This works are of a
> very different and heterogeneous nature and I wouldn't need to bother with
> what is inside them because they are dockerized and executed by Kubernetes
> at some point.
>
> To sum up, I already have the layer of Celery workers with a DAG executor
> inside which knows the right order of the tasks and knows how to manage
> everything concerning the *workflow*. These components will submit jobs
> (through Kubernetes API) and will wait for them to be executed and then
> continue with the remaining tasks asking Kubernetes to run them until the
> *workflow* ends.
>
> I have read about a somehow related issue in Github:
> https://github.com/kubernetes/kubernetes/pull/17787
> https://github.com/kubernetes/kubernetes/pull/18827
>
> I couldn't determine if this is closed or it is coming in a future release.
>
>
> Thanks in advance!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-users+unsubscr...@googlegroups.com.
> To post to this group, send email to kubernetes-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Kubernetes user discussion and Q" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.