Glad you figured out what was going on (I was about to say, I don't know python well but that code doesn't look like it could possibly work for a streaming connection).
We are working on generating a python client. I will make sure we try the watch feature. On Fri, Oct 14, 2016 at 5:15 AM, Diego Rodríguez Rodríguez < diegorgz...@gmail.com> wrote: > I have identified the problem. It has nothing to do with Kubernetes, it is > about how Python's requests module handles this kind of connections. > Further information can be found in: > > https://github.com/kennethreitz/requests/issues/2433 ---> I have tested > it, and it does not work for me. > https://cobe.io/blog/posts/kubernetes-watch-python/ ---> Talks about > the problem and provides a link to the Python module they have created to > wrap the k8s API calls > > The module is https://bitbucket.org/cobeio/kube/overview and it might be > of help for someone who runs in that kind of troubles, for me does not work > because as I could read in their docs, they do not support k8s jobs yet. > > > On 14 October 2016 at 10:36, Diego Rodríguez Rodríguez < > diegorgz...@gmail.com> wrote: > >> This is what I am doing already >> >> def get_job_state(job_id): >>> response = requests.get('{}{}/{}'.format('https://kubernetes', >>> ENDPOINT, job_id), >>> headers={'Authorization': 'Bearer {}'.format( >>> TOKEN >>> )}, >>> verify=CA_CERT_PATH) >>> >>> if response.ok: >>> res_dict = json.loads(response.text) >>> resource_version = res_dict['metadata']['resourceVersion'] >>> while not (res_dict['status'].get('succeeded', False) or >>> res_dict['status'].get('failed', False)): >>> response = requests.get('{0}{1}/{2}?watch >>> =true&resourceVersion={3}' >>> .format('https://kubernetes', >>> ENDPOINT, job_id, >>> resource_version), >>> headers={'Authorization': 'Bearer >>> {}'. >>> format( >>> TOKEN >>> )}, >>> verify=CA_CERT_PATH) >>> >>> if response.ok: >>> res_dict = json.loads(response.text) >>> else: >>> # handle this >>> >> But it is not working, it is doing just the same as when I was using it >> without watch and RV. >> >> >> >> On 13 October 2016 at 19:13, 'Daniel Smith' via Kubernetes user >> discussion and Q&A <kubernetes-users@googlegroups.com> wrote: >> >>> Watch definitely should do the thing you want. >>> >>> In a loop: >>> 1. List. Note the returned RV. This gives you the current state of the >>> system. >>> 2. Watch, passing the RV. >>> 3. Process events until the watch closes or times out. >>> >>> On Thu, Oct 13, 2016 at 7:43 AM, Diego Rodríguez Rodríguez < >>> diegorgz...@gmail.com> wrote: >>> >>>> Thanks for the advice but I have already tried the `watch` parameter >>>> and as far as I understand it does not work as expected. It gives me the >>>> same output as if I did not use the `watch` parameter even if I provide the >>>> first `resourceVersion` in order to be sure that it is comparing with the >>>> first state of the job. >>>> >>>> I am making reference to http://kubernetes.io/docs/api- >>>> reference/extensions/v1beta1/operations/ on the 'watch changes to an >>>> object of kind Job' section. >>>> >>>> To be honest I have never used something like this `watch` parameter >>>> before, so I may be misunderstanding it, but I am afraid I am not. >>>> >>>> On 13 October 2016 at 16:22, Rodrigo Campos <rodrig...@gmail.com> >>>> wrote: >>>> >>>>> You can probably watch via the API and get notified, but Haven't used >>>>> it. >>>>> >>>>> But if you are building something not trivial, you may be better off >>>>> creating a third party resource that handles this and creates kubernetes >>>>> jobs under the hood. You will have more flexibility, etc. >>>>> >>>>> >>>>> On Thursday, October 13, 2016, Diego Rodríguez Rodríguez < >>>>> diegorgz...@gmail.com> wrote: >>>>> >>>>>> I have already created an issue >>>>>> <https://github.com/kubernetes/kubernetes/issues/34715> in Github. >>>>>> >>>>>> Building on top of that, and closely related to >>>>>> https://github.com/kubernetes/kubernetes/issues/34363 and to >>>>>> https://github.com/kubernetes/kubernetes/pull/18827 I have the >>>>>> following question. Since I am planning to use Kubernetes as a executing >>>>>> backend for the workflow execution environment I am building, at some >>>>>> point >>>>>> we will need to know the current state of a job. Well, this time has >>>>>> come, >>>>>> and for now the only thing I am able to do is polling the API ( >>>>>> kubernetes.io/docs/api-reference/extensions/v1beta1/operations/). >>>>>> Taking into account that this is unacceptable, and I only did it for >>>>>> testing purposes, what is the recommended way in Kubernetes for doing so? >>>>>> (Keep in mind that I should do it from inside the cluster, no kubectl >>>>>> command) If it is not possible right now, will it be possible soon? I am >>>>>> working on a quite big project and since it is starting, and the first >>>>>> problems we are facing are related to this issues, it is time to decide >>>>>> if >>>>>> I continue with Kubernetes or I have to start looking for alternatives. >>>>>> >>>>>> Thanks for your commitment >>>>>> >>>>>> On Wednesday, 12 October 2016 23:54:16 UTC+2, Derek Carr wrote: >>>>>>> >>>>>>> The quota controller observes pod deletions, and queues the >>>>>>> pertinent quotas for processing in response. The replenishment should >>>>>>> be >>>>>>> relatively quick now. >>>>>>> >>>>>>> We actually exercise this behavior in e2e runs. >>>>>>> >>>>>>> 1. What version of Kubernetes are you running? >>>>>>> 2. How many quotas? >>>>>>> 3. How many pods? >>>>>>> >>>>>>> My guess is that there is a conflict of some kind during the quota >>>>>>> sync, and the 5 minutes you are seeing is a full re-sync interval for >>>>>>> the >>>>>>> quota controller. >>>>>>> >>>>>>> It would be good to try to determine what the conflict is so we can >>>>>>> fix. >>>>>>> >>>>>>> If you have a quick set of steps I can run to reproduce that would >>>>>>> be great. >>>>>>> >>>>>>> I recommend opening an issue with the above details and cc >>>>>>> @derekwaynecarr >>>>>>> >>>>>>> Thanks, >>>>>>> Derek >>>>>>> >>>>>>> On Wednesday, October 12, 2016 at 3:18:07 PM UTC-4, David Eads wrote: >>>>>>>> >>>>>>>> Quota reconcilation of pods was supposed to run at a faster rate. >>>>>>>> I'm not sure if 5 minutes is that faster rate. >>>>>>>> >>>>>>>> *Derek:* Do you recall where and how the fast pod replenishment is? >>>>>>>> >>>>>>>> >>>>>>>> On Wednesday, October 12, 2016 at 1:28:45 PM UTC-4, Daniel Smith >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Ah, you're blocked because the quota check reconciles slowly. The >>>>>>>>> quick fix is probably to just get more quota. >>>>>>>>> >>>>>>>>> +David who may know of an already existing issue about this. >>>>>>>>> >>>>>>>>> On Wed, Oct 12, 2016 at 5:39 AM, Diego Rodríguez Rodríguez < >>>>>>>>> diego...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> I have already configured my cluster to test what you have >>>>>>>>>> stated. What I have done so far is to create a ResourceQuota which >>>>>>>>>> takes >>>>>>>>>> care that there will not be more than 4 pods running. Then I ask for >>>>>>>>>> say 20 >>>>>>>>>> jobs to be completed. >>>>>>>>>> >>>>>>>>>> What happens in reality is that the first 4 jobs are completed >>>>>>>>>> and then, even though the pods are completed and therefore resources >>>>>>>>>> are >>>>>>>>>> available, it takes some minutes (around 4-5 minutes) to have >>>>>>>>>> another 4 >>>>>>>>>> jobs being completed. Indeed, what you said is true, however it is no >>>>>>>>>> practical because a delay of minutes can not be assumed. >>>>>>>>>> The waiting jobs events look like this: >>>>>>>>>> `FailedCreate Error creating: pods >>>>>>>>>> "d59fa9b2-6c6b-4dc5-b149-7f89b35421bf-10-" is forbidden: >>>>>>>>>> Exceeded quota: compute-resources, requested: pods=1, used: pods=4, >>>>>>>>>> limited: pods=4` >>>>>>>>>> So, it fails because there are no resources but it is trying it >>>>>>>>>> again only after some minutes. This behavior is far from the desired >>>>>>>>>> one, >>>>>>>>>> which is relaying on Kubernetes for the execution of a set of tasks >>>>>>>>>> no >>>>>>>>>> matter the resources available, just getting them executed as soon as >>>>>>>>>> possible. >>>>>>>>>> >>>>>>>>>> thanks >>>>>>>>>> >>>>>>>>>> On Monday, 10 October 2016 19:24:46 UTC+2, Daniel Smith wrote: >>>>>>>>>>> >>>>>>>>>>> If the system lacks resources, Pods will remain "Pending" until >>>>>>>>>>> resources become available. Cluster scalers may use pending pods as >>>>>>>>>>> a >>>>>>>>>>> signal that the cluster size should be increased. >>>>>>>>>>> >>>>>>>>>>> On Mon, Oct 10, 2016 at 5:58 AM, Diego Rodríguez Rodríguez < >>>>>>>>>>> diego...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hello, I have a doubt about how Kubernetes' jobs are handled. >>>>>>>>>>>> >>>>>>>>>>>> I have a queue to submit certain amount of incoming tasks >>>>>>>>>>>> (Celery and RabbitMQ take care of this). Each one of this tasks >>>>>>>>>>>> is, in >>>>>>>>>>>> fact, a *workflow* which will be executed in a worker (Celery >>>>>>>>>>>> worker with a DAG executor inside). Each step of the *workflow* >>>>>>>>>>>> is a Docker image with an input file and an output file. >>>>>>>>>>>> >>>>>>>>>>>> My question is, if I submit jobs from the workflow engine >>>>>>>>>>>> directly to the Kubernetes API, what happens I at some point there >>>>>>>>>>>> are no >>>>>>>>>>>> more resources? Will the remaining tasks be kept or will they be >>>>>>>>>>>> lost? My >>>>>>>>>>>> goal is to treat Kubernetes' jobs as a black box to submit works >>>>>>>>>>>> to. This >>>>>>>>>>>> works are of a very different and heterogeneous nature and I >>>>>>>>>>>> wouldn't need >>>>>>>>>>>> to bother with what is inside them because they are dockerized and >>>>>>>>>>>> executed >>>>>>>>>>>> by Kubernetes at some point. >>>>>>>>>>>> >>>>>>>>>>>> To sum up, I already have the layer of Celery workers with a >>>>>>>>>>>> DAG executor inside which knows the right order of the tasks and >>>>>>>>>>>> knows how >>>>>>>>>>>> to manage everything concerning the *workflow*. These >>>>>>>>>>>> components will submit jobs (through Kubernetes API) and will wait >>>>>>>>>>>> for them >>>>>>>>>>>> to be executed and then continue with the remaining tasks asking >>>>>>>>>>>> Kubernetes >>>>>>>>>>>> to run them until the *workflow* ends. >>>>>>>>>>>> >>>>>>>>>>>> I have read about a somehow related issue in Github: >>>>>>>>>>>> https://github.com/kubernetes/kubernetes/pull/17787 >>>>>>>>>>>> https://github.com/kubernetes/kubernetes/pull/18827 >>>>>>>>>>>> >>>>>>>>>>>> I couldn't determine if this is closed or it is coming in a >>>>>>>>>>>> future release. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thanks in advance! >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> Google Groups "Kubernetes user discussion and Q&A" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>> it, send an email to kubernetes-use...@googlegroups.com. >>>>>>>>>>>> To post to this group, send email to >>>>>>>>>>>> kubernet...@googlegroups.com. >>>>>>>>>>>> Visit this group at https://groups.google.com/grou >>>>>>>>>>>> p/kubernetes-users. >>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "Kubernetes user discussion and Q&A" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to kubernetes-use...@googlegroups.com. >>>>>>>>>> To post to this group, send email to kubernet...@googlegroups.com >>>>>>>>>> . >>>>>>>>>> Visit this group at https://groups.google.com/grou >>>>>>>>>> p/kubernetes-users. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Kubernetes user discussion and Q&A" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to kubernetes-users+unsubscr...@googlegroups.com. >>>>>> To post to this group, send email to kubernetes-users@googlegroups. >>>>>> com. >>>>>> Visit this group at https://groups.google.com/group/kubernetes-users. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "Kubernetes user discussion and Q&A" group. >>>>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>>>> pic/kubernetes-users/qd5Ua2UqQNo/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> kubernetes-users+unsubscr...@googlegroups.com. >>>>> To post to this group, send email to kubernetes-users@googlegroups.com >>>>> . >>>>> Visit this group at https://groups.google.com/group/kubernetes-users. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Kubernetes user discussion and Q&A" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to kubernetes-users+unsubscr...@googlegroups.com. >>>> To post to this group, send email to kubernetes-users@googlegroups.com. >>>> Visit this group at https://groups.google.com/group/kubernetes-users. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Kubernetes user discussion and Q&A" group. >>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>> pic/kubernetes-users/qd5Ua2UqQNo/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> kubernetes-users+unsubscr...@googlegroups.com. >>> To post to this group, send email to kubernetes-users@googlegroups.com. >>> Visit this group at https://groups.google.com/group/kubernetes-users. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> > -- > You received this message because you are subscribed to the Google Groups > "Kubernetes user discussion and Q&A" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to kubernetes-users+unsubscr...@googlegroups.com. > To post to this group, send email to kubernetes-users@googlegroups.com. > Visit this group at https://groups.google.com/group/kubernetes-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
