Re: [openstack-dev] [all] propose to upgrade python kubernetes (the k8s python client) to 4.0.0 which breaks oslo.service

[Lingxian Kong]

Thanks, leyal. I've already changed the service framework from oslo.service
to cotyledon https://review.openstack.org/#/c/530428/, and it works
perfectly fine.


Cheers,
Lingxian Kong (Larry)

On Mon, Jan 8, 2018 at 2:47 AM, Eyal Leshem  wrote:

> Hi Lingxian,
>
> I uploaded a patch for kuryr-kubernetes  that monkey-patch the ThreadPool
> with
> GreenPool (https://review.openstack.org/#/c/530655/4/kuryr_kubernetes/
> thread_pool_patch.py).
>
> It's support only apply_async - but that should be enough for k8s.
>
> That can be dangers - if you use ThreadPool in other places in your code,
> but in such case you can't run with eventlet anyway.
>
> hope that helps,
> leyal
>
>
>
>
> On 4 January 2018 at 23:45, Lingxian Kong  wrote:
>
>> On Tue, Jan 2, 2018 at 1:56 AM, Eyal Leshem  wrote:
>>
>>> Hi ,
>>>
>>> According to https://github.com/eventlet/eventlet/issues/147 - it's
>>> looks that eventlet
>>> has issue with "multiprocessing.pool".
>>>
>>> The ThreadPool used in code that auto-generated by swagger.
>>>
>>> Possible workaround for that is to monky-patch the client library ,
>>> and replace the pool with greenpool.
>>>
>>
>> Hi, leyal, I'm not very familar with eventlet, but how can I monkey patch
>> kubernetes python lib?
>> The only way I can see now is to replace oslo.service with something
>> else, e.g. cotyledon, avoid to use eventlet, that's a signaficant change
>> though. I also found this bug https://bugs.launchpad.net
>> /taskflow/+bug/1225275 in taskflow, they chose to not use
>> multiprocessing module.
>>
>> Any other suggestions are welcomed!
>>
>>
>>>
>>> If someone has better workaround, please share that with us :)
>>>
>>> btw , I don't think that should be treated as compatibility issue
>>> in the client python as it's an eventlet issue..
>>>
>>> Thanks ,
>>> leyal
>>>
>>
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] propose to upgrade python kubernetes (the k8s python client) to 4.0.0 which breaks oslo.service

[Eyal Leshem]

Hi Lingxian,

I uploaded a patch for kuryr-kubernetes  that monkey-patch the ThreadPool
with
GreenPool (
https://review.openstack.org/#/c/530655/4/kuryr_kubernetes/thread_pool_patch.py
).

It's support only apply_async - but that should be enough for k8s.

That can be dangers - if you use ThreadPool in other places in your code,
but in such case you can't run with eventlet anyway.

hope that helps,
leyal




On 4 January 2018 at 23:45, Lingxian Kong  wrote:

> On Tue, Jan 2, 2018 at 1:56 AM, Eyal Leshem  wrote:
>
>> Hi ,
>>
>> According to https://github.com/eventlet/eventlet/issues/147 - it's
>> looks that eventlet
>> has issue with "multiprocessing.pool".
>>
>> The ThreadPool used in code that auto-generated by swagger.
>>
>> Possible workaround for that is to monky-patch the client library ,
>> and replace the pool with greenpool.
>>
>
> Hi, leyal, I'm not very familar with eventlet, but how can I monkey patch
> kubernetes python lib?
> The only way I can see now is to replace oslo.service with something else,
> e.g. cotyledon, avoid to use eventlet, that's a signaficant change though.
> I also found this bug https://bugs.launchpad.net/taskflow/+bug/1225275 in
> taskflow, they chose to not use multiprocessing module.
>
> Any other suggestions are welcomed!
>
>
>>
>> If someone has better workaround, please share that with us :)
>>
>> btw , I don't think that should be treated as compatibility issue
>> in the client python as it's an eventlet issue..
>>
>> Thanks ,
>> leyal
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] propose to upgrade python kubernetes (the k8s python client) to 4.0.0 which breaks oslo.service

[Lingxian Kong]

On Tue, Jan 2, 2018 at 1:56 AM, Eyal Leshem  wrote:

> Hi ,
>
> According to https://github.com/eventlet/eventlet/issues/147 - it's looks
> that eventlet
> has issue with "multiprocessing.pool".
>
> The ThreadPool used in code that auto-generated by swagger.
>
> Possible workaround for that is to monky-patch the client library ,
> and replace the pool with greenpool.
>

Hi, leyal, I'm not very familar with eventlet, but how can I monkey patch
kubernetes python lib?
The only way I can see now is to replace oslo.service with something else,
e.g. cotyledon, avoid to use eventlet, that's a signaficant change though.
I also found this bug https://bugs.launchpad.net/taskflow/+bug/1225275 in
taskflow, they chose to not use multiprocessing module.

Any other suggestions are welcomed!


>
> If someone has better workaround, please share that with us :)
>
> btw , I don't think that should be treated as compatibility issue
> in the client python as it's an eventlet issue..
>
> Thanks ,
> leyal
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] propose to upgrade python kubernetes (the k8s python client) to 4.0.0 which breaks oslo.service

[Eyal Leshem]

Hi ,

According to https://github.com/eventlet/eventlet/issues/147 - it's looks
that eventlet
has issue with "multiprocessing.pool".

The ThreadPool used in code that auto-generated by swagger.

Possible workaround for that is to monky-patch the client library ,
and replace the pool with greenpool.

If someone has better workaround, please share that with us :)

btw , I don't think that should be treated as compatibility issue
in the client python as it's an eventlet issue..

Thanks ,
leyal



On 1 January 2018 at 12:08, Lingxian Kong  wrote:

> I edited the topic just for attention.
>
> However, the new kubernetes client version breaks the services that's
> using oslo.service which relies on eventlet library. Some error logs below:
>
> (Pdb) n
> > /vagrant/qinling/qinling/orchestrator/kubernetes/
> manager.py(49)__init__()
> -> client = api_client.ApiClient(configuration=config)
> (Pdb) n
> Exception in thread Thread-2:
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
> self.run()
>   File "/usr/lib/python2.7/threading.py", line 754, in run
> self.__target(*self.__args, **self.__kwargs)
>   File "/usr/lib/python2.7/multiprocessing/pool.py", line 325, in
> _handle_workers
> while thread._state == RUN or (pool._cache and thread._state !=
> TERMINATE):
> AttributeError: '_MainThread' object has no attribute '_state'
>
> I did a google search, found this: https://github.com/
> eventlet/eventlet/issues/147
>
> multiprocessing.pool was introduced since 4.0.0 (threading lib was used
> before)
>
> I assume this is an backward incompatible change.
>
> Any suggestion?
>
> Cheers,
> Lingxian Kong (Larry)
>
> On Wed, Dec 13, 2017 at 10:41 PM, Eyal Leshem  wrote:
>
>> Hi Lingxian,
>>
>> It's should -  under the assumption of uses only the v1 models ( and not
>> v1_alpha or v1_beta).
>> see : https://kubernetes.io/docs/reference/api-overview/
>>
>> thanks ,
>> leyal
>>
>> On 13 December 2017 at 11:16, Lingxian Kong  wrote:
>>
>>> hi, leyal,
>>>
>>> I suppose the upgrade is backward compatible, right?
>>>
>>>
>>> Cheers,
>>> Lingxian Kong (Larry)
>>>
>>> On Wed, Dec 13, 2017 at 8:51 PM, Eyal Leshem 
>>> wrote:
>>>
 Hi all ,

 In order to use kubernetes client that support network-policies ,
 we plan to upgrade the python kubernetes package  from  1.0.0 to 4.0.0.

 any objections ?

 thanks,
 leyal

 clarification:
 The purposed changed is for kubernetes-python-client - that called just
  "kubernetes" in  pypi

 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.op
 enstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


>>>
>>> 
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.op
>>> enstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] propose to upgrade python kubernetes (the k8s python client) to 4.0.0 which breaks oslo.service

[Lingxian Kong]

I edited the topic just for attention.

However, the new kubernetes client version breaks the services that's using
oslo.service which relies on eventlet library. Some error logs below:

(Pdb) n
> /vagrant/qinling/qinling/orchestrator/kubernetes/manager.py(49)__init__()
-> client = api_client.ApiClient(configuration=config)
(Pdb) n
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 325, in
_handle_workers
while thread._state == RUN or (pool._cache and thread._state !=
TERMINATE):
AttributeError: '_MainThread' object has no attribute '_state'

I did a google search, found this:
https://github.com/eventlet/eventlet/issues/147

multiprocessing.pool was introduced since 4.0.0 (threading lib was used
before)

I assume this is an backward incompatible change.

Any suggestion?

Cheers,
Lingxian Kong (Larry)

On Wed, Dec 13, 2017 at 10:41 PM, Eyal Leshem  wrote:

> Hi Lingxian,
>
> It's should -  under the assumption of uses only the v1 models ( and not
> v1_alpha or v1_beta).
> see : https://kubernetes.io/docs/reference/api-overview/
>
> thanks ,
> leyal
>
> On 13 December 2017 at 11:16, Lingxian Kong  wrote:
>
>> hi, leyal,
>>
>> I suppose the upgrade is backward compatible, right?
>>
>>
>> Cheers,
>> Lingxian Kong (Larry)
>>
>> On Wed, Dec 13, 2017 at 8:51 PM, Eyal Leshem  wrote:
>>
>>> Hi all ,
>>>
>>> In order to use kubernetes client that support network-policies ,
>>> we plan to upgrade the python kubernetes package  from  1.0.0 to 4.0.0.
>>>
>>> any objections ?
>>>
>>> thanks,
>>> leyal
>>>
>>> clarification:
>>> The purposed changed is for kubernetes-python-client - that called just
>>>  "kubernetes" in  pypi
>>>
>>> 
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.op
>>> enstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] propose to upgrade python kubernetes (the k8s python client) to 4.0.0

[Tony Breeds]

On Wed, Dec 13, 2017 at 10:16:29PM +1300, Lingxian Kong wrote:
> hi, leyal,
> 
> I suppose the upgrade is backward compatible, right?

By, semantic versioning, defintion 1.y.z is NOT compatible with 4.y.z.
That's what a major bump means.  And in this case we're doing 3 major
bumps.  The only question is "will the incompatibilities affect
OpenStack?'

I suggest we handle this the same way we do for other potentially
breakign chnages.

1. Create an openstack/requirements review to bump kubernetes in
   upper-constratins.txt
2. Creatre an openstack/requirements review to bump the minimum
   acceptable version in global-requirements.
3. Run a no-op Testing only change in all comumping projects that
   'Depends-On' the chnage from 1 preferably with a unique topic
4. Once testing from step 3 validates that it's a non-breaking change
   merge the chnage from 1.
5. After some time, typically 5 days, merge the chnage from 2.

If we discover an issue after that we'll deal with it as well as
possible but we did our best to verify that it shouldn't happen.

Yours Tony.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] propose to upgrade python kubernetes (the k8s python client) to 4.0.0

[mdulko]

On Wed, 2017-12-13 at 12:18 +0200, Eyal Leshem wrote:
> 3.0.0 is align with the kubernetes-1.7 api spec.
> In 1.8 there is some fields that had been added to the k8s-network-
> policy. 
> so for me 4.0.0 will be much better. 


Okay, then I don't see a point to block that, we can work around the
issues I've listed.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] propose to upgrade python kubernetes (the k8s python client) to 4.0.0

[Eyal Leshem]

3.0.0 is align with the kubernetes-1.7 api spec.
In 1.8 there is some fields that had been added to the k8s-network-policy.
so for me 4.0.0 will be much better.

thanks,
leyal

On 13 December 2017 at 11:49,  wrote:

> On Wed, 2017-12-13 at 09:51 +0200, Eyal Leshem wrote:
> > Hi all ,
> >
> > In order to use kubernetes client that support network-policies ,
> > we plan to upgrade the python kubernetes package  from  1.0.0 to
> > 4.0.0.
> >
> > any objections ?
>
> There's a couple of issues ([1], [2]) introduced in 4.0.0 that can
> affect kuryr-tempest-plugin and kuryr-kubernetes in the future. Are we
> able to stick with 3.0.0 for now or is it too old to support network
> policies?
>
> [1] https://github.com/kubernetes-incubator/client-python/issues/409
> [2] https://github.com/kubernetes-incubator/client-python/issues/415
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] propose to upgrade python kubernetes (the k8s python client) to 4.0.0

[Lingxian Kong]

cool, thanks!


Cheers,
Lingxian Kong (Larry)

On Wed, Dec 13, 2017 at 10:41 PM, Eyal Leshem  wrote:

> Hi Lingxian,
>
> It's should -  under the assumption of uses only the v1 models ( and not
> v1_alpha or v1_beta).
> see : https://kubernetes.io/docs/reference/api-overview/
>
> thanks ,
> leyal
>
> On 13 December 2017 at 11:16, Lingxian Kong  wrote:
>
>> hi, leyal,
>>
>> I suppose the upgrade is backward compatible, right?
>>
>>
>> Cheers,
>> Lingxian Kong (Larry)
>>
>> On Wed, Dec 13, 2017 at 8:51 PM, Eyal Leshem  wrote:
>>
>>> Hi all ,
>>>
>>> In order to use kubernetes client that support network-policies ,
>>> we plan to upgrade the python kubernetes package  from  1.0.0 to 4.0.0.
>>>
>>> any objections ?
>>>
>>> thanks,
>>> leyal
>>>
>>> clarification:
>>> The purposed changed is for kubernetes-python-client - that called just
>>>  "kubernetes" in  pypi
>>>
>>> 
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.op
>>> enstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] propose to upgrade python kubernetes (the k8s python client) to 4.0.0

[mdulko]

On Wed, 2017-12-13 at 09:51 +0200, Eyal Leshem wrote:
> Hi all ,
> 
> In order to use kubernetes client that support network-policies , 
> we plan to upgrade the python kubernetes package  from  1.0.0 to
> 4.0.0. 
> 
> any objections ? 

There's a couple of issues ([1], [2]) introduced in 4.0.0 that can
affect kuryr-tempest-plugin and kuryr-kubernetes in the future. Are we
able to stick with 3.0.0 for now or is it too old to support network
policies?

[1] https://github.com/kubernetes-incubator/client-python/issues/409
[2] https://github.com/kubernetes-incubator/client-python/issues/415

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] propose to upgrade python kubernetes (the k8s python client) to 4.0.0

[Eyal Leshem]

Hi Lingxian,

It's should -  under the assumption of uses only the v1 models ( and not
v1_alpha or v1_beta).
see : https://kubernetes.io/docs/reference/api-overview/

thanks ,
leyal

On 13 December 2017 at 11:16, Lingxian Kong  wrote:

> hi, leyal,
>
> I suppose the upgrade is backward compatible, right?
>
>
> Cheers,
> Lingxian Kong (Larry)
>
> On Wed, Dec 13, 2017 at 8:51 PM, Eyal Leshem  wrote:
>
>> Hi all ,
>>
>> In order to use kubernetes client that support network-policies ,
>> we plan to upgrade the python kubernetes package  from  1.0.0 to 4.0.0.
>>
>> any objections ?
>>
>> thanks,
>> leyal
>>
>> clarification:
>> The purposed changed is for kubernetes-python-client - that called just
>>  "kubernetes" in  pypi
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] propose to upgrade python kubernetes (the k8s python client) to 4.0.0

[Lingxian Kong]

hi, leyal,

I suppose the upgrade is backward compatible, right?


Cheers,
Lingxian Kong (Larry)

On Wed, Dec 13, 2017 at 8:51 PM, Eyal Leshem  wrote:

> Hi all ,
>
> In order to use kubernetes client that support network-policies ,
> we plan to upgrade the python kubernetes package  from  1.0.0 to 4.0.0.
>
> any objections ?
>
> thanks,
> leyal
>
> clarification:
> The purposed changed is for kubernetes-python-client - that called just
>  "kubernetes" in  pypi
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev