Re: [openstack-dev] [devstack][infra] pip vs psutil

2018-04-16 Thread Jens Harbott 
2018-04-16 7:46 GMT+00:00 Ian Wienand :
> On 04/15/2018 09:32 PM, Gary Kotton wrote:
>>
>> The gate is currently broken with
>>  https://launchpad.net/bugs/1763966.
>> https://review.openstack.org/#/c/561427/
>>  Can unblock us in the short term. Any other ideas?
>
>
> I'm thinking this is probably along the lines of the best idea.  I
> left a fairly long comment on this in [1], but the root issue here is
> that if a system package is created using distutils (rather than
> setuptools) we end up with this problem with pip10.
>
> That means the problem occurs when we a) try to overwrite a system
> package and b) that package has been created using distutils.  This
> means it is a small(er) subset of packages that cause this problem.
> Ergo, our best option might be to see if we can avoid such packages on
> a one-by-one basis, like here.
>
> In some cases, we could just delete the .egg-info file, which is
> approximately what was happening before anyway.
>
> In this particular case, the psutils package is used by glance & the
> peakmem tracker.  Under USE_PYTHON3, devstack's pip_install_gr only
> installs the python3 library; however the peakmem tracker always uses
> python2 -- leaing to missing library the failures in [2].  I have two
> thoughts; either install for both python2 & 3 always [3] or make
> peakmem tracker obey USE_PYTHON3 [4].  We can discuss the approach in
> the reviews.
>
> The other option is to move everything to virtualenv's, so we never
> conflict with a system package, as suggested by clarkb [5] or
> pabelanger [6].  These are more invasive changes, but also arguably
> more correct.
>
> Note diskimage-builder, and hence our image generation for some
> platforms, is also broken.  Working on that in [7].

The cap in devstack has been merged in master and stable/queens, other
merges are being help up by unstable volume checks or so it seems.

There is also another issue caused by pip 10 treating some former
warning as error now. I've tried to list all "global" (Infra+QA)
related issues in [8], feel free to amend as needed.

[8] https://etherpad.openstack.org/p/pip10-mitigation

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack][infra] pip vs psutil

2018-04-16 Thread Ian Wienand 

On 04/15/2018 09:32 PM, Gary Kotton wrote:

The gate is currently broken with
 https://launchpad.net/bugs/1763966. https://review.openstack.org/#/c/561427/
 Can unblock us in the short term. Any other ideas?


I'm thinking this is probably along the lines of the best idea.  I
left a fairly long comment on this in [1], but the root issue here is
that if a system package is created using distutils (rather than
setuptools) we end up with this problem with pip10.

That means the problem occurs when we a) try to overwrite a system
package and b) that package has been created using distutils.  This
means it is a small(er) subset of packages that cause this problem.
Ergo, our best option might be to see if we can avoid such packages on
a one-by-one basis, like here.

In some cases, we could just delete the .egg-info file, which is
approximately what was happening before anyway.

In this particular case, the psutils package is used by glance & the
peakmem tracker.  Under USE_PYTHON3, devstack's pip_install_gr only
installs the python3 library; however the peakmem tracker always uses
python2 -- leaing to missing library the failures in [2].  I have two
thoughts; either install for both python2 & 3 always [3] or make
peakmem tracker obey USE_PYTHON3 [4].  We can discuss the approach in
the reviews.

The other option is to move everything to virtualenv's, so we never
conflict with a system package, as suggested by clarkb [5] or
pabelanger [6].  These are more invasive changes, but also arguably
more correct.

Note diskimage-builder, and hence our image generation for some
platforms, is also broken.  Working on that in [7].

-i


[1] https://github.com/pypa/pip/issues/4805#issuecomment-340987536
[2] https://review.openstack.org/561427
[3] https://review.openstack.org/561524
[4] https://review.openstack.org/561525
[5] https://review.openstack.org/558930
[6] https://review.openstack.org/#/c/552939
[7] https://review.openstack.org/#/c/561479/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack][infra] pip vs psutil

2018-04-16 Thread Slawomir Kaplonski 
Right. Thx Gary :)

> Wiadomość napisana przez Gary Kotton  w dniu 16.04.2018, 
> o godz. 09:14:
> 
> Hi,
> I think that we need https://review.openstack.org/561471 until we have a 
> proper solution.
> Thanks
> Gary
> 
> On 4/16/18, 10:13 AM, "Slawomir Kaplonski"  wrote:
> 
>Hi,
> 
>I just wanted to ask if there is any ongoing work on 
> https://bugs.launchpad.net/devstack/+bug/1763966 to fix grenade failures? It 
> looks that e.g. all grenade jobs in neutron are broken currently :/
> 
>> Wiadomość napisana przez Gary Kotton  w dniu 15.04.2018, 
>> o godz. 13:32:
>> 
>> Hi,
>> The gate is currently broken with https://launchpad.net/bugs/1763966. 
>> https://review.openstack.org/#/c/561427/ Can unblock us in the short term. 
>> Any other ideas?
>> Thanks
>> Gary
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
>— 
>Best regards
>Slawek Kaplonski
>skapl...@redhat.com
> 
> 
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

— 
Best regards
Slawek Kaplonski
skapl...@redhat.com


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack][infra] pip vs psutil

2018-04-16 Thread Gary Kotton 
Hi,
I think that we need https://review.openstack.org/561471 until we have a proper 
solution.
Thanks
Gary

On 4/16/18, 10:13 AM, "Slawomir Kaplonski"  wrote:

Hi,

I just wanted to ask if there is any ongoing work on 
https://bugs.launchpad.net/devstack/+bug/1763966 to fix grenade failures? It 
looks that e.g. all grenade jobs in neutron are broken currently :/

> Wiadomość napisana przez Gary Kotton  w dniu 
15.04.2018, o godz. 13:32:
> 
> Hi,
> The gate is currently broken with https://launchpad.net/bugs/1763966. 
https://review.openstack.org/#/c/561427/ Can unblock us in the short term. Any 
other ideas?
> Thanks
> Gary
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

— 
Best regards
Slawek Kaplonski
skapl...@redhat.com


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack][infra] pip vs psutil

2018-04-16 Thread Slawomir Kaplonski 
Hi,

I just wanted to ask if there is any ongoing work on 
https://bugs.launchpad.net/devstack/+bug/1763966 to fix grenade failures? It 
looks that e.g. all grenade jobs in neutron are broken currently :/

> Wiadomość napisana przez Gary Kotton  w dniu 15.04.2018, 
> o godz. 13:32:
> 
> Hi,
> The gate is currently broken with https://launchpad.net/bugs/1763966. 
> https://review.openstack.org/#/c/561427/ Can unblock us in the short term. 
> Any other ideas?
> Thanks
> Gary
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

— 
Best regards
Slawek Kaplonski
skapl...@redhat.com


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev