On Thu, Apr 5, 2018 at 9:27 PM Clark Boylan <[email protected]> wrote:
> On Mon, Apr 2, 2018, at 9:13 AM, Clark Boylan wrote: > > On Mon, Apr 2, 2018, at 8:06 AM, Matthew Thode wrote: > > > On 18-03-31 15:00:27, Jeremy Stanley wrote: > > > > According to a notice[1] posted to the pypa-announce and > > > > distutils-sig mailing lists, pip 10.0.0.b1 is on PyPI now and 10.0.0 > > > > is expected to be released in two weeks (over the April 14/15 > > > > weekend). We know it's at least going to start breaking[2] DevStack > > > > and we need to come up with a plan for addressing that, but we don't > > > > know how much more widespread the problem might end up being so > > > > encourage everyone to try it out now where they can. > > > > > > > > > > I'd like to suggest locking down pip/setuptools/wheel like openstack > > > ansible is doing in > > > > https://github.com/openstack/openstack-ansible/blob/master/global-requirement-pins.txt > > > > > > We could maintain it as a separate constraints file (or infra could > > > maintian it, doesn't mater). The file would only be used for the > > > initial get-pip install. > > > > In the past we've done our best to avoid pinning these tools because 1) > > we've told people they should use latest for openstack to work and 2) it > > is really difficult to actually control what versions of these tools end > > up on your systems if not latest. > > > > I would strongly push towards addressing the distutils package deletion > > problem that we've run into with pip10 instead. One of the approaches > > thrown out that pabelanger is working on is to use a common virtualenv > > for devstack and avoid the system package conflict entirely. > > I was mistaken and pabelanger was working to get devstack's USE_VENV > option working which installs each service (if the service supports it) > into its own virtualenv. There are two big drawbacks to this. This first is > that we would lose coinstallation of all the openstack services which is > one way we ensure they all work together at the end of the day. The second > is that not all services in "base" devstack support USE_VENV and I doubt > many plugins do either (neutron apparently doesn't?). > > I've since worked out a change that passes tempest using a global > virtualenv installed devstack at https://review.openstack.org/#/c/558930/. > This needs to be cleaned up so that we only check for and install the > virtualenv(s) once and we need to handle mixed python2 and python3 > environments better (so that you can run a python2 swift and python3 > everything else). > > The other major issue we've run into is that nova file injection (which is > tested by tempest) seems to require either libguestfs or nbd. libguestfs > bindings for python aren't available on pypi and instead we get them from > system packaging. This means if we want libguestfs support we have to > enable system site packages when using virtualenvs. The alternative is to > use nbd which apparently isn't preferred by nova and doesn't work under > current devstack anyways. > > Why is this a problem? Well the new pip10 behavior that breaks devstack is > pip10's refusable to remove distutils installed packages. Distro packages > by and large are distutils packaged which means if you mix system packages > and pip installed packages there is a good chance something will break (and > it does break for current devstack). I'm not sure that using a virtualenv > with system site packages enabled will sufficiently protect us from this > case (but we should test it further). Also it feels wrong to enable system > packages in a virtualenv if the entire point is avoiding system python > packages. > > I'm not sure what the best option is here but if we can show that system > site packages with virtualenvs is viable with pip10 and people want to move > forward with devstack using a global virtualenv we can work to clean up > this change and make it mergeable. > Thanks Clark for looking into this. One of the things that will break using a global virtual env is the "all-plugin" Tempest tox environment, which is still used in a few places [0]. The "all-plugin" tox environment is deprecated anyways, so this may actually push things moving in the right direction. Some background the "all-plugin": Tempest plugins used to live in tree for many projects - for Tempest to discover those plugins, "all-plugin" installs Tempest in a virtual environment with system site-packages enabled. After the Tempest plugin community goal in Queens, most plugins are now hosted in a dedicated repository, and "all-plugin" should not be needed anymore. Andrea Frittoli (andreaf) [0] http://codesearch.openstack.org/?q=all-plugin&i=nope&files=&repos= > > Clark > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
