On 07/12/2018 08:37 AM, Clark Boylan wrote:
On Wed, Jul 11, 2018, at 9:34 PM, Tony Breeds wrote:
Hi Folks,
We have a pit of a problem in openstack/requirements and I'd liek to
chat about it.
Currently when we generate constraints we create a venv for each
(system) python supplied on the command line, install all of
global-requirements into that venv and capture the pip freeze.
Where this falls down is if we want to generate a freeze for python 3.4
and 3.5 we need an image that has both of those. We cheated and just
'clone' them so if python3 is 3.4 we copy the results to 3.5 and vice
versa. This kinda worked for a while but it has drawbacks.
I can see a few of options:
1. Build pythons from source and use that to construct the venv
[please no]
Fungi mentions that 3.3 and 3.4 don't build easily on modern linux distros.
However, 3.3 and 3.4 are also unsupported by Python at this point, maybe we can
ignore them and focus on 3.5 and forward? We don't build new freeze lists for
the stable branches, this is just a concern for master right?
FWIW, I use pyenv for python versions on my laptop and love it. I've
completely given up on distro-provided python for my own usage.
2. Generate the constraints in an F28 image. My F28 has ample python
versions:
- /usr/bin/python2.6
- /usr/bin/python2.7
- /usr/bin/python3.3
- /usr/bin/python3.4
- /usr/bin/python3.5
- /usr/bin/python3.6
- /usr/bin/python3.7
I don't know how valid this still is but in the past fedora images
have been seen as unstable and hard to keep current. If that isn't
still the feeling then we could go down this path. Currently there a
few minor problems with bindep.txt on fedora and generate-constraints
doesn't work with py3 but these are pretty minor really.
I think most of the problems with Fedora stability are around bringing up a
new Fedora every 6 months or so. They tend to change sufficiently within that
time period to make this a fairly involved exercise. But once working they work
for the ~13 months of support they offer. I know Paul Belanger would like to
iterate more quickly and just keep the most recent Fedora available (rather
than ~2).
3. Use docker images for python and generate the constraints with
them. I've hacked up something we could use as a base for that in:
https://review.openstack.org/581948
There are lots of open questions:
- How do we make this nodepool/cloud provider friendly ?
* Currently the containers just talk to the main debian mirrors.
Do we have debian packages? If so we could just do sed magic.
http://$MIRROR/debian (http://mirror.dfw.rax.openstack.org/debian for example)
should be a working amd64 debian package mirror.
- Do/Can we run a registry per provider?
We do not, but we do have a caching dockerhub registry proxy in each
region/provider. http://$MIRROR:8081/registry-1.docker if using older docker
and http://$MIRROR:8082 for current docker. This was a compromise between
caching the Internet and reliability.
there is also
https://review.openstack.org/#/c/580730/
which adds a role to install docker and configure it to use the correct
registry.
- Can we generate and caches these images and only run pip install -U
g-r to speed up the build
Between cached upstream python docker images and prebuilt wheels mirrored in
every cloud provider region I wonder if this will save a significant amount of
time? May be worth starting without this and working from there if it remains
slow.
- Are we okay with using docker this way?
Should be fine, particularly if we are consuming the official Python images.
Agree. python:3.6 and friends are great.
I like #2 the most but I wanted to seek wider feedback.
I think each proposed option should work as long as we understand the
limitations each presents. #2 should work fine if we have individuals
interested and able to spin up new Fedora images and migrate jobs to that image
after releases happen.
Clark
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
