On 19/10/18 12:30 PM, Clark Boylan wrote:
On Fri, Oct 19, 2018, at 8:17 AM, Zane Bitter wrote:
Unit Tests

For unit tests, the most important thing is to test on the versions of
Python we target. It's less important to be using the exact distro that
we want to target, because unit tests generally won't interact with
stuff outside of Python.

I'd like to propose that we handle this by setting up a unit test
template in openstack-zuul-jobs for each release. So for Stein we'd have
openstack-python3-stein-jobs. This template would contain:

Because zuul config is branch specific we could set up every project to use a 
`openstack-python3-jobs` template then define that template differently on each 
branch. This would mean you only have to update the location where the template 
is defined and not need to update every other project after cutting a stable 
branch. I would suggest we take advantage of that to reduce churn.

There was a reason I didn't propose that approach: in practice you can't add a new gating test to a centralised zuul template definition. If you do, many projects will break because the change is not self-testing. At best you'll be pitchforked by an angry mob of people who can't get anything but py37 fixes through the gate, and at worst they'll all stop using the template to get unblocked and then never go back to it.

We don't need everyone to cut over at the same time. We just need them to do it in the space of one release cycle. One patch every 6 months is not an excessive amount of churn.

* A voting gate job for the highest minor version of py3 we want to
support in that release.
* A voting gate job for the lowest minor version of py3 we want to
support in that release.
* A periodic job for any interim minor releases.
* (Starting late in the cycle) a non-voting check job for the highest
minor version of py3 we want to support in the *next* release (if
different), on the master branch only.

So, for example, (and this is still under active debate) for Stein we
might have gating jobs for py35 and py37, with a periodic job for py36.
The T jobs might only have voting py36 and py37 jobs, but late in the T
cycle we might add a non-voting py38 job on master so that people who
haven't switched to the U template yet can see what, if anything,
they'll need to fix.

We'll run the unit tests on any distro we can find that supports the
version of Python we want. It could be a non-LTS Ubuntu, Fedora, Debian
unstable, whatever it takes. We won't wait for an LTS Ubuntu to have a
particular Python version before trying to test it.

Before the start of each cycle, the TC would determine which range of
versions we want to support, on the basis of the latest one we can find
in any distro and the earliest one we're likely to need in one of the
supported Linux distros. There will be a project-wide goal to switch the
testing template from e.g. openstack-python3-stein-jobs to
openstack-python3-treasure-jobs for every repo before the end of the
cycle. We'll have goal champions as usual following up and helping teams
with the process. We'll know where the problem areas are because we'll
have added non-voting jobs for any new Python versions to the previous
release's template.

I don't know that this needs to be a project wide goal if you can just update 
the template on the master branch where the template is defined. Do that then 
every project is now running with the up to date version of the template. We 
should probably advertise when this is happening with some links to python 
version x.y breakages/features, but the process itself should be quick.

Either way, it'll be project teams themselves fixing any broken tests due to a new version being added. So we can either have a formal project-wide goal where we project-manage that process across the space of a release, or a de-facto project-wide goal where we break everybody and then nothing gets merged until they fix it.

As for python version range selection I worry that that the criteria about 
relies on too much guesswork.

Some guesswork is going to be inevitable, unfortunately, (we have no way of knowing what will be in CentOS 8, for example) but I agree that we should try to tighten up the criteria as much as possible.

I do think we should do our best to test future incoming versions of python 
even while not officially supporting them. We will have to support them at some 
point, either directly or via some later version that includes the changes from 
that intermediate version.

+1, I think we should try to add support for higher versions as soon as possible. It may take a long time to get into an LTS release, but there's bound to be _some_ distro out there where people want to use it. (Case in point: Debian really wanted py37 support in Rocky, at which point a working 3.7 wasn't even available in _any_ Ubuntu release, let alone an LTS). That's why I said "the latest one we can find in any distro" - if we have any way to test it at all then we should.

Could the criteria be:
Support the lowest version of python on a supported distro release and the 
highest version of python on a supported distro

As of now we can't objectively determine the minimum version because it isn't the future yet. That may change once every distro is on Python 3 though.

and test (but not support so we can drop testing for this python version on 
stable branches) the current latest release of python?

That's certainly worth considering; we could tighten up the range on stable branches. I think we'd need to hear from Ubuntu and Debian folks what they think about that. My guess is that they'd prefer even stable branches to continue testing recent versions, so that they could be used on Debian unstable and on Ubuntu between LTS releases.

This is objective and doesn't require anyone to guess at what versions need to 
be supported.

Integration Tests

Integration tests do test, amongst other things, integration with
non-openstack-supplied things in the distro, so it's important that we
test on the actual distros we have identified as popular.[2] It's also
important that every project be testing on the same distro at the end of
a release, so we can be sure they all work together for users.

When a new release of CentOS or a new LTS release of Ubuntu comes out,
the TC will create a project-wide goal for the *next* release cycle to
switch all integration tests over to that distro. It's up to individual
projects to make the switch for the tests that they own (e.g. it'd be
the QA team for Tempest, but other individual projects for their own
jobs). Again, there'll be a goal champion to monitor and follow up.


Overall this approach seems fine. It is basically the approach we've used in 
the past with the addition of explicit testing of future python (which we've 
never really done before). I do think we need to avoid letting every project 
move at their own pace. This is what we did with the last python and distro 
switchover (trusty to xenial) and a number of projects failed to get it done 
within the release cycle. Instead I think we should rely on shared branch 
specific zuul templates that can be updated centrally when the next release is 

For those just joining, we discussed this on IRC yesterday.[1] fungi mentioned that we tried two different approaches for precise->trusty and trusty->xenial, and they both failed in different ways. The first time infra gave teams time to prepare and then eventually cut the build over for everyone, with the result that lots of things broke. The second time infra allowed teams to switch over in their own time, with the result that a lot of things released with tests running on an outdated LTS release.

We do have one new tool in our toolbox: project-wide goals. They're not magically going to solve the problem, but at least they give us some visibility into what has and has not happened. Perhaps we could provide periodic, rather than experimental, jobs to test with so that the goal champions can track where the likely problems are in the lead-up to a switchover? As far as the LTS distro goes, I don't have a strong opinion on which approach is the least worst. I think if we want to have a switchover (just after milestone-2 maybe?) then we should just write that into the project-wide goal.


[1] http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2018-10-18.log.html#t2018-10-18T15:16:13

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Reply via email to