On 19/10/18 12:30 PM, Clark Boylan wrote:
On Fri, Oct 19, 2018, at 8:17 AM, Zane Bitter wrote:
Unit Tests
----------
For unit tests, the most important thing is to test on the versions of
Python we target. It's less important to be using the exact distro that
we want to target, because unit tests generally won't interact with
stuff outside of Python.
I'd like to propose that we handle this by setting up a unit test
template in openstack-zuul-jobs for each release. So for Stein we'd have
openstack-python3-stein-jobs. This template would contain:
Because zuul config is branch specific we could set up every project to use a
`openstack-python3-jobs` template then define that template differently on each
branch. This would mean you only have to update the location where the template
is defined and not need to update every other project after cutting a stable
branch. I would suggest we take advantage of that to reduce churn.
There was a reason I didn't propose that approach: in practice you can't
add a new gating test to a centralised zuul template definition. If you
do, many projects will break because the change is not self-testing. At
best you'll be pitchforked by an angry mob of people who can't get
anything but py37 fixes through the gate, and at worst they'll all stop
using the template to get unblocked and then never go back to it.
We don't need everyone to cut over at the same time. We just need them
to do it in the space of one release cycle. One patch every 6 months is
not an excessive amount of churn.
* A voting gate job for the highest minor version of py3 we want to
support in that release.
* A voting gate job for the lowest minor version of py3 we want to
support in that release.
* A periodic job for any interim minor releases.
* (Starting late in the cycle) a non-voting check job for the highest
minor version of py3 we want to support in the *next* release (if
different), on the master branch only.
So, for example, (and this is still under active debate) for Stein we
might have gating jobs for py35 and py37, with a periodic job for py36.
The T jobs might only have voting py36 and py37 jobs, but late in the T
cycle we might add a non-voting py38 job on master so that people who
haven't switched to the U template yet can see what, if anything,
they'll need to fix.
We'll run the unit tests on any distro we can find that supports the
version of Python we want. It could be a non-LTS Ubuntu, Fedora, Debian
unstable, whatever it takes. We won't wait for an LTS Ubuntu to have a
particular Python version before trying to test it.
Before the start of each cycle, the TC would determine which range of
versions we want to support, on the basis of the latest one we can find
in any distro and the earliest one we're likely to need in one of the
supported Linux distros. There will be a project-wide goal to switch the
testing template from e.g. openstack-python3-stein-jobs to
openstack-python3-treasure-jobs for every repo before the end of the
cycle. We'll have goal champions as usual following up and helping teams
with the process. We'll know where the problem areas are because we'll
have added non-voting jobs for any new Python versions to the previous
release's template.
I don't know that this needs to be a project wide goal if you can just update
the template on the master branch where the template is defined. Do that then
every project is now running with the up to date version of the template. We
should probably advertise when this is happening with some links to python
version x.y breakages/features, but the process itself should be quick.
Either way, it'll be project teams themselves fixing any broken tests
due to a new version being added. So we can either have a formal
project-wide goal where we project-manage that process across the space
of a release, or a de-facto project-wide goal where we break everybody
and then nothing gets merged until they fix it.
As for python version range selection I worry that that the criteria about
relies on too much guesswork.
Some guesswork is going to be inevitable, unfortunately, (we have no way
of knowing what will be in CentOS 8, for example) but I agree that we
should try to tighten up the criteria as much as possible.
I do think we should do our best to test future incoming versions of python
even while not officially supporting them. We will have to support them at some
point, either directly or via some later version that includes the changes from
that intermediate version.
+1, I think we should try to add support for higher versions as soon as
possible. It may take a long time to get into an LTS release, but
there's bound to be _some_ distro out there where people want to use it.
(Case in point: Debian really wanted py37 support in Rocky, at which
point a working 3.7 wasn't even available in _any_ Ubuntu release, let
alone an LTS). That's why I said "the latest one we can find in any
distro" - if we have any way to test it at all then we should.
Could the criteria be:
Support the lowest version of python on a supported distro release and the
highest version of python on a supported distro
As of now we can't objectively determine the minimum version because it
isn't the future yet. That may change once every distro is on Python 3
though.
and test (but not support so we can drop testing for this python version on
stable branches) the current latest release of python?
That's certainly worth considering; we could tighten up the range on
stable branches. I think we'd need to hear from Ubuntu and Debian folks
what they think about that. My guess is that they'd prefer even stable
branches to continue testing recent versions, so that they could be used
on Debian unstable and on Ubuntu between LTS releases.
This is objective and doesn't require anyone to guess at what versions need to
be supported.
Integration Tests
-----------------
Integration tests do test, amongst other things, integration with
non-openstack-supplied things in the distro, so it's important that we
test on the actual distros we have identified as popular.[2] It's also
important that every project be testing on the same distro at the end of
a release, so we can be sure they all work together for users.
When a new release of CentOS or a new LTS release of Ubuntu comes out,
the TC will create a project-wide goal for the *next* release cycle to
switch all integration tests over to that distro. It's up to individual
projects to make the switch for the tests that they own (e.g. it'd be
the QA team for Tempest, but other individual projects for their own
jobs). Again, there'll be a goal champion to monitor and follow up.
[1]
https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation-timeline.html
[2]
https://governance.openstack.org/tc/reference/project-testing-interface.html#linux-distributions
Overall this approach seems fine. It is basically the approach we've used in
the past with the addition of explicit testing of future python (which we've
never really done before). I do think we need to avoid letting every project
move at their own pace. This is what we did with the last python and distro
switchover (trusty to xenial) and a number of projects failed to get it done
within the release cycle. Instead I think we should rely on shared branch
specific zuul templates that can be updated centrally when the next release is
cut.
For those just joining, we discussed this on IRC yesterday.[1] fungi
mentioned that we tried two different approaches for precise->trusty and
trusty->xenial, and they both failed in different ways. The first time
infra gave teams time to prepare and then eventually cut the build over
for everyone, with the result that lots of things broke. The second time
infra allowed teams to switch over in their own time, with the result
that a lot of things released with tests running on an outdated LTS release.
We do have one new tool in our toolbox: project-wide goals. They're not
magically going to solve the problem, but at least they give us some
visibility into what has and has not happened. Perhaps we could provide
periodic, rather than experimental, jobs to test with so that the goal
champions can track where the likely problems are in the lead-up to a
switchover? As far as the LTS distro goes, I don't have a strong opinion
on which approach is the least worst. I think if we want to have a
switchover (just after milestone-2 maybe?) then we should just write
that into the project-wide goal.
cheers,
Zane.
[1]
http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2018-10-18.log.html#t2018-10-18T15:16:13
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
