Re: [openstack-dev] Proposal for a process to keep up with Python releases
On 10/26/18 7:11 PM, Zane Bitter wrote: > On 26/10/18 5:09 AM, Thomas Goirand wrote: >> On 10/22/18 9:12 PM, Zane Bitter wrote: >>> On 22/10/18 10:33 AM, Thomas Goirand wrote: This can only happen if we have supporting distribution packages for it. IMO, this is a call for using Debian Testing or even Sid in the gate. >>> >>> It depends on which versions we choose to support, but if necessary yes. >> >> If what we want is to have early detection of problems with latest >> versions of Python, then there's not so many alternatives. > > I think a lot depends on the relative timing of the Python release, the > various distro release cycles, and the OpenStack release cycle. We > established that for 3.7 that's the only way we could have done it in > Rocky; for 3.8, who knows. No need for a crystal ball... Python 3.8 is scheduled to be released in summer 2019. As Buster is to be frozen early the same year, it should be out before it. So, there's a lot of chance that Python 3.8 will be in Debian Sid/Bullseye before anywhere else again, probably just after the release of the OpenStack T release, meaning it most likely will be broken again in Debian Sid. > I agree that bugs with future versions of Python are always worth fixing > ASAP, whether or not we are able to test them in the gate. :) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for a process to keep up with Python releases
Zane Bitter writes: > On 19/10/18 11:17 AM, Zane Bitter wrote: >> I'd like to propose that we handle this by setting up a unit test >> template in openstack-zuul-jobs for each release. So for Stein we'd have >> openstack-python3-stein-jobs. This template would contain: >> >> * A voting gate job for the highest minor version of py3 we want to >> support in that release. >> * A voting gate job for the lowest minor version of py3 we want to >> support in that release. >> * A periodic job for any interim minor releases. >> * (Starting late in the cycle) a non-voting check job for the highest >> minor version of py3 we want to support in the *next* release (if >> different), on the master branch only. >> >> So, for example, (and this is still under active debate) for Stein we >> might have gating jobs for py35 and py37, with a periodic job for py36. >> The T jobs might only have voting py36 and py37 jobs, but late in the T >> cycle we might add a non-voting py38 job on master so that people who >> haven't switched to the U template yet can see what, if anything, >> they'll need to fix. > > Just to make it easier to visualise, here is an example for how the Zuul > config _might_ look now if we had adopted this proposal during Rocky: > > https://review.openstack.org/611947 > > And instead of having a project-wide goal in Stein to add > `openstack-python36-jobs` to the list that currently includes > `openstack-python35-jobs` in each project's Zuul config[1], we'd have > had a goal to change `openstack-python3-rocky-jobs` to > `openstack-python3-stein-jobs` in each project's Zuul config. If we set up the template before we branch stein for T, we could generate a patch as part of the branching process. Doug > > - ZB > > > [1] > https://governance.openstack.org/tc/goals/stein/python3-first.html#python-3-6-unit-test-jobs > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for a process to keep up with Python releases
On 26/10/18 5:09 AM, Thomas Goirand wrote: On 10/22/18 9:12 PM, Zane Bitter wrote: On 22/10/18 10:33 AM, Thomas Goirand wrote: This can only happen if we have supporting distribution packages for it. IMO, this is a call for using Debian Testing or even Sid in the gate. It depends on which versions we choose to support, but if necessary yes. If what we want is to have early detection of problems with latest versions of Python, then there's not so many alternatives. I think a lot depends on the relative timing of the Python release, the various distro release cycles, and the OpenStack release cycle. We established that for 3.7 that's the only way we could have done it in Rocky; for 3.8, who knows. I don't really understand why you're writing that it "depends on which version we choose to support". The current version of the resolution[1] says that we'll choose the latest released version "we can feasibly use for testing", while making clear that availability in an Ubuntu LTS release is *not* a requirement for feasibility. But it doesn't require the TC to choose the latest version available from python.org if we're not able to build an image that we can successfully use for testing in time before the beginning of the release cycle. [1] https://review.openstack.org/613145 That's the kind of answer which I found very frustrating when I submit a bug, and I'm being replied "we don't support this version". My reasoning is, the earlier we detect and fix problems, the better, and that's orthogonal to to what version of Python we want to support. Delaying bugfix and latest Python version compat leads to nowhere, and best is to test with it if possible (even in a non-voting mode). I agree that bugs with future versions of Python are always worth fixing ASAP, whether or not we are able to test them in the gate. cheers, Zane. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for a process to keep up with Python releases
On 10/25/18 3:43 PM, Zane Bitter wrote: On 25/10/18 1:38 PM, William M Edmonds wrote: Zane Bitter wrote on 10/22/2018 03:12:46 PM: > On 22/10/18 10:33 AM, Thomas Goirand wrote: > > On 10/19/18 5:17 PM, Zane Bitter wrote: > >> Integration Tests > >> - > >> > >> Integration tests do test, amongst other things, integration with > >> non-openstack-supplied things in the distro, so it's important that we > >> test on the actual distros we have identified as popular.[2] It's also > >> important that every project be testing on the same distro at the end of > >> a release, so we can be sure they all work together for users. > > > > I find very disturbing to see the project only leaning toward these only > > 2 distributions. Why not SuSE & Debian? > > The bottom line is it's because targeting those two catches 88% of our > users. (For once I did not make this statistic up.) > > Also note that in practice I believe almost everything is actually > tested on Ubuntu LTS, and only TripleO is testing on CentOS. It's > difficult to imagine how to slot another distro into the mix without > doubling up on jobs. I think you meant 78%, assuming you were looking at the latest User Survey results [1], page 55. Still a hefty number. I never know how to read those weird 3-way bar charts they have in the user survey, but that actually adds up to 91% by the looks of it (I believe you forgot to count RHEL). The numbers were actually slightly lower in the full-year data for 2017 that I used (from https://www.openstack.org/analytics - I can't give you a direct link because Javascript ). It is important to note that the User Survey lumps all versions of a given OS together, whereas the TC reference [2] only considers the latest LTS/stable version. If the User Survey split out latests LTS/stable versions vs. others (e.g. Ubuntu 16.04 LTS), I expect we'd see Ubuntu 18.04 LTS + Centos 7 adding up to much less than 78%. This is true, although we don't know by how much. (FWIW I can almost guarantee that virtually all of the CentOS/RHEL users are on 7, but I'm sure the same is not the case for Ubuntu 16.04.) In this context I don't think the version matters though. The original question was why we are focusing our test efforts on Ubuntu and CentOS, and the answer is that ~90% of our users are on those platforms. The specific version they're on right now doesn't really matter - even if they're on an older one, chances are eventually they'll move to a newer release of that same OS. [1] https://www.openstack.org/assets/survey/April2017SurveyReport.pdf [2] https://governance.openstack.org/tc/reference/project-testing-interface.html#linux-distributions __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for a process to keep up with Python releases
On 10/22/18 9:12 PM, Zane Bitter wrote: > On 22/10/18 10:33 AM, Thomas Goirand wrote: >> This can only happen if we have supporting distribution packages for it. >> IMO, this is a call for using Debian Testing or even Sid in the gate. > > It depends on which versions we choose to support, but if necessary yes. If what we want is to have early detection of problems with latest versions of Python, then there's not so many alternatives. I don't really understand why you're writing that it "depends on which version we choose to support". That's the kind of answer which I found very frustrating when I submit a bug, and I'm being replied "we don't support this version". My reasoning is, the earlier we detect and fix problems, the better, and that's orthogonal to to what version of Python we want to support. Delaying bugfix and latest Python version compat leads to nowhere, and best is to test with it if possible (even in a non-voting mode). Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for a process to keep up with Python releases
On 25/10/18 1:38 PM, William M Edmonds wrote: Zane Bitter wrote on 10/22/2018 03:12:46 PM: > On 22/10/18 10:33 AM, Thomas Goirand wrote: > > On 10/19/18 5:17 PM, Zane Bitter wrote: > >> Integration Tests > >> - > >> > >> Integration tests do test, amongst other things, integration with > >> non-openstack-supplied things in the distro, so it's important that we > >> test on the actual distros we have identified as popular.[2] It's also > >> important that every project be testing on the same distro at the end of > >> a release, so we can be sure they all work together for users. > > > > I find very disturbing to see the project only leaning toward these only > > 2 distributions. Why not SuSE & Debian? > > The bottom line is it's because targeting those two catches 88% of our > users. (For once I did not make this statistic up.) > > Also note that in practice I believe almost everything is actually > tested on Ubuntu LTS, and only TripleO is testing on CentOS. It's > difficult to imagine how to slot another distro into the mix without > doubling up on jobs. I think you meant 78%, assuming you were looking at the latest User Survey results [1], page 55. Still a hefty number. I never know how to read those weird 3-way bar charts they have in the user survey, but that actually adds up to 91% by the looks of it (I believe you forgot to count RHEL). The numbers were actually slightly lower in the full-year data for 2017 that I used (from https://www.openstack.org/analytics - I can't give you a direct link because Javascript ). It is important to note that the User Survey lumps all versions of a given OS together, whereas the TC reference [2] only considers the latest LTS/stable version. If the User Survey split out latests LTS/stable versions vs. others (e.g. Ubuntu 16.04 LTS), I expect we'd see Ubuntu 18.04 LTS + Centos 7 adding up to much less than 78%. This is true, although we don't know by how much. (FWIW I can almost guarantee that virtually all of the CentOS/RHEL users are on 7, but I'm sure the same is not the case for Ubuntu 16.04.) [1] https://www.openstack.org/assets/survey/April2017SurveyReport.pdf [2] https://governance.openstack.org/tc/reference/project-testing-interface.html#linux-distributions __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for a process to keep up with Python releases
Zane Bitter wrote on 10/22/2018 03:12:46 PM: > On 22/10/18 10:33 AM, Thomas Goirand wrote: > > On 10/19/18 5:17 PM, Zane Bitter wrote: > >> Integration Tests > >> - > >> > >> Integration tests do test, amongst other things, integration with > >> non-openstack-supplied things in the distro, so it's important that we > >> test on the actual distros we have identified as popular.[2] It's also > >> important that every project be testing on the same distro at the end of > >> a release, so we can be sure they all work together for users. > > > > I find very disturbing to see the project only leaning toward these only > > 2 distributions. Why not SuSE & Debian? > > The bottom line is it's because targeting those two catches 88% of our > users. (For once I did not make this statistic up.) > > Also note that in practice I believe almost everything is actually > tested on Ubuntu LTS, and only TripleO is testing on CentOS. It's > difficult to imagine how to slot another distro into the mix without > doubling up on jobs. I think you meant 78%, assuming you were looking at the latest User Survey results [1], page 55. Still a hefty number. It is important to note that the User Survey lumps all versions of a given OS together, whereas the TC reference [2] only considers the latest LTS/stable version. If the User Survey split out latests LTS/stable versions vs. others (e.g. Ubuntu 16.04 LTS), I expect we'd see Ubuntu 18.04 LTS + Centos 7 adding up to much less than 78%. [1] https://www.openstack.org/assets/survey/April2017SurveyReport.pdf [2] https://governance.openstack.org/tc/reference/project-testing-interface.html#linux-distributions __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for a process to keep up with Python releases
There seems to be agreement that this is broadly a good direction to pursue, so I proposed a TC resolution. Let's shift discussion to the review: https://review.openstack.org/613145 cheers, Zane. On 19/10/18 11:17 AM, Zane Bitter wrote: There hasn't been a Python 2 release in 8 years, and during that time we've gotten used to the idea that that's the way things go. However, with the switch to Python 3 looming (we will drop support for Python 2 in the U release[1]), history is no longer a good guide: Python 3 releases drop as often as every year. We are already feeling the pain from this, as Linux distros have largely already completed the shift to Python 3, and those that have are on versions newer than the py35 we currently have in gate jobs. We have traditionally held to the principle that we want each release to support the latest release of CentOS and the latest LTS release of Ubuntu, as they existed at the beginning of the release cycle.[2] Currently this means in practice one version of py2 and one of py3, but in the future it will mean two, usually different, versions of py3. There are two separate issues that we need to address: unit tests (we'll define this as code tested in isolation, within or spawned from within the testing process), and integration tests (we'll define this as code running in its own process, tested from the outside). I have two separate but related proposal for how to handle those. I'd like to avoid discussion which versions of things we think should be supported in Stein in this thread. Let's come up with a process that we think is a good one to take into T and beyond, and then retroactively apply it to Stein. Competing proposals are of course welcome, in addition to feedback on this one. Unit Tests -- For unit tests, the most important thing is to test on the versions of Python we target. It's less important to be using the exact distro that we want to target, because unit tests generally won't interact with stuff outside of Python. I'd like to propose that we handle this by setting up a unit test template in openstack-zuul-jobs for each release. So for Stein we'd have openstack-python3-stein-jobs. This template would contain: * A voting gate job for the highest minor version of py3 we want to support in that release. * A voting gate job for the lowest minor version of py3 we want to support in that release. * A periodic job for any interim minor releases. * (Starting late in the cycle) a non-voting check job for the highest minor version of py3 we want to support in the *next* release (if different), on the master branch only. So, for example, (and this is still under active debate) for Stein we might have gating jobs for py35 and py37, with a periodic job for py36. The T jobs might only have voting py36 and py37 jobs, but late in the T cycle we might add a non-voting py38 job on master so that people who haven't switched to the U template yet can see what, if anything, they'll need to fix. We'll run the unit tests on any distro we can find that supports the version of Python we want. It could be a non-LTS Ubuntu, Fedora, Debian unstable, whatever it takes. We won't wait for an LTS Ubuntu to have a particular Python version before trying to test it. Before the start of each cycle, the TC would determine which range of versions we want to support, on the basis of the latest one we can find in any distro and the earliest one we're likely to need in one of the supported Linux distros. There will be a project-wide goal to switch the testing template from e.g. openstack-python3-stein-jobs to openstack-python3-treasure-jobs for every repo before the end of the cycle. We'll have goal champions as usual following up and helping teams with the process. We'll know where the problem areas are because we'll have added non-voting jobs for any new Python versions to the previous release's template. Integration Tests - Integration tests do test, amongst other things, integration with non-openstack-supplied things in the distro, so it's important that we test on the actual distros we have identified as popular.[2] It's also important that every project be testing on the same distro at the end of a release, so we can be sure they all work together for users. When a new release of CentOS or a new LTS release of Ubuntu comes out, the TC will create a project-wide goal for the *next* release cycle to switch all integration tests over to that distro. It's up to individual projects to make the switch for the tests that they own (e.g. it'd be the QA team for Tempest, but other individual projects for their own jobs). Again, there'll be a goal champion to monitor and follow up. [1] https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation-timeline.html [2] https://governance.openstack.org/tc/reference/project-testing-interface.html#linux-distributions
Re: [openstack-dev] Proposal for a process to keep up with Python releases
On Mon, Oct 22, 2018 at 04:33:49PM +0200, Thomas Goirand wrote: > One of the reoccurring problem that I'm facing in Debian is that not > only Python 3 version is lagging behind, but OpenStack dependencies are > also lagging behind the distro. Often, the answer is "we don't support > this or that version of X", which of course is very frustrating. Can you provide a recent instance of this? I feel like we have a misunderstanding here. > One > thing which would be super nice, would be a non-voting gate job that > test with the latest version of every Python dependencies as well, so we > get to see breakage early. We've stopped seeing them since we decided it > breaks too often and we would hide problems behind the > global-requirement thing. We watch for this in requirements where everyday we update to the latest co-installable dependencies[1] and gate on them. If that passes it gets merged into the repo and used by all projects. Where we could do better is making the failures visible, and we're open to suggestions there. We have the following caps: cmd2!=0.8.3,<0.9.0;python_version<'3.0' # MIT construct<2.9 # MIT Django<2;python_version<'3.0' # BSD Django<2.1;python_version>='3.0' # BSD django-floppyforms<2 # BSD elasticsearch<3.0.0 # Apache-2.0 jsonpath-rw<2.0 # Apache-2.0 jsonschema<3.0.0 # MIT PrettyTable<0.8 # BSD python-congressclient<2000 # Apache-2.0 warlock<2 # Apache-2.0 XStatic-jQuery<2 # MIT License These of course do impact the dependencies that are considered co-installable and we're working towards minimising this list. You can see from[1] that the lates urllib is incompatible with botocore. So we'll exclude that from the update and try again. Meanwahile we'll file a bug (Possibly a patch) in botocore to get the cap removed or bumped. Yours Tony. [1] https://review.openstack.org/#/c/612252/ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for a process to keep up with Python releases
On 22/10/18 10:33 AM, Thomas Goirand wrote: On 10/19/18 5:17 PM, Zane Bitter wrote: We have traditionally held to the principle that we want each release to support the latest release of CentOS and the latest LTS release of Ubuntu, as they existed at the beginning of the release cycle.[2] Currently this means in practice one version of py2 and one of py3, but in the future it will mean two, usually different, versions of py3. That's not very nice to forget about the Debian case, which usually closely precedes Ubuntu. If you want to support Ubuntu better, then supporting Debian better helps. I usually get the issue before everyone, as Sid is the distro which is updated the most often. Therefore, please make sure to include Debian in your proposal. This is something that needs to be addressed separately I think. It has been our long-standing, documented testing policy. If you want to change it, make a proposal. For the purposes of this discussion though, the main point to take away from the paragraph you quoted is that once Python2 is EOL there will rarely be a _single_ version of Python3 that is sufficient to support even 2 distros, let alone more. I haven't forgotten about you, and in fact one of the goals of this process is to ensure that we stay up-to-date and not get into situations like you had in Rocky where we were two releases behind. Debian will definitely benefit from that. For unit tests, the most important thing is to test on the versions of Python we target. It's less important to be using the exact distro that we want to target, because unit tests generally won't interact with stuff outside of Python. One of the reoccurring problem that I'm facing in Debian is that not only Python 3 version is lagging behind, but OpenStack dependencies are also lagging behind the distro. Often, the answer is "we don't support this or that version of X", which of course is very frustrating. One thing which would be super nice, would be a non-voting gate job that test with the latest version of every Python dependencies as well, so we get to see breakage early. We've stopped seeing them since we decided it breaks too often and we would hide problems behind the global-requirement thing. I'll leave this to the requirements team, who are more qualified to comment. And sometimes, we have weird interactions. For example, taskflow was broken in Python 3.7 before this patch: https://salsa.debian.org/openstack-team/libs/python-taskflow/commit/6a10261a8a147d901c07a6e7272dc75b9f4d0988 which broke multiple packages using it. Funny thing, it looks like it wouldn't have happen if we didn't have a pre-version of Python 3.7.1 in Sid, apparently. Anyway, this can happen again. So, for example, (and this is still under active debate) for Stein we might have gating jobs for py35 and py37, with a periodic job for py36. The T jobs might only have voting py36 and py37 jobs, but late in the T cycle we might add a non-voting py38 job on master so that people who haven't switched to the U template yet can see what, if anything, they'll need to fix. This can only happen if we have supporting distribution packages for it. IMO, this is a call for using Debian Testing or even Sid in the gate. It depends on which versions we choose to support, but if necessary yes. We'll run the unit tests on any distro we can find that supports the version of Python we want. It could be a non-LTS Ubuntu, Fedora, Debian unstable, whatever it takes. We won't wait for an LTS Ubuntu to have a particular Python version before trying to test it. I very much agree with that. Before the start of each cycle, the TC would determine which range of versions we want to support, on the basis of the latest one we can find in any distro and the earliest one we're likely to need in one of the supported Linux distros. Release of Python aren't aligned with OpenStack cycles. Python 3.7 appeared late in the Rocky cycle. Therefore, unfortunately, doing what you propose above doesn't address the issue. This is valuable feedback; it's important to know where there are real-world cases that we're not addressing. Python 3.7 was released 3 weeks after rocky-2 and only 4 weeks before rocky-3. TBH I find it hard to imagine any process that would have led us to attempt to get every OpenStack project supporting 3.7 in Rocky without a radical change in our conception of how OpenStack is distributed. On the bright side, under this process we would have had 3.6 support in Ocata and we could have automatically added a non-voting (or periodic) 3.7 job during Rocky development as soon as a distro was available for testing, which would at least have made it easier to locate problems earlier even if we didn't get full 3.7 support until the Stein release. Integration Tests - Integration tests do test, amongst other things, integration with non-openstack-supplied things in the distro, so it's important that we test on the
Re: [openstack-dev] Proposal for a process to keep up with Python releases
On Mon, Oct 22, 2018, at 7:33 AM, Thomas Goirand wrote: > On 10/19/18 5:17 PM, Zane Bitter wrote: snip > > Integration Tests > > - > > > > Integration tests do test, amongst other things, integration with > > non-openstack-supplied things in the distro, so it's important that we > > test on the actual distros we have identified as popular.[2] It's also > > important that every project be testing on the same distro at the end of > > a release, so we can be sure they all work together for users. > > I find very disturbing to see the project only leaning toward these only > 2 distributions. Why not SuSE & Debian? It has to do with previous statements about distro support from the TC: https://governance.openstack.org/tc/reference/project-testing-interface.html#linux-distributions is the [2] above. Changing this would be an orthogonal piece of work even though there is relationship between the two topics. Zane's proposal can accommodate change in the distro support assertion, but is focused on figuring out which python versions to test with that as one of the inputs. Clark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for a process to keep up with Python releases
On 10/19/18 5:17 PM, Zane Bitter wrote: > We have traditionally held to the principle that we want each release to > support the latest release of CentOS and the latest LTS release of > Ubuntu, as they existed at the beginning of the release cycle.[2] > Currently this means in practice one version of py2 and one of py3, but > in the future it will mean two, usually different, versions of py3. That's not very nice to forget about the Debian case, which usually closely precedes Ubuntu. If you want to support Ubuntu better, then supporting Debian better helps. I usually get the issue before everyone, as Sid is the distro which is updated the most often. Therefore, please make sure to include Debian in your proposal. > For unit tests, the most important thing is to test on the versions of > Python we target. It's less important to be using the exact distro that > we want to target, because unit tests generally won't interact with > stuff outside of Python. One of the reoccurring problem that I'm facing in Debian is that not only Python 3 version is lagging behind, but OpenStack dependencies are also lagging behind the distro. Often, the answer is "we don't support this or that version of X", which of course is very frustrating. One thing which would be super nice, would be a non-voting gate job that test with the latest version of every Python dependencies as well, so we get to see breakage early. We've stopped seeing them since we decided it breaks too often and we would hide problems behind the global-requirement thing. And sometimes, we have weird interactions. For example, taskflow was broken in Python 3.7 before this patch: https://salsa.debian.org/openstack-team/libs/python-taskflow/commit/6a10261a8a147d901c07a6e7272dc75b9f4d0988 which broke multiple packages using it. Funny thing, it looks like it wouldn't have happen if we didn't have a pre-version of Python 3.7.1 in Sid, apparently. Anyway, this can happen again. > So, for example, (and this is still under active debate) for Stein we > might have gating jobs for py35 and py37, with a periodic job for py36. > The T jobs might only have voting py36 and py37 jobs, but late in the T > cycle we might add a non-voting py38 job on master so that people who > haven't switched to the U template yet can see what, if anything, > they'll need to fix. This can only happen if we have supporting distribution packages for it. IMO, this is a call for using Debian Testing or even Sid in the gate. > We'll run the unit tests on any distro we can find that supports the > version of Python we want. It could be a non-LTS Ubuntu, Fedora, Debian > unstable, whatever it takes. We won't wait for an LTS Ubuntu to have a > particular Python version before trying to test it. I very much agree with that. > Before the start of each cycle, the TC would determine which range of > versions we want to support, on the basis of the latest one we can find > in any distro and the earliest one we're likely to need in one of the > supported Linux distros. Release of Python aren't aligned with OpenStack cycles. Python 3.7 appeared late in the Rocky cycle. Therefore, unfortunately, doing what you propose above doesn't address the issue. > Integration Tests > - > > Integration tests do test, amongst other things, integration with > non-openstack-supplied things in the distro, so it's important that we > test on the actual distros we have identified as popular.[2] It's also > important that every project be testing on the same distro at the end of > a release, so we can be sure they all work together for users. I find very disturbing to see the project only leaning toward these only 2 distributions. Why not SuSE & Debian? Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for a process to keep up with Python releases
On Fri, Oct 19, 2018 at 3:46 PM Zane Bitter wrote: > On 19/10/18 12:30 PM, Clark Boylan wrote: > > On Fri, Oct 19, 2018, at 8:17 AM, Zane Bitter wrote: > >> Unit Tests > >> -- > >> > >> For unit tests, the most important thing is to test on the versions of > >> Python we target. It's less important to be using the exact distro that > >> we want to target, because unit tests generally won't interact with > >> stuff outside of Python. > >> > >> I'd like to propose that we handle this by setting up a unit test > >> template in openstack-zuul-jobs for each release. So for Stein we'd have > >> openstack-python3-stein-jobs. This template would contain: > > > > Because zuul config is branch specific we could set up every project to > use a `openstack-python3-jobs` template then define that template > differently on each branch. This would mean you only have to update the > location where the template is defined and not need to update every other > project after cutting a stable branch. I would suggest we take advantage of > that to reduce churn. > > There was a reason I didn't propose that approach: in practice you can't > add a new gating test to a centralised zuul template definition. If you > do, many projects will break because the change is not self-testing. At > best you'll be pitchforked by an angry mob of people who can't get > anything but py37 fixes through the gate, and at worst they'll all stop > using the template to get unblocked and then never go back to it. > > We don't need everyone to cut over at the same time. We just need them > to do it in the space of one release cycle. One patch every 6 months is > not an excessive amount of churn. > > >> * A voting gate job for the highest minor version of py3 we want to > >> support in that release. > >> * A voting gate job for the lowest minor version of py3 we want to > >> support in that release. > >> * A periodic job for any interim minor releases. > >> * (Starting late in the cycle) a non-voting check job for the highest > >> minor version of py3 we want to support in the *next* release (if > >> different), on the master branch only. > >> > >> So, for example, (and this is still under active debate) for Stein we > >> might have gating jobs for py35 and py37, with a periodic job for py36. > >> The T jobs might only have voting py36 and py37 jobs, but late in the T > >> cycle we might add a non-voting py38 job on master so that people who > >> haven't switched to the U template yet can see what, if anything, > >> they'll need to fix. > >> > >> We'll run the unit tests on any distro we can find that supports the > >> version of Python we want. It could be a non-LTS Ubuntu, Fedora, Debian > >> unstable, whatever it takes. We won't wait for an LTS Ubuntu to have a > >> particular Python version before trying to test it. > >> > >> Before the start of each cycle, the TC would determine which range of > >> versions we want to support, on the basis of the latest one we can find > >> in any distro and the earliest one we're likely to need in one of the > >> supported Linux distros. There will be a project-wide goal to switch the > >> testing template from e.g. openstack-python3-stein-jobs to > >> openstack-python3-treasure-jobs for every repo before the end of the > >> cycle. We'll have goal champions as usual following up and helping teams > >> with the process. We'll know where the problem areas are because we'll > >> have added non-voting jobs for any new Python versions to the previous > >> release's template. > > > > I don't know that this needs to be a project wide goal if you can just > update the template on the master branch where the template is defined. Do > that then every project is now running with the up to date version of the > template. We should probably advertise when this is happening with some > links to python version x.y breakages/features, but the process itself > should be quick. > > Either way, it'll be project teams themselves fixing any broken tests > due to a new version being added. So we can either have a formal > project-wide goal where we project-manage that process across the space > of a release, or a de-facto project-wide goal where we break everybody > and then nothing gets merged until they fix it. > > > As for python version range selection I worry that that the criteria > about relies on too much guesswork. > > Some guesswork is going to be inevitable, unfortunately, (we have no way > of knowing what will be in CentOS 8, for example) but I agree that we > should try to tighten up the criteria as much as possible. > > > I do think we should do our best to test future incoming versions of > python even while not officially supporting them. We will have to support > them at some point, either directly or via some later version that includes > the changes from that intermediate version. > > +1, I think we should try to add support for higher versions as soon as > possible. It may take a long time to get into an
Re: [openstack-dev] Proposal for a process to keep up with Python releases
On Fri, 19 Oct 2018, Zane Bitter wrote: Just to make it easier to visualise, here is an example for how the Zuul config _might_ look now if we had adopted this proposal during Rocky: https://review.openstack.org/611947 And instead of having a project-wide goal in Stein to add `openstack-python36-jobs` to the list that currently includes `openstack-python35-jobs` in each project's Zuul config[1], we'd have had a goal to change `openstack-python3-rocky-jobs` to `openstack-python3-stein-jobs` in each project's Zuul config. I like this, because it involves conscious actions, awareness and self-testing by each project to move forward to a thing with a reasonable name (the cycle name). I don't think we should call that "churn". "Intention" might be a better word. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for a process to keep up with Python releases
On 19/10/18 11:17 AM, Zane Bitter wrote: I'd like to propose that we handle this by setting up a unit test template in openstack-zuul-jobs for each release. So for Stein we'd have openstack-python3-stein-jobs. This template would contain: * A voting gate job for the highest minor version of py3 we want to support in that release. * A voting gate job for the lowest minor version of py3 we want to support in that release. * A periodic job for any interim minor releases. * (Starting late in the cycle) a non-voting check job for the highest minor version of py3 we want to support in the *next* release (if different), on the master branch only. So, for example, (and this is still under active debate) for Stein we might have gating jobs for py35 and py37, with a periodic job for py36. The T jobs might only have voting py36 and py37 jobs, but late in the T cycle we might add a non-voting py38 job on master so that people who haven't switched to the U template yet can see what, if anything, they'll need to fix. Just to make it easier to visualise, here is an example for how the Zuul config _might_ look now if we had adopted this proposal during Rocky: https://review.openstack.org/611947 And instead of having a project-wide goal in Stein to add `openstack-python36-jobs` to the list that currently includes `openstack-python35-jobs` in each project's Zuul config[1], we'd have had a goal to change `openstack-python3-rocky-jobs` to `openstack-python3-stein-jobs` in each project's Zuul config. - ZB [1] https://governance.openstack.org/tc/goals/stein/python3-first.html#python-3-6-unit-test-jobs __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for a process to keep up with Python releases
On 19/10/18 12:30 PM, Clark Boylan wrote: On Fri, Oct 19, 2018, at 8:17 AM, Zane Bitter wrote: Unit Tests -- For unit tests, the most important thing is to test on the versions of Python we target. It's less important to be using the exact distro that we want to target, because unit tests generally won't interact with stuff outside of Python. I'd like to propose that we handle this by setting up a unit test template in openstack-zuul-jobs for each release. So for Stein we'd have openstack-python3-stein-jobs. This template would contain: Because zuul config is branch specific we could set up every project to use a `openstack-python3-jobs` template then define that template differently on each branch. This would mean you only have to update the location where the template is defined and not need to update every other project after cutting a stable branch. I would suggest we take advantage of that to reduce churn. There was a reason I didn't propose that approach: in practice you can't add a new gating test to a centralised zuul template definition. If you do, many projects will break because the change is not self-testing. At best you'll be pitchforked by an angry mob of people who can't get anything but py37 fixes through the gate, and at worst they'll all stop using the template to get unblocked and then never go back to it. We don't need everyone to cut over at the same time. We just need them to do it in the space of one release cycle. One patch every 6 months is not an excessive amount of churn. * A voting gate job for the highest minor version of py3 we want to support in that release. * A voting gate job for the lowest minor version of py3 we want to support in that release. * A periodic job for any interim minor releases. * (Starting late in the cycle) a non-voting check job for the highest minor version of py3 we want to support in the *next* release (if different), on the master branch only. So, for example, (and this is still under active debate) for Stein we might have gating jobs for py35 and py37, with a periodic job for py36. The T jobs might only have voting py36 and py37 jobs, but late in the T cycle we might add a non-voting py38 job on master so that people who haven't switched to the U template yet can see what, if anything, they'll need to fix. We'll run the unit tests on any distro we can find that supports the version of Python we want. It could be a non-LTS Ubuntu, Fedora, Debian unstable, whatever it takes. We won't wait for an LTS Ubuntu to have a particular Python version before trying to test it. Before the start of each cycle, the TC would determine which range of versions we want to support, on the basis of the latest one we can find in any distro and the earliest one we're likely to need in one of the supported Linux distros. There will be a project-wide goal to switch the testing template from e.g. openstack-python3-stein-jobs to openstack-python3-treasure-jobs for every repo before the end of the cycle. We'll have goal champions as usual following up and helping teams with the process. We'll know where the problem areas are because we'll have added non-voting jobs for any new Python versions to the previous release's template. I don't know that this needs to be a project wide goal if you can just update the template on the master branch where the template is defined. Do that then every project is now running with the up to date version of the template. We should probably advertise when this is happening with some links to python version x.y breakages/features, but the process itself should be quick. Either way, it'll be project teams themselves fixing any broken tests due to a new version being added. So we can either have a formal project-wide goal where we project-manage that process across the space of a release, or a de-facto project-wide goal where we break everybody and then nothing gets merged until they fix it. As for python version range selection I worry that that the criteria about relies on too much guesswork. Some guesswork is going to be inevitable, unfortunately, (we have no way of knowing what will be in CentOS 8, for example) but I agree that we should try to tighten up the criteria as much as possible. I do think we should do our best to test future incoming versions of python even while not officially supporting them. We will have to support them at some point, either directly or via some later version that includes the changes from that intermediate version. +1, I think we should try to add support for higher versions as soon as possible. It may take a long time to get into an LTS release, but there's bound to be _some_ distro out there where people want to use it. (Case in point: Debian really wanted py37 support in Rocky, at which point a working 3.7 wasn't even available in _any_ Ubuntu release, let alone an LTS). That's why I said "the latest one we can find in any distro" - if we
Re: [openstack-dev] Proposal for a process to keep up with Python releases
On 10/19/2018 01:58 PM, Andreas Jaeger wrote: On 19/10/2018 18.30, Clark Boylan wrote: > [...] Because zuul config is branch specific we could set up every project to use a `openstack-python3-jobs` template then define that template differently on each branch. This would mean you only have to update the location where the template is defined and not need to update every other project after cutting a stable branch. I would suggest we take advantage of that to reduce churn. Alternative we have a single "openstack-python3-jobs" template in an unbranched repo like openstack-zuul-jobs and define different jobs per branch. The end result would be the same, each repo uses the same template and no changes are needed for the repo when branching... Yes - I agree that we should take advantage of zuul's branching support. And I agree with Andreas that we should just use branch matchers in openstack-zuul-jobs to do it. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for a process to keep up with Python releases
On 19/10/2018 18.30, Clark Boylan wrote:
> [...]
Because zuul config is branch specific we could set up every project to use a
`openstack-python3-jobs` template then define that template differently on each
branch. This would mean you only have to update the location where the template
is defined and not need to update every other project after cutting a stable
branch. I would suggest we take advantage of that to reduce churn.
Alternative we have a single "openstack-python3-jobs" template in an
unbranched repo like openstack-zuul-jobs and define different jobs per
branch.
The end result would be the same, each repo uses the same template and
no changes are needed for the repo when branching...
Andreas
--
Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for a process to keep up with Python releases
On Fri, Oct 19, 2018, at 8:17 AM, Zane Bitter wrote: > There hasn't been a Python 2 release in 8 years, and during that time > we've gotten used to the idea that that's the way things go. However, > with the switch to Python 3 looming (we will drop support for Python 2 > in the U release[1]), history is no longer a good guide: Python 3 > releases drop as often as every year. We are already feeling the pain > from this, as Linux distros have largely already completed the shift to > Python 3, and those that have are on versions newer than the py35 we > currently have in gate jobs. > > We have traditionally held to the principle that we want each release to > support the latest release of CentOS and the latest LTS release of > Ubuntu, as they existed at the beginning of the release cycle.[2] > Currently this means in practice one version of py2 and one of py3, but > in the future it will mean two, usually different, versions of py3. > > There are two separate issues that we need to address: unit tests (we'll > define this as code tested in isolation, within or spawned from within > the testing process), and integration tests (we'll define this as code > running in its own process, tested from the outside). I have two > separate but related proposal for how to handle those. > > I'd like to avoid discussion which versions of things we think should be > supported in Stein in this thread. Let's come up with a process that we > think is a good one to take into T and beyond, and then retroactively > apply it to Stein. Competing proposals are of course welcome, in > addition to feedback on this one. > > Unit Tests > -- > > For unit tests, the most important thing is to test on the versions of > Python we target. It's less important to be using the exact distro that > we want to target, because unit tests generally won't interact with > stuff outside of Python. > > I'd like to propose that we handle this by setting up a unit test > template in openstack-zuul-jobs for each release. So for Stein we'd have > openstack-python3-stein-jobs. This template would contain: Because zuul config is branch specific we could set up every project to use a `openstack-python3-jobs` template then define that template differently on each branch. This would mean you only have to update the location where the template is defined and not need to update every other project after cutting a stable branch. I would suggest we take advantage of that to reduce churn. > > * A voting gate job for the highest minor version of py3 we want to > support in that release. > * A voting gate job for the lowest minor version of py3 we want to > support in that release. > * A periodic job for any interim minor releases. > * (Starting late in the cycle) a non-voting check job for the highest > minor version of py3 we want to support in the *next* release (if > different), on the master branch only. > > So, for example, (and this is still under active debate) for Stein we > might have gating jobs for py35 and py37, with a periodic job for py36. > The T jobs might only have voting py36 and py37 jobs, but late in the T > cycle we might add a non-voting py38 job on master so that people who > haven't switched to the U template yet can see what, if anything, > they'll need to fix. > > We'll run the unit tests on any distro we can find that supports the > version of Python we want. It could be a non-LTS Ubuntu, Fedora, Debian > unstable, whatever it takes. We won't wait for an LTS Ubuntu to have a > particular Python version before trying to test it. > > Before the start of each cycle, the TC would determine which range of > versions we want to support, on the basis of the latest one we can find > in any distro and the earliest one we're likely to need in one of the > supported Linux distros. There will be a project-wide goal to switch the > testing template from e.g. openstack-python3-stein-jobs to > openstack-python3-treasure-jobs for every repo before the end of the > cycle. We'll have goal champions as usual following up and helping teams > with the process. We'll know where the problem areas are because we'll > have added non-voting jobs for any new Python versions to the previous > release's template. I don't know that this needs to be a project wide goal if you can just update the template on the master branch where the template is defined. Do that then every project is now running with the up to date version of the template. We should probably advertise when this is happening with some links to python version x.y breakages/features, but the process itself should be quick. As for python version range selection I worry that that the criteria about relies on too much guesswork. I do think we should do our best to test future incoming versions of python even while not officially supporting them. We will have to support them at some point, either directly or via some later version
