Re: [openstack-dev] [all][tc] final stages of python 3 transition
On 2018-05-29 15:51:31 -0400 (-0400), Doug Hellmann wrote: [...] > Could we, for example, look at the set of packages installed under > python2 and report errors if any OpenStack packages end up there? [...] This sounds like a marvellous solution. -- Jeremy Stanley signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] final stages of python 3 transition
Excerpts from Matthew Treinish's message of 2018-05-29 16:16:50 -0400: > On Tue, May 29, 2018 at 03:51:31PM -0400, Doug Hellmann wrote: > > Following up on this topic, at the Forum discussion last week (see > > https://etherpad.openstack.org/p/YVR-python-2-deprecation-timeline) the > > general plan outlined below was acceptable to most of the folks in the > > room with a few small changes (included below). > > > > Excerpts from Doug Hellmann's message of 2018-04-25 16:54:46 -0400: > > > It's time to talk about the next steps in our migration from python > > > 2 to python 3. > > > > > > Up to this point we have mostly focused on reaching a state where > > > we support both versions of the language. We are not quite there > > > with all projects, as you can see by reviewing the test coverage > > > status information at > > > https://wiki.openstack.org/wiki/Python3#Python_3_Status_of_OpenStack_projects > > > > > > Still, we need to press on to the next phase of the migration, which > > > I have been calling "Python 3 first". This is where we use python > > > 3 as the default, for everything, and set up the exceptions we need > > > for anything that still requires python 2. > > > > > > To reach that stage, we need to: > > > > > > 1. Change the documentation and release notes jobs to use python 3. > > >(The Oslo team recently completed this, and found that we did > > >need to make a few small code changes to get them to work.) > > > 2. Change (or duplicate) all functional test jobs to run under > > >python 3. > > > 3. Change the packaging jobs to use python 3. > > > 4. Update devstack to use 3 by default and require setting a flag to > > >use 2. (This may trigger other job changes.) > > > > Also: > > > > - Ensure that devstack configures mod_wsgi (or whatever WSGI service) to > > use Python 3 when deploying API components. > > The python 3 dsvm jobs already do this for the most part. All API services > that > support running as a wsgi application run under uwsgi with a single apache > redirecting traffic to those. This is the supported model for running wsgi > services on devstack. Currently keystone, glance, nova, placement, and cinder > run their API servers this way. Neutron doesn't run under as wsgi app (I > don't recall why this was never implemented for neutron) and swift doesn't run > in the py3 jobs at all. You can see an example of this here: OK, good, thank you for clarifying that. I think the folks in the room weren't 100% sure, so the point was to double check. And it sounds like we still need to do that for projects using devstack plugins, based on your next comment. > > http://logs.openstack.org/08/550108/3/gate/tempest-full-py3/df744ef/controller/logs/ > > For other services it depends on how they implemented their devstack plugin. I > haven't done an inventory on how all the plugins are running things, so I > don't > know what the status of each project is there. > > > - Test "python version skew" within a service during a rolling upgrade > > across multiple hosts. > > - Add an integration test job that does not include python2 on the host > > at all. > > > > That last item may block us from using other tools, such as ansible, > > that rely on python2. If the point of such a test is to ensure that > > we are properly installing (and running) our tools under python3, > > maybe *that's* what we want to check, instead of forbidding a python2 > > package at all? Could we, for example, look at the set of packages > > installed under python2 and report errors if any OpenStack packages end > > up there? > > > > > > > > At that point, all of our deliverables will be produced using python > > > 3, and we can be relatively confident that if we no longer had > > > access to python 2 we could still continue operating. We could also > > > start updating deployment tools to use either python 3 or 2, so > > > that users could actually deploy using the python 3 versions of > > > services. > > > > > > Somewhere in that time frame our third-party CI systems will need > > > to ensure they have python 3 support as well. > > > > > > After the "Python 3 first" phase is completed we should release > > > one series using the packages built with python 3. Perhaps Stein? > > > Or is that too ambitious? > > > > > > Next, we will be ready to address the prerequisites for "Python 3 > > > only," which will allow us to drop Python 2 support. > > > > > > We need to wait to drop python 2 support as a community, rather > > > than going one project at a time, to avoid doubling the work of > > > downstream consumers such as distros and independent deployers. We > > > don't want them to have to package all (or even a large number) of > > > the dependencies of OpenStack twice because they have to install > > > some services running under python 2 and others under 3. Ideally > > > they would be able to upgrade all of the services on a node together > > > as part of their transition to the
Re: [openstack-dev] [all][tc] final stages of python 3 transition
On Tue, May 29, 2018 at 03:51:31PM -0400, Doug Hellmann wrote: > Following up on this topic, at the Forum discussion last week (see > https://etherpad.openstack.org/p/YVR-python-2-deprecation-timeline) the > general plan outlined below was acceptable to most of the folks in the > room with a few small changes (included below). > > Excerpts from Doug Hellmann's message of 2018-04-25 16:54:46 -0400: > > It's time to talk about the next steps in our migration from python > > 2 to python 3. > > > > Up to this point we have mostly focused on reaching a state where > > we support both versions of the language. We are not quite there > > with all projects, as you can see by reviewing the test coverage > > status information at > > https://wiki.openstack.org/wiki/Python3#Python_3_Status_of_OpenStack_projects > > > > Still, we need to press on to the next phase of the migration, which > > I have been calling "Python 3 first". This is where we use python > > 3 as the default, for everything, and set up the exceptions we need > > for anything that still requires python 2. > > > > To reach that stage, we need to: > > > > 1. Change the documentation and release notes jobs to use python 3. > >(The Oslo team recently completed this, and found that we did > >need to make a few small code changes to get them to work.) > > 2. Change (or duplicate) all functional test jobs to run under > >python 3. > > 3. Change the packaging jobs to use python 3. > > 4. Update devstack to use 3 by default and require setting a flag to > >use 2. (This may trigger other job changes.) > > Also: > > - Ensure that devstack configures mod_wsgi (or whatever WSGI service) to > use Python 3 when deploying API components. The python 3 dsvm jobs already do this for the most part. All API services that support running as a wsgi application run under uwsgi with a single apache redirecting traffic to those. This is the supported model for running wsgi services on devstack. Currently keystone, glance, nova, placement, and cinder run their API servers this way. Neutron doesn't run under as wsgi app (I don't recall why this was never implemented for neutron) and swift doesn't run in the py3 jobs at all. You can see an example of this here: http://logs.openstack.org/08/550108/3/gate/tempest-full-py3/df744ef/controller/logs/ For other services it depends on how they implemented their devstack plugin. I haven't done an inventory on how all the plugins are running things, so I don't know what the status of each project is there. > - Test "python version skew" within a service during a rolling upgrade > across multiple hosts. > - Add an integration test job that does not include python2 on the host > at all. > > That last item may block us from using other tools, such as ansible, > that rely on python2. If the point of such a test is to ensure that > we are properly installing (and running) our tools under python3, > maybe *that's* what we want to check, instead of forbidding a python2 > package at all? Could we, for example, look at the set of packages > installed under python2 and report errors if any OpenStack packages end > up there? > > > > > At that point, all of our deliverables will be produced using python > > 3, and we can be relatively confident that if we no longer had > > access to python 2 we could still continue operating. We could also > > start updating deployment tools to use either python 3 or 2, so > > that users could actually deploy using the python 3 versions of > > services. > > > > Somewhere in that time frame our third-party CI systems will need > > to ensure they have python 3 support as well. > > > > After the "Python 3 first" phase is completed we should release > > one series using the packages built with python 3. Perhaps Stein? > > Or is that too ambitious? > > > > Next, we will be ready to address the prerequisites for "Python 3 > > only," which will allow us to drop Python 2 support. > > > > We need to wait to drop python 2 support as a community, rather > > than going one project at a time, to avoid doubling the work of > > downstream consumers such as distros and independent deployers. We > > don't want them to have to package all (or even a large number) of > > the dependencies of OpenStack twice because they have to install > > some services running under python 2 and others under 3. Ideally > > they would be able to upgrade all of the services on a node together > > as part of their transition to the new version, without ending up > > with a python 2 version of a dependency along side a python 3 version > > of the same package. > > > > The remaining items could be fixed earlier, but this is the point > > at which they would block us: > > > > 1. Fix oslo.service functional tests -- the Oslo team needs help > >maintaining this library. Alternatively, we could move all > >services to use cotyledon (https://pypi.org/project/cotyledon/). > > > > 2. Finish the unit test and
Re: [openstack-dev] [all][tc] final stages of python 3 transition
Following up on this topic, at the Forum discussion last week (see https://etherpad.openstack.org/p/YVR-python-2-deprecation-timeline) the general plan outlined below was acceptable to most of the folks in the room with a few small changes (included below). Excerpts from Doug Hellmann's message of 2018-04-25 16:54:46 -0400: > It's time to talk about the next steps in our migration from python > 2 to python 3. > > Up to this point we have mostly focused on reaching a state where > we support both versions of the language. We are not quite there > with all projects, as you can see by reviewing the test coverage > status information at > https://wiki.openstack.org/wiki/Python3#Python_3_Status_of_OpenStack_projects > > Still, we need to press on to the next phase of the migration, which > I have been calling "Python 3 first". This is where we use python > 3 as the default, for everything, and set up the exceptions we need > for anything that still requires python 2. > > To reach that stage, we need to: > > 1. Change the documentation and release notes jobs to use python 3. >(The Oslo team recently completed this, and found that we did >need to make a few small code changes to get them to work.) > 2. Change (or duplicate) all functional test jobs to run under >python 3. > 3. Change the packaging jobs to use python 3. > 4. Update devstack to use 3 by default and require setting a flag to >use 2. (This may trigger other job changes.) Also: - Ensure that devstack configures mod_wsgi (or whatever WSGI service) to use Python 3 when deploying API components. - Test "python version skew" within a service during a rolling upgrade across multiple hosts. - Add an integration test job that does not include python2 on the host at all. That last item may block us from using other tools, such as ansible, that rely on python2. If the point of such a test is to ensure that we are properly installing (and running) our tools under python3, maybe *that's* what we want to check, instead of forbidding a python2 package at all? Could we, for example, look at the set of packages installed under python2 and report errors if any OpenStack packages end up there? > > At that point, all of our deliverables will be produced using python > 3, and we can be relatively confident that if we no longer had > access to python 2 we could still continue operating. We could also > start updating deployment tools to use either python 3 or 2, so > that users could actually deploy using the python 3 versions of > services. > > Somewhere in that time frame our third-party CI systems will need > to ensure they have python 3 support as well. > > After the "Python 3 first" phase is completed we should release > one series using the packages built with python 3. Perhaps Stein? > Or is that too ambitious? > > Next, we will be ready to address the prerequisites for "Python 3 > only," which will allow us to drop Python 2 support. > > We need to wait to drop python 2 support as a community, rather > than going one project at a time, to avoid doubling the work of > downstream consumers such as distros and independent deployers. We > don't want them to have to package all (or even a large number) of > the dependencies of OpenStack twice because they have to install > some services running under python 2 and others under 3. Ideally > they would be able to upgrade all of the services on a node together > as part of their transition to the new version, without ending up > with a python 2 version of a dependency along side a python 3 version > of the same package. > > The remaining items could be fixed earlier, but this is the point > at which they would block us: > > 1. Fix oslo.service functional tests -- the Oslo team needs help >maintaining this library. Alternatively, we could move all >services to use cotyledon (https://pypi.org/project/cotyledon/). > > 2. Finish the unit test and functional test ports so that all of >our tests can run under python 3 (this implies that the services >all run under python 3, so there is no more porting to do). > > Finally, after we have *all* tests running on python 3, we can > safely drop python 2. We clarified that we would only drop python 2 support on master. That clarification also raised the point that eventually backports may become more difficult if master is using python 3 features, but Matt and Tony agreed that we could potentially have rebasing issues with fixes today so while this is a new source of such issues it isn't a completely new problem and our stable backport policies already address it. > > We have previously discussed the end of the T cycle as the point > at which we would have all of those tests running, and if that holds > true we could reasonably drop python 2 during the beginning of the > U cycle, in late 2019 and before the 2020 cut-off point when upstream > python 2 support will be dropped. This date as the earliest point at which existing
Re: [openstack-dev] [all][tc] final stages of python 3 transition
Excerpts from Clark Boylan's message of 2018-04-26 11:22:15 -0700: > On Thu, Apr 26, 2018, at 7:27 AM, Sean McGinnis wrote: > > On Wed, Apr 25, 2018 at 04:54:46PM -0400, Doug Hellmann wrote: > > > It's time to talk about the next steps in our migration from python > > > 2 to python 3. > > > > > > [...] > > > > > > 2. Change (or duplicate) all functional test jobs to run under > > >python 3. > > > > As a test I ran Cinder functional and unit test jobs on bionic using 3.6. > > All > > went well. > > > > That made me realize something though - right now we have jobs that > > explicitly > > say py35, both for unit tests and functional tests. But I realized setting > > up > > these test jobs that it works to just specify "basepython = python3" or run > > unit tests with "tox -e py3". Then with that, it just depends on whether the > > job runs on xenial or bionic as to whether the job is run with py35 or py36. > > > > It is less explicit, so I see some downside to that, but would it make > > sense to > > change jobs to drop the minor version to make it more flexible and easy to > > make > > these transitions? > > One reason to use it would be local user simplicity. Rather than need to > explicitly add new python3 releases to the default env list so that it does > what we want every year or two we can just list py3,py2,linters in the > default list and get most of the way there for local users. Then we can > continue to be more specific in the CI jobs if that is desirable. > > I do think we likely want to be explicit about the python versions we are > using in CI testing. This makes it clear to developers who may need to > reproduce or just understand why failures happen what platform is used. It > also makes it explicit that "openstack runs on $pythonversion". > > Clark > Including support for local users to refer to "py3" makes sense, as long as we don't come to rely on it in CI. Users can also always be more explicit if they need to be when running tests locally. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] final stages of python 3 transition
On Thu, Apr 26, 2018, at 7:27 AM, Sean McGinnis wrote: > On Wed, Apr 25, 2018 at 04:54:46PM -0400, Doug Hellmann wrote: > > It's time to talk about the next steps in our migration from python > > 2 to python 3. > > > > [...] > > > > 2. Change (or duplicate) all functional test jobs to run under > >python 3. > > As a test I ran Cinder functional and unit test jobs on bionic using 3.6. All > went well. > > That made me realize something though - right now we have jobs that explicitly > say py35, both for unit tests and functional tests. But I realized setting up > these test jobs that it works to just specify "basepython = python3" or run > unit tests with "tox -e py3". Then with that, it just depends on whether the > job runs on xenial or bionic as to whether the job is run with py35 or py36. > > It is less explicit, so I see some downside to that, but would it make sense > to > change jobs to drop the minor version to make it more flexible and easy to > make > these transitions? One reason to use it would be local user simplicity. Rather than need to explicitly add new python3 releases to the default env list so that it does what we want every year or two we can just list py3,py2,linters in the default list and get most of the way there for local users. Then we can continue to be more specific in the CI jobs if that is desirable. I do think we likely want to be explicit about the python versions we are using in CI testing. This makes it clear to developers who may need to reproduce or just understand why failures happen what platform is used. It also makes it explicit that "openstack runs on $pythonversion". Clark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] final stages of python 3 transition
On 2018-04-26 13:17:33 -0400 (-0400), Paul Belanger wrote: [...] > This maybe okay, will allow others to comment, but the main reason > I am not a fan, is we can no longer infer the nodeset by looking > at the job name. tox-py3 could be xenial or bionic. This brings back a question we've struggled with for years: are we testing against "Python X.Y" or are we testing against "Python as provided by distro Z"? Depending on how you think about that, one solution or the other is technically a more accurate reflection of our choice here. -- Jeremy Stanley signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] final stages of python 3 transition
On Thu, Apr 26, 2018 at 09:27:31AM -0500, Sean McGinnis wrote: > On Wed, Apr 25, 2018 at 04:54:46PM -0400, Doug Hellmann wrote: > > It's time to talk about the next steps in our migration from python > > 2 to python 3. > > > > [...] > > > > 2. Change (or duplicate) all functional test jobs to run under > >python 3. > > As a test I ran Cinder functional and unit test jobs on bionic using 3.6. All > went well. > > That made me realize something though - right now we have jobs that explicitly > say py35, both for unit tests and functional tests. But I realized setting up > these test jobs that it works to just specify "basepython = python3" or run > unit tests with "tox -e py3". Then with that, it just depends on whether the > job runs on xenial or bionic as to whether the job is run with py35 or py36. > > It is less explicit, so I see some downside to that, but would it make sense > to > change jobs to drop the minor version to make it more flexible and easy to > make > these transitions? > I still think using tox-py35 / tox-py36 makes sense, as those jobs are already setup to use the specific nodeset of ubuntu-xenial or ubuntu-bionic. If we did move to just tox-py3, it would actually result if more code projects would need to add to their .zuul.yaml files: -project: check: jobs: - tox-py35 -project: check: jobs: - tox-py3: nodeset: ubuntu-xenial This maybe okay, will allow others to comment, but the main reason I am not a fan, is we can no longer infer the nodeset by looking at the job name. tox-py3 could be xenial or bionic. Paul __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] final stages of python 3 transition
On Wed, Apr 25, 2018 at 04:54:46PM -0400, Doug Hellmann wrote: > It's time to talk about the next steps in our migration from python > 2 to python 3. > > [...] > > 2. Change (or duplicate) all functional test jobs to run under >python 3. As a test I ran Cinder functional and unit test jobs on bionic using 3.6. All went well. That made me realize something though - right now we have jobs that explicitly say py35, both for unit tests and functional tests. But I realized setting up these test jobs that it works to just specify "basepython = python3" or run unit tests with "tox -e py3". Then with that, it just depends on whether the job runs on xenial or bionic as to whether the job is run with py35 or py36. It is less explicit, so I see some downside to that, but would it make sense to change jobs to drop the minor version to make it more flexible and easy to make these transitions? __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] final stages of python 3 transition
On 04/25/2018 11:40 PM, Jeremy Stanley wrote: > It may be worth considering how this interacts with the switch of > our default test platform from Ubuntu 16.04 (which provides Python > 3.5) to 18.04 (which provides Python 3.6). If we switch from 3.5 to > 3.6 before we change most remaining jobs over to Python 3.x versions > then it gives us a chance to spot differences between 3.5 and 3.6 at > that point. I don't think you'll find lots of issues, as all Debian and Gentoo packages were built against Python 3.6, and hopefully, prometheanfire and myself have reported the issues. > So I guess that raises the question: switch to Python 3.5 by default > for most jobs in Rocky and then have a potentially more disruptive > default platform switch with Python 3.5->3.6 at the beginning of > Stein, or wait until the default platform switch to move from Python > 2.7 to 3.6 as the job default? I can see some value in each option. I'd love to see gating on both Python 3.5 and 3.6 if possible. Also, can we restart the attempts (non-voting) gating jobs with Debian Sid? That's always were we get all updates first. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] final stages of python 3 transition
On 2018-04-25 18:25:24 -0400 (-0400), Doug Hellmann wrote: [...] > Does 18.04 include a python 2 option? Yes, 2.7.15 if packages.ubuntu.com is to be believed. > What is the target for completing the changeover? The first or > second milestone for Stein, or the end of the cycle? I would expect us to want to pull the trigger after whatever grace period the cycle-trailing projects are comfortable with (but certainly before the first milestone I would think?). > It would be useful to have some input from the project teams who > have no unit or functional test jobs running for 3.5, since they > will have the most work to do to cope with the upgrade overall. Yes, it would in my opinion anyway. > Who is coordinating Ubuntu upgrade work and setting up the > experimental jobs? We have preliminary ubuntu-bionic images available already (officially it doesn't release until tomorrow), and some teams have started to use them for experimental or non-voting jobs: http://codesearch.openstack.org/?q=ubuntu-bionic -- Jeremy Stanley signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] final stages of python 3 transition
On Wed, Apr 25, 2018, at 3:25 PM, Doug Hellmann wrote: > Excerpts from Jeremy Stanley's message of 2018-04-25 21:40:37 +: > > On 2018-04-25 16:54:46 -0400 (-0400), Doug Hellmann wrote: > > [...] > > > Still, we need to press on to the next phase of the migration, which > > > I have been calling "Python 3 first". This is where we use python > > > 3 as the default, for everything, and set up the exceptions we need > > > for anything that still requires python 2. > > [...] > > > > It may be worth considering how this interacts with the switch of > > our default test platform from Ubuntu 16.04 (which provides Python > > 3.5) to 18.04 (which provides Python 3.6). If we switch from 3.5 to > > 3.6 before we change most remaining jobs over to Python 3.x versions > > then it gives us a chance to spot differences between 3.5 and 3.6 at > > that point. Given that the 14.04 to 16.04 migration, where we > > attempted to allow projects to switch at their own pace, didn't go > > so well we're hoping to do a "big bang" migration instead for 18.04 > > and expect teams who haven't set up experimental jobs ahead of time > > to work out remaining blockers after the flag day before they can go > > back to business as usual. Since the 18.04 release is happening so > > far into the Rocky cycle, we're likely to want to do that at the > > start of Stein instead when it will be less disruptive. > > > > So I guess that raises the question: switch to Python 3.5 by default > > for most jobs in Rocky and then have a potentially more disruptive > > default platform switch with Python 3.5->3.6 at the beginning of > > Stein, or wait until the default platform switch to move from Python > > 2.7 to 3.6 as the job default? I can see some value in each option. > > Does 18.04 include a python 2 option? It does, https://packages.ubuntu.com/bionic/python2.7. > > What is the target for completing the changeover? The first or > second milestone for Stein, or the end of the cycle? Previously we've tried to do the transition in OpenStack release that is under development when the LTS releases. However we've offset things a bit now so that may not be as feasible. I would expect that if we waited for the next cycle we would do it very early in that cycle. For the transition from python 3.5 on Xenial to 3.6 on Bionic we may want to keep the python 3.5 jobs on Xenial but add in non voting python 3.6 jobs to every project running Xenial python3.5 jobs. Then those projects can toggle them to voting 3.6 jobs if/when they start working. Then we can decide at a later time if continuing to support python 3.5 (and testing it) is worthwhile. > > It would be useful to have some input from the project teams who > have no unit or functional test jobs running for 3.5, since they > will have the most work to do to cope with the upgrade overall. > > Who is coordinating Ubuntu upgrade work and setting up the experimental > jobs? Paul Belanger has been doing much of the work to get the images up and running and helping some projects start to run early jobs on the beta images. I expect Paul would want to continue to carry the transition through to the end. Clark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] final stages of python 3 transition
Excerpts from Jeremy Stanley's message of 2018-04-25 21:40:37 +: > On 2018-04-25 16:54:46 -0400 (-0400), Doug Hellmann wrote: > [...] > > Still, we need to press on to the next phase of the migration, which > > I have been calling "Python 3 first". This is where we use python > > 3 as the default, for everything, and set up the exceptions we need > > for anything that still requires python 2. > [...] > > It may be worth considering how this interacts with the switch of > our default test platform from Ubuntu 16.04 (which provides Python > 3.5) to 18.04 (which provides Python 3.6). If we switch from 3.5 to > 3.6 before we change most remaining jobs over to Python 3.x versions > then it gives us a chance to spot differences between 3.5 and 3.6 at > that point. Given that the 14.04 to 16.04 migration, where we > attempted to allow projects to switch at their own pace, didn't go > so well we're hoping to do a "big bang" migration instead for 18.04 > and expect teams who haven't set up experimental jobs ahead of time > to work out remaining blockers after the flag day before they can go > back to business as usual. Since the 18.04 release is happening so > far into the Rocky cycle, we're likely to want to do that at the > start of Stein instead when it will be less disruptive. > > So I guess that raises the question: switch to Python 3.5 by default > for most jobs in Rocky and then have a potentially more disruptive > default platform switch with Python 3.5->3.6 at the beginning of > Stein, or wait until the default platform switch to move from Python > 2.7 to 3.6 as the job default? I can see some value in each option. Does 18.04 include a python 2 option? What is the target for completing the changeover? The first or second milestone for Stein, or the end of the cycle? It would be useful to have some input from the project teams who have no unit or functional test jobs running for 3.5, since they will have the most work to do to cope with the upgrade overall. Who is coordinating Ubuntu upgrade work and setting up the experimental jobs? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] final stages of python 3 transition
On 2018-04-25 16:54:46 -0400 (-0400), Doug Hellmann wrote: [...] > Still, we need to press on to the next phase of the migration, which > I have been calling "Python 3 first". This is where we use python > 3 as the default, for everything, and set up the exceptions we need > for anything that still requires python 2. [...] It may be worth considering how this interacts with the switch of our default test platform from Ubuntu 16.04 (which provides Python 3.5) to 18.04 (which provides Python 3.6). If we switch from 3.5 to 3.6 before we change most remaining jobs over to Python 3.x versions then it gives us a chance to spot differences between 3.5 and 3.6 at that point. Given that the 14.04 to 16.04 migration, where we attempted to allow projects to switch at their own pace, didn't go so well we're hoping to do a "big bang" migration instead for 18.04 and expect teams who haven't set up experimental jobs ahead of time to work out remaining blockers after the flag day before they can go back to business as usual. Since the 18.04 release is happening so far into the Rocky cycle, we're likely to want to do that at the start of Stein instead when it will be less disruptive. So I guess that raises the question: switch to Python 3.5 by default for most jobs in Rocky and then have a potentially more disruptive default platform switch with Python 3.5->3.6 at the beginning of Stein, or wait until the default platform switch to move from Python 2.7 to 3.6 as the job default? I can see some value in each option. -- Jeremy Stanley signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all][tc] final stages of python 3 transition
It's time to talk about the next steps in our migration from python 2 to python 3. Up to this point we have mostly focused on reaching a state where we support both versions of the language. We are not quite there with all projects, as you can see by reviewing the test coverage status information at https://wiki.openstack.org/wiki/Python3#Python_3_Status_of_OpenStack_projects Still, we need to press on to the next phase of the migration, which I have been calling "Python 3 first". This is where we use python 3 as the default, for everything, and set up the exceptions we need for anything that still requires python 2. To reach that stage, we need to: 1. Change the documentation and release notes jobs to use python 3. (The Oslo team recently completed this, and found that we did need to make a few small code changes to get them to work.) 2. Change (or duplicate) all functional test jobs to run under python 3. 3. Change the packaging jobs to use python 3. 4. Update devstack to use 3 by default and require setting a flag to use 2. (This may trigger other job changes.) At that point, all of our deliverables will be produced using python 3, and we can be relatively confident that if we no longer had access to python 2 we could still continue operating. We could also start updating deployment tools to use either python 3 or 2, so that users could actually deploy using the python 3 versions of services. Somewhere in that time frame our third-party CI systems will need to ensure they have python 3 support as well. After the "Python 3 first" phase is completed we should release one series using the packages built with python 3. Perhaps Stein? Or is that too ambitious? Next, we will be ready to address the prerequisites for "Python 3 only," which will allow us to drop Python 2 support. We need to wait to drop python 2 support as a community, rather than going one project at a time, to avoid doubling the work of downstream consumers such as distros and independent deployers. We don't want them to have to package all (or even a large number) of the dependencies of OpenStack twice because they have to install some services running under python 2 and others under 3. Ideally they would be able to upgrade all of the services on a node together as part of their transition to the new version, without ending up with a python 2 version of a dependency along side a python 3 version of the same package. The remaining items could be fixed earlier, but this is the point at which they would block us: 1. Fix oslo.service functional tests -- the Oslo team needs help maintaining this library. Alternatively, we could move all services to use cotyledon (https://pypi.org/project/cotyledon/). 2. Finish the unit test and functional test ports so that all of our tests can run under python 3 (this implies that the services all run under python 3, so there is no more porting to do). Finally, after we have *all* tests running on python 3, we can safely drop python 2. We have previously discussed the end of the T cycle as the point at which we would have all of those tests running, and if that holds true we could reasonably drop python 2 during the beginning of the U cycle, in late 2019 and before the 2020 cut-off point when upstream python 2 support will be dropped. I need some info from the deployment tool teams to understand whether they would be ready to take the plunge during T or U and start deploying only the python 3 version. Are there other upgrade issues that need to be addressed to support moving from 2 to 3? Something that might be part of the platform(s), rather than OpenStack itself? What else have I missed in these phases? Other jobs? Other blocking conditions? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
