Re: [openstack-dev] [python3] tempest and grenade conversion to python 3.6
On Wed, 19 Sep 2018 02:28:29 +0900 Doug Hellmann wrote > Excerpts from Clark Boylan's message of 2018-09-18 09:53:45 -0700: > > On Tue, Sep 18, 2018, at 9:46 AM, Nate Johnston wrote: > > > Hello python 3.6 champions, > > > > > > I have looked around a little, and I don't see a method for me to > > > specifically select the version of python that the tempest and grenade > > > jobs for my project (neutron) are using. I assume one of four things > > > is at play here: > > > > > > A. These projects already shifted to python 3 and I don't have to worry > > > about it > > > > > > B. There is a toggle for the python version I just have not seen yet > > > > > > C. These projects are still on python 2 and need help to do a conversion > > > to python 3, which would affect all customers > > > > > > D. Something else that I have failed to imagine > > > > > > Could you elaborate which of these options properly reflects the state > > > of affairs? If the answer is "C" then perhaps we can start a discussion > > > on that migration. > > > > For our devstack and grenade jobs tempest is installed using tox [0]. And > > since the full testenv in tempest's tox.ini doesn't specify a python > > version [1] I expect that it will attempt a python2 virtualenv on every > > platform (Arch linux may be an exception but we don't test that). > > > > I think that means C is the situation here. To change that you can set > > basepython to python3 (see [2] for an example) which will run tempest > > under whichever python3 is present on the system. The one gotcha for this > > is that it will break tempest on centos which does not have python3. Maybe > > the thing to do there is add a full-python2 testenv that centos can run? > > > > [0] > > https://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/tempest#n653 > > [1] https://git.openstack.org/cgit/openstack/tempest/tree/tox.ini#n74 > > [2] https://git.openstack.org/cgit/openstack-infra/zuul/tree/tox.ini#n7 > > > > Hope this helps, > > Clark > > > > While having tempest run under python 3 would be great, I'm not sure > that's necessary in order to test a service. > > Don't those jobs use devstack to install the system being tested? And > devstack uses some environment variables to control the version of > python. For example the tempest-full-py3 job [1] defines USE_PYTHON3 as > 'true'. > > What's probably missing is a version of the grenade job that allows us > to control that USE_PYTHON3 variable before and after the upgrade. > > I see a few different grenade jobs (neutron-grenade, > neutron-grenade-multinode, > legacy-grenade-dsvm-neutron-multinode-live-migration, possibly others). > Which ones are "current" and would make a good candidate as a base for a > new job? All these are legacy job, only name changed so i will not recommend them to use as base. Currently those are on neutron repo instead of grenade. We discussed this in PTG about finishing the grenade base zuul v3 job work so that other project can use as base. Work is in progress[1] and on priority[2] for us to finish as early as possible. [1] https://review.openstack.org/#/q/topic:grenade_zuulv3+(status:open+OR+status:merged) [2] https://etherpad.openstack.org/p/qa-stein-priority -gmann > > Doug > > [1] http://git.openstack.org/cgit/openstack/tempest/tree/.zuul.yaml#n70 > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [python3] tempest and grenade conversion to python 3.6
On Tue, Sep 18, 2018 at 09:52:50PM -0500, Matt Riedemann wrote: > On 9/18/2018 12:28 PM, Doug Hellmann wrote: > > What's probably missing is a version of the grenade job that allows us > > to control that USE_PYTHON3 variable before and after the upgrade. > > > > I see a few different grenade jobs (neutron-grenade, > > neutron-grenade-multinode, > > legacy-grenade-dsvm-neutron-multinode-live-migration, possibly others). > > Which ones are "current" and would make a good candidate as a base for a > > new job? > > Grenade just runs devstack on the old side (e.g. stable/rocky) using the > devstack stackrc file (which could have USE_PYTHON3 in it), runs tempest > 'smoke' tests to create some resources, saves off some information about > those resources in a "database" (just an ini file), then runs devstack on > the new side (e.g. master) using the new side stackrc file and verifies > those saved off resources made it through the upgrade. It's all bash so > there isn't anything python-specific about grenade. This isn't quite right, we run devstack on the old side. But, on the new side we don't actually run devstack. Grenade updates the code, runs DB migrations (and any other mandatory upgrade steps), and then just relaunches the service. That's kind of the point to make sure new code works with old config. The target (ie new side) stackrc and localrc/local.conf are there for the common functions shared between devstack and grenade which are used to do things like pull the code and start services to make sure they run against the proper branches. Since there isn't any point in reimplementing the same exact thing. But we don't do a full devstack run, that's why you see only see stack.sh run once in the logs on a grenade job. > > I saw, but didn't comment on, the other thread about if it would be possible > to create a grenade-2to3 job. I'd think that is pretty doable based on the > USE_PYTHON3 variable. We'd just have that False on the old side, and True on > the new side, and devstack will do it's thing. Right now the USE_PYTHON3 > variable is global in devstack-gate [1] (which is the thing that > orchestrates the grenade run for the legacy jobs), but I'm sure we could > hack that to be specific to the base (old) and target (new) release for the > grenade run. I don't think this will work because we won't be running any initial python 3 setup on the system. I think it will just update paths and try to use python3 pip and python3 paths for things, but it will be missing the things it needs for those to work. It's probably worth a try either way (a quick experiment to say definitively) but my gut is telling me that it's not going to be that simple. -Matt Treinish > > [1] > https://github.com/openstack-infra/devstack-gate/blob/95fa4343104eafa655375cce3546d27139211d13/devstack-vm-gate-wrap.sh#L434 > signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [python3] tempest and grenade conversion to python 3.6
On 9/18/2018 9:52 PM, Matt Riedemann wrote: On 9/18/2018 12:28 PM, Doug Hellmann wrote: What's probably missing is a version of the grenade job that allows us to control that USE_PYTHON3 variable before and after the upgrade. I see a few different grenade jobs (neutron-grenade, neutron-grenade-multinode, legacy-grenade-dsvm-neutron-multinode-live-migration, possibly others). Which ones are "current" and would make a good candidate as a base for a new job? Grenade just runs devstack on the old side (e.g. stable/rocky) using the devstack stackrc file (which could have USE_PYTHON3 in it), runs tempest 'smoke' tests to create some resources, saves off some information about those resources in a "database" (just an ini file), then runs devstack on the new side (e.g. master) using the new side stackrc file and verifies those saved off resources made it through the upgrade. It's all bash so there isn't anything python-specific about grenade. I saw, but didn't comment on, the other thread about if it would be possible to create a grenade-2to3 job. I'd think that is pretty doable based on the USE_PYTHON3 variable. We'd just have that False on the old side, and True on the new side, and devstack will do it's thing. Right now the USE_PYTHON3 variable is global in devstack-gate [1] (which is the thing that orchestrates the grenade run for the legacy jobs), but I'm sure we could hack that to be specific to the base (old) and target (new) release for the grenade run. [1] https://github.com/openstack-infra/devstack-gate/blob/95fa4343104eafa655375cce3546d27139211d13/devstack-vm-gate-wrap.sh#L434 To answer Doug's original question, neutron-grenade-multinode is probably best to model for a new job if you want to test rolling upgrades, because that job has two compute nodes and leaves one on the 'old' side so it would upgrade the controller services and one compute to Stein and leave the other compute at Rocky. So if you start with python2 on the old side and upgrade to python3 for everything except one compute, you'll have a pretty good idea of whether or not that rolling upgrade works through our various services and libraries, like the oslo.messaging stuff noted in the other thread. -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [python3] tempest and grenade conversion to python 3.6
On 9/18/2018 12:28 PM, Doug Hellmann wrote: What's probably missing is a version of the grenade job that allows us to control that USE_PYTHON3 variable before and after the upgrade. I see a few different grenade jobs (neutron-grenade, neutron-grenade-multinode, legacy-grenade-dsvm-neutron-multinode-live-migration, possibly others). Which ones are "current" and would make a good candidate as a base for a new job? Grenade just runs devstack on the old side (e.g. stable/rocky) using the devstack stackrc file (which could have USE_PYTHON3 in it), runs tempest 'smoke' tests to create some resources, saves off some information about those resources in a "database" (just an ini file), then runs devstack on the new side (e.g. master) using the new side stackrc file and verifies those saved off resources made it through the upgrade. It's all bash so there isn't anything python-specific about grenade. I saw, but didn't comment on, the other thread about if it would be possible to create a grenade-2to3 job. I'd think that is pretty doable based on the USE_PYTHON3 variable. We'd just have that False on the old side, and True on the new side, and devstack will do it's thing. Right now the USE_PYTHON3 variable is global in devstack-gate [1] (which is the thing that orchestrates the grenade run for the legacy jobs), but I'm sure we could hack that to be specific to the base (old) and target (new) release for the grenade run. [1] https://github.com/openstack-infra/devstack-gate/blob/95fa4343104eafa655375cce3546d27139211d13/devstack-vm-gate-wrap.sh#L434 -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [python3] tempest and grenade conversion to python 3.6
Excerpts from Clark Boylan's message of 2018-09-18 09:53:45 -0700: > On Tue, Sep 18, 2018, at 9:46 AM, Nate Johnston wrote: > > Hello python 3.6 champions, > > > > I have looked around a little, and I don't see a method for me to > > specifically select the version of python that the tempest and grenade > > jobs for my project (neutron) are using. I assume one of four things > > is at play here: > > > > A. These projects already shifted to python 3 and I don't have to worry > > about it > > > > B. There is a toggle for the python version I just have not seen yet > > > > C. These projects are still on python 2 and need help to do a conversion > > to python 3, which would affect all customers > > > > D. Something else that I have failed to imagine > > > > Could you elaborate which of these options properly reflects the state > > of affairs? If the answer is "C" then perhaps we can start a discussion > > on that migration. > > For our devstack and grenade jobs tempest is installed using tox [0]. And > since the full testenv in tempest's tox.ini doesn't specify a python version > [1] I expect that it will attempt a python2 virtualenv on every platform > (Arch linux may be an exception but we don't test that). > > I think that means C is the situation here. To change that you can set > basepython to python3 (see [2] for an example) which will run tempest under > whichever python3 is present on the system. The one gotcha for this is that > it will break tempest on centos which does not have python3. Maybe the thing > to do there is add a full-python2 testenv that centos can run? > > [0] > https://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/tempest#n653 > [1] https://git.openstack.org/cgit/openstack/tempest/tree/tox.ini#n74 > [2] https://git.openstack.org/cgit/openstack-infra/zuul/tree/tox.ini#n7 > > Hope this helps, > Clark > While having tempest run under python 3 would be great, I'm not sure that's necessary in order to test a service. Don't those jobs use devstack to install the system being tested? And devstack uses some environment variables to control the version of python. For example the tempest-full-py3 job [1] defines USE_PYTHON3 as 'true'. What's probably missing is a version of the grenade job that allows us to control that USE_PYTHON3 variable before and after the upgrade. I see a few different grenade jobs (neutron-grenade, neutron-grenade-multinode, legacy-grenade-dsvm-neutron-multinode-live-migration, possibly others). Which ones are "current" and would make a good candidate as a base for a new job? Doug [1] http://git.openstack.org/cgit/openstack/tempest/tree/.zuul.yaml#n70 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [python3] tempest and grenade conversion to python 3.6
On Tue, Sep 18, 2018, at 9:46 AM, Nate Johnston wrote: > Hello python 3.6 champions, > > I have looked around a little, and I don't see a method for me to > specifically select the version of python that the tempest and grenade > jobs for my project (neutron) are using. I assume one of four things > is at play here: > > A. These projects already shifted to python 3 and I don't have to worry > about it > > B. There is a toggle for the python version I just have not seen yet > > C. These projects are still on python 2 and need help to do a conversion > to python 3, which would affect all customers > > D. Something else that I have failed to imagine > > Could you elaborate which of these options properly reflects the state > of affairs? If the answer is "C" then perhaps we can start a discussion > on that migration. For our devstack and grenade jobs tempest is installed using tox [0]. And since the full testenv in tempest's tox.ini doesn't specify a python version [1] I expect that it will attempt a python2 virtualenv on every platform (Arch linux may be an exception but we don't test that). I think that means C is the situation here. To change that you can set basepython to python3 (see [2] for an example) which will run tempest under whichever python3 is present on the system. The one gotcha for this is that it will break tempest on centos which does not have python3. Maybe the thing to do there is add a full-python2 testenv that centos can run? [0] https://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/tempest#n653 [1] https://git.openstack.org/cgit/openstack/tempest/tree/tox.ini#n74 [2] https://git.openstack.org/cgit/openstack-infra/zuul/tree/tox.ini#n7 Hope this helps, Clark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [python3] tempest and grenade conversion to python 3.6
Hello python 3.6 champions, I have looked around a little, and I don't see a method for me to specifically select the version of python that the tempest and grenade jobs for my project (neutron) are using. I assume one of four things is at play here: A. These projects already shifted to python 3 and I don't have to worry about it B. There is a toggle for the python version I just have not seen yet C. These projects are still on python 2 and need help to do a conversion to python 3, which would affect all customers D. Something else that I have failed to imagine Could you elaborate which of these options properly reflects the state of affairs? If the answer is "C" then perhaps we can start a discussion on that migration. Thanks! Nate Johnston __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
