Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
Excerpts from Andrey Kurilin's message of 2018-08-09 13:35:53 +0300: > Hi Doug! > > I'm ready to port our job to openstack-zuul-jobs repo, but I expect that > Community will not accept it. > > The result of rally unittests is different between environments with python > 3.7 final release and python 3.7.0~b3 . > There is at least one failed test at python 3.7.0~b3 which is not > reproducible at py27,py34,py35,py36,py37-final , > so I'm not sure that it is a good decision to add py37 job based on > ubuntu-bionic. > > As for Rally, I applied the easiest thing which occurred to me - just use > external python ppa (deadsnakes) to > install the final release of Python 3.7. > Such a way is satisfying for Rally community and it cannot be used as the > main one for the whole OpenStack. Yes, I think we don't want to use that approach for most of the jobs. The point is to test on the Python packaged in the distro. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
Hi Doug! I'm ready to port our job to openstack-zuul-jobs repo, but I expect that Community will not accept it. The result of rally unittests is different between environments with python 3.7 final release and python 3.7.0~b3 . There is at least one failed test at python 3.7.0~b3 which is not reproducible at py27,py34,py35,py36,py37-final , so I'm not sure that it is a good decision to add py37 job based on ubuntu-bionic. As for Rally, I applied the easiest thing which occurred to me - just use external python ppa (deadsnakes) to install the final release of Python 3.7. Such a way is satisfying for Rally community and it cannot be used as the main one for the whole OpenStack. ср, 8 авг. 2018 г. в 16:35, Doug Hellmann : > Excerpts from Andrey Kurilin's message of 2018-08-08 15:25:01 +0300: > > Thanks Thomas for pointing to the issue, I checked it locally and here is > > an update for openstack/rally (rally framework without in-tree OpenStack > > plugins) project: > > > > - added unittest job with py37 env > > It would be really useful if you could help set up a job definition in > openstack-zuul-jobs like we have for openstack-tox-py36 [1], so that other > projects can easily add the job, too. Do you have time to do that? > > Doug > > [1] > http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/jobs.yaml#n354 > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Best regards, Andrey Kurilin. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
On Wed, Aug 8, 2018 at 10:29 AM, Mehdi Abaakouk wrote: > On Wed, Aug 08, 2018 at 08:35:20AM -0400, Corey Bryant wrote: > >> On Wed, Aug 8, 2018 at 3:43 AM, Thomas Goirand wrote: >> >> On 08/07/2018 06:10 PM, Corey Bryant wrote: >>> > I was concerned that there wouldn't be any >>> > gating until Ubuntu 20.04 (April 2020) >>> Same over here. I'm concerned that it takes another 2 years, which >>> really, we cannot afford. >>> >>> > but Py3.7 is available in bionic today. >>> >> > Yeah but it's the beta3 version. > > Yes, that's something I mentioned before but it was snipped from the conversation. We're going to try and get that updated. Corey -- > Mehdi Abaakouk > mail: sil...@sileht.net > irc: sileht > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
On Wed, Aug 08, 2018 at 08:35:20AM -0400, Corey Bryant wrote: On Wed, Aug 8, 2018 at 3:43 AM, Thomas Goirand wrote: On 08/07/2018 06:10 PM, Corey Bryant wrote: > I was concerned that there wouldn't be any > gating until Ubuntu 20.04 (April 2020) Same over here. I'm concerned that it takes another 2 years, which really, we cannot afford. > but Py3.7 is available in bionic today. Yeah but it's the beta3 version. -- Mehdi Abaakouk mail: sil...@sileht.net irc: sileht __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
On Wed, Aug 08, 2018 at 09:35:04AM -0400, Doug Hellmann wrote: Excerpts from Andrey Kurilin's message of 2018-08-08 15:25:01 +0300: Thanks Thomas for pointing to the issue, I checked it locally and here is an update for openstack/rally (rally framework without in-tree OpenStack plugins) project: - added unittest job with py37 env It would be really useful if you could help set up a job definition in openstack-zuul-jobs like we have for openstack-tox-py36 [1], so that other projects can easily add the job, too. Do you have time to do that? I have already done this kind of stuff for Telemetry project. And our project already gate on py37. The only restriction is that we have to use a fedora-28 instance with the python-3.7 package installed manually (via bindep.txt). Ubuntu 18.04 LTS only a beta version of python 3.7 in universe repo. So I'm guessing we have to wait next Ubuntu LTS to add this job everywhere. https://github.com/openstack/ceilometer/blob/master/.zuul.yaml#L12 https://github.com/openstack/ceilometer/blob/master/bindep.txt#L7 Cheers, -- Mehdi Abaakouk mail: sil...@sileht.net irc: sileht __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
Excerpts from Andrey Kurilin's message of 2018-08-08 15:25:01 +0300: > Thanks Thomas for pointing to the issue, I checked it locally and here is > an update for openstack/rally (rally framework without in-tree OpenStack > plugins) project: > > - added unittest job with py37 env It would be really useful if you could help set up a job definition in openstack-zuul-jobs like we have for openstack-tox-py36 [1], so that other projects can easily add the job, too. Do you have time to do that? Doug [1] http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/jobs.yaml#n354 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
On Wed, Aug 8, 2018 at 3:43 AM, Thomas Goirand wrote: > On 08/07/2018 06:10 PM, Corey Bryant wrote: > > I was concerned that there wouldn't be any > > gating until Ubuntu 20.04 (April 2020) > Same over here. I'm concerned that it takes another 2 years, which > really, we cannot afford. > > > but Py3.7 is available in bionic today. > > Is Bionic going to be released with Py3.7? In Debconf18 in Taiwan, Doko > didn't seem completely sure about it. > > Bionic was released with py3.7 in April 2018. Py3.6 is the default for Bionic but Py3.7 is available. https://launchpad.net/ubuntu/+source/python3.7 Corey Cheers, > > Thomas Goirand (zigo) > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
Thanks Thomas for pointing to the issue, I checked it locally and here is an update for openstack/rally (rally framework without in-tree OpenStack plugins) project: - added unittest job with py37 env - fixed all issues - released a new version (1.1.0) As for the openstack/rally-openstack (rally plugins for OpenStack platform), I'm planning to do the same this week. пн, 6 авг. 2018 г. в 20:11, Thomas Goirand : > On 08/02/2018 10:43 AM, Andrey Kurilin wrote: > > There's also some "raise StopIteration" issues in: > > - ceilometer > > - cinder > > - designate > > - glance > > - glare > > - heat > > - karbor > > - manila > > - murano > > - networking-ovn > > - neutron-vpnaas > > - nova > > - rally > > > > > > Can you provide any traceback or steps to reproduce the issue for Rally > > project ? > > I'm not sure there's any. The only thing I know is that it has stop > StopIteration stuff, but I'm not sure if they are part of generators, in > which case they should simply be replaced by "return" if you want it to > be py 3.7 compatible. > > I didn't have time to investigate these, but at least Glance was > affected, and a patch was sent (as well as an async patch). None of them > has been merged yet: > > https://review.openstack.org/#/c/586050/ > https://review.openstack.org/#/c/586716/ > > That'd be ok if at least there was some reviews. It looks like nobody > cares but Debian & Ubuntu people... :( > > Cheers, > > Thomas Goirand (zigo) > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Best regards, Andrey Kurilin. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
On 08/07/2018 06:10 PM, Corey Bryant wrote: > I was concerned that there wouldn't be any > gating until Ubuntu 20.04 (April 2020) Same over here. I'm concerned that it takes another 2 years, which really, we cannot afford. > but Py3.7 is available in bionic today. Is Bionic going to be released with Py3.7? In Debconf18 in Taiwan, Doko didn't seem completely sure about it. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
On Tue, Aug 7, 2018 at 11:28 AM, Zane Bitter wrote: > Top posting to avoid getting into the weeds. > > * OpenStack is indeed lagging behind > * The road to 3.7 (and eventually 3.8) runs through 3.6 > * As part of the project-wide python3-first goal we aim to have everything > working on 3.6 for Stein, so we are making some progress at least > * As of now we are *not* dropping support for 3.5 in Stein > * No matter what we do, the specific issue you're encountering is > structural: we don't add support for a Python version in the gate until it > is available in an Ubuntu LTS release, and that doesn't happen until after > it is available in Debian, so you will always have the problem that new > Python versions will be introduced in Debian before we have a gate for them > Thanks for mentioning this. I was concerned that there wouldn't be any gating until Ubuntu 20.04 (April 2020) but Py3.7 is available in bionic today. It's a bit older version but I think that's just because we're early py3.7 stages, so we'll try to get that updated. Thanks, Corey * Structural problems require structural solutions; "everybody work > harder/pay more attention/prioritise differently" will not do it > * I don't see any evidence that people are refusing to review patches that > fix 3.7 issues, and I certainly don't think fixing them is 'controversial' > > > On 07/08/18 10:11, Thomas Goirand wrote: > >> On 08/07/2018 03:24 PM, Sean Mooney wrote: >> >>> so im not sure pushing for python 3.7 is the right thing to do. also i >>> would not >>> assume all distros will ship 3.7 in the near term. i have not check >>> lately but >>> i believe cento 7 unless make 3.4 and 3.6 available in the default repos. >>> ubuntu 18.04 ships with 3.6 i believe >>> >> >> The current plan for Debian is that we'll be trying to push for Python >> 3.7 for Buster, which freezes in January. This freeze date means that >> it's going to be Rocky that will end up in the next Debian release. If >> Python 3.7 is a failure, then late November, we will remove Python 3.7 >> from Unstable and let Buster release with 3.6. >> >> As for Ubuntu, it is currently unclear if 18.10 will be released with >> Python 3.7 or not, but I believe they are trying to do that. If not, >> then 19.04 will for sure be released with Python 3.7. >> >> im not sure about other linux distros but since most openstack >>> deployment are done >>> on LTS releases of operating systems i would suspect that python 3.6 >>> will be the main >>> python 3 versions we see deployed in production for some time. >>> >> >> In short: that's wrong. >> >> having a 3.7 gate is not a bad idea but priority wise have a 3.6 gate >>> would be much higher on my list. >>> >> >> Wrong list. One version behind. >> >> i think we as a community will have to decide on the minimum and >>> maximum python 3 versions >>> we support for each release and adjust as we go forward. >>> >> >> Whatever the OpenStack community decides is not going to change what >> distributions like Debian will do. This type of reasoning lacks a much >> needed humility. >> >> i would suggst a min of 3.5 and max of 3.6 for rocky. >>> >> >> My suggestion is that these bugs are of very high importance and that >> they should at least deserve attention. That the gate for Python 3.7 >> isn't ready, I can understand, as everyone's time is limited. This >> doesn't mean that the OpenStack community at large should just dismiss >> patches that are important for downstream. >> >> for stien perhaps bump that to min of 3.6 max 3.7 but i think this is >>> something that needs to be address community wide >>> via a governance resolution rather then per project. >>> >> >> At this point, dropping 3.5 isn't a good idea either, even for Stein. >> >> it will also >>> impact the external python lib we can depend on too which is >>> another reason i think thie need to be a comuntiy wide discussion and >>> goal that is informed by what distros are doing but >>> not mandated by what any one distro is doing. >>> regards >>> sean. >>> >> >> Postponing any attempt to support anything current is always a bad idea. >> I don't see why there's even a controversy when one attempts to fix bugs >> that will, sooner or later, also hit the gate. >> >> Cheers, >> >> Thomas Goirand (zigo) >> >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib >> e >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
Excerpts from Thomas Goirand's message of 2018-08-07 16:11:43 +0200: > On 08/07/2018 03:24 PM, Sean Mooney wrote: > > > i think we as a community will have to decide on the minimum and > > maximum python 3 versions > > we support for each release and adjust as we go forward. > > Whatever the OpenStack community decides is not going to change what > distributions like Debian will do. This type of reasoning lacks a much > needed humility. That goes both ways, Thomas. We're in the middle of the RC1 deadline week for Rocky right now. This is not a great time to be pushing for new work unrelated to finishing that release. > > it will also > > impact the external python lib we can depend on too which is > > another reason i think thie need to be a comuntiy wide discussion and > > goal that is informed by what distros are doing but > > not mandated by what any one distro is doing. > > regards > > sean. > > Postponing any attempt to support anything current is always a bad idea. > I don't see why there's even a controversy when one attempts to fix bugs > that will, sooner or later, also hit the gate. The community is not prepared to support 3.7 today. That doesn't mean we will never support it, just that it is not the most important thing for us to be doing right now. We'll get there. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
Top posting to avoid getting into the weeds. * OpenStack is indeed lagging behind * The road to 3.7 (and eventually 3.8) runs through 3.6 * As part of the project-wide python3-first goal we aim to have everything working on 3.6 for Stein, so we are making some progress at least * As of now we are *not* dropping support for 3.5 in Stein * No matter what we do, the specific issue you're encountering is structural: we don't add support for a Python version in the gate until it is available in an Ubuntu LTS release, and that doesn't happen until after it is available in Debian, so you will always have the problem that new Python versions will be introduced in Debian before we have a gate for them * Structural problems require structural solutions; "everybody work harder/pay more attention/prioritise differently" will not do it * I don't see any evidence that people are refusing to review patches that fix 3.7 issues, and I certainly don't think fixing them is 'controversial' On 07/08/18 10:11, Thomas Goirand wrote: On 08/07/2018 03:24 PM, Sean Mooney wrote: so im not sure pushing for python 3.7 is the right thing to do. also i would not assume all distros will ship 3.7 in the near term. i have not check lately but i believe cento 7 unless make 3.4 and 3.6 available in the default repos. ubuntu 18.04 ships with 3.6 i believe The current plan for Debian is that we'll be trying to push for Python 3.7 for Buster, which freezes in January. This freeze date means that it's going to be Rocky that will end up in the next Debian release. If Python 3.7 is a failure, then late November, we will remove Python 3.7 from Unstable and let Buster release with 3.6. As for Ubuntu, it is currently unclear if 18.10 will be released with Python 3.7 or not, but I believe they are trying to do that. If not, then 19.04 will for sure be released with Python 3.7. im not sure about other linux distros but since most openstack deployment are done on LTS releases of operating systems i would suspect that python 3.6 will be the main python 3 versions we see deployed in production for some time. In short: that's wrong. having a 3.7 gate is not a bad idea but priority wise have a 3.6 gate would be much higher on my list. Wrong list. One version behind. i think we as a community will have to decide on the minimum and maximum python 3 versions we support for each release and adjust as we go forward. Whatever the OpenStack community decides is not going to change what distributions like Debian will do. This type of reasoning lacks a much needed humility. i would suggst a min of 3.5 and max of 3.6 for rocky. My suggestion is that these bugs are of very high importance and that they should at least deserve attention. That the gate for Python 3.7 isn't ready, I can understand, as everyone's time is limited. This doesn't mean that the OpenStack community at large should just dismiss patches that are important for downstream. for stien perhaps bump that to min of 3.6 max 3.7 but i think this is something that needs to be address community wide via a governance resolution rather then per project. At this point, dropping 3.5 isn't a good idea either, even for Stein. it will also impact the external python lib we can depend on too which is another reason i think thie need to be a comuntiy wide discussion and goal that is informed by what distros are doing but not mandated by what any one distro is doing. regards sean. Postponing any attempt to support anything current is always a bad idea. I don't see why there's even a controversy when one attempts to fix bugs that will, sooner or later, also hit the gate. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
On 08/07/2018 03:24 PM, Sean Mooney wrote: > so im not sure pushing for python 3.7 is the right thing to do. also i would > not > assume all distros will ship 3.7 in the near term. i have not check lately but > i believe cento 7 unless make 3.4 and 3.6 available in the default repos. > ubuntu 18.04 ships with 3.6 i believe The current plan for Debian is that we'll be trying to push for Python 3.7 for Buster, which freezes in January. This freeze date means that it's going to be Rocky that will end up in the next Debian release. If Python 3.7 is a failure, then late November, we will remove Python 3.7 from Unstable and let Buster release with 3.6. As for Ubuntu, it is currently unclear if 18.10 will be released with Python 3.7 or not, but I believe they are trying to do that. If not, then 19.04 will for sure be released with Python 3.7. > im not sure about other linux distros but since most openstack > deployment are done > on LTS releases of operating systems i would suspect that python 3.6 > will be the main > python 3 versions we see deployed in production for some time. In short: that's wrong. > having a 3.7 gate is not a bad idea but priority wise have a 3.6 gate > would be much higher on my list. Wrong list. One version behind. > i think we as a community will have to decide on the minimum and > maximum python 3 versions > we support for each release and adjust as we go forward. Whatever the OpenStack community decides is not going to change what distributions like Debian will do. This type of reasoning lacks a much needed humility. > i would suggst a min of 3.5 and max of 3.6 for rocky. My suggestion is that these bugs are of very high importance and that they should at least deserve attention. That the gate for Python 3.7 isn't ready, I can understand, as everyone's time is limited. This doesn't mean that the OpenStack community at large should just dismiss patches that are important for downstream. > for stien perhaps bump that to min of 3.6 max 3.7 but i think this is > something that needs to be address community wide > via a governance resolution rather then per project. At this point, dropping 3.5 isn't a good idea either, even for Stein. > it will also > impact the external python lib we can depend on too which is > another reason i think thie need to be a comuntiy wide discussion and > goal that is informed by what distros are doing but > not mandated by what any one distro is doing. > regards > sean. Postponing any attempt to support anything current is always a bad idea. I don't see why there's even a controversy when one attempts to fix bugs that will, sooner or later, also hit the gate. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
On 7 August 2018 at 12:52, Thomas Goirand wrote: > On 08/06/2018 09:02 PM, Sean McGinnis wrote: >>> >>> I didn't have time to investigate these, but at least Glance was >>> affected, and a patch was sent (as well as an async patch). None of them >>> has been merged yet: >>> >>> https://review.openstack.org/#/c/586050/ >>> https://review.openstack.org/#/c/586716/ >>> >>> That'd be ok if at least there was some reviews. It looks like nobody >>> cares but Debian & Ubuntu people... :( >>> >> >> Keep in mind that your priorities are different than everyone elses. There >> are >> large parts of the community still working on Python 3.5 support (our >> officially supported Python 3 version), as well as smaller teams overall >> working on things like critical bugs. >> >> Unless and until we declare Python 3.7 as our new target (which I don't think >> we are ready to do yet), these kinds of patches will be on a best effort >> basis. > > This is exactly what I'm complaining about. OpenStack upstream has very > wrong priorities. If we really are to switch to Python 3, then we got to > make sure we're current, because that's the version distros are end up > running. Or maybe we only care if "it works on devstack" (tm)? python 3.7 has some backward incompatible changes if i recall correctly such as forked thread not inheriting open file descriptor form the parent. i dont think that will bite us but it might mess with privsep deamon though i think we fork a full process not a thread in that case. the point im trying to make here is that following the latest python versions is likely going to require us to either A.) use only the backwards compatible subset or B.) make some code test what versions of python 3 we are using the same way the six package does. so im not sure pushing for python 3.7 is the right thing to do. also i would not assume all distros will ship 3.7 in the near term. i have not check lately but i believe cento 7 unless make 3.4 and 3.6 available in the default repos. ubuntu 18.04 ships with 3.6 i believe im not sure about other linux distros but since most openstack deployment are done on LTS releases of operating systems i would suspect that python 3.6 will be the main python 3 versions we see deployed in production for some time. having a 3.7 gate is not a bad idea but priority wise have a 3.6 gate would be much higher on my list. i think we as a community will have to decide on the minimum and maximum python 3 versions we support for each release and adjust as we go forward. i would suggst a min of 3.5 and max of 3.6 for rocky. for stien perhaps bump that to min of 3.6 max 3.7 but i think this is something that needs to be address community wide via a governance resolution rather then per project. it will also impact the external python lib we can depend on too which is another reason i think thie need to be a comuntiy wide discussion and goal that is informed by what distros are doing but not mandated by what any one distro is doing. regards sean. > > Cheers, > > Thomas Goirand (zigo) > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
On 08/06/2018 09:02 PM, Sean McGinnis wrote: >> >> I didn't have time to investigate these, but at least Glance was >> affected, and a patch was sent (as well as an async patch). None of them >> has been merged yet: >> >> https://review.openstack.org/#/c/586050/ >> https://review.openstack.org/#/c/586716/ >> >> That'd be ok if at least there was some reviews. It looks like nobody >> cares but Debian & Ubuntu people... :( >> > > Keep in mind that your priorities are different than everyone elses. There are > large parts of the community still working on Python 3.5 support (our > officially supported Python 3 version), as well as smaller teams overall > working on things like critical bugs. > > Unless and until we declare Python 3.7 as our new target (which I don't think > we are ready to do yet), these kinds of patches will be on a best effort > basis. This is exactly what I'm complaining about. OpenStack upstream has very wrong priorities. If we really are to switch to Python 3, then we got to make sure we're current, because that's the version distros are end up running. Or maybe we only care if "it works on devstack" (tm)? Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
On 06/08/18 13:11, Thomas Goirand wrote: On 08/02/2018 10:43 AM, Andrey Kurilin wrote: There's also some "raise StopIteration" issues in: - ceilometer - cinder - designate - glance - glare - heat - karbor - manila - murano - networking-ovn - neutron-vpnaas - nova - rally Can you provide any traceback or steps to reproduce the issue for Rally project ? I assume Thomas is only trying to run the unit tests, since that's what he has to do to verify the package? I'm not sure there's any. The only thing I know is that it has stop StopIteration stuff, but I'm not sure if they are part of generators, in which case they should simply be replaced by "return" if you want it to be py 3.7 compatible. I was about to say nobody is doing 'raise StopIteration' where they mean 'return' until I saw that the Glance tests apparently were :D The main issue though is when StopIteration is raised by one thing that happens to be called from *another* generator. e.g. many of the Heat tests that are failing are because we supplied a too-short list of side-effects to a mock and calling next() on them raises StopIteration, but because the calls were happening from inside a generator the StopIterations previously just got swallowed. If no generator were involved the test would have failed with the StopIteration exception. (Note: this was a bug - either in the code or more likely the tests. The purpose of the change in py37 was to expose this kind of bug wherever it exists.) I didn't have time to investigate these, but at least Glance was affected, and a patch was sent (as well as an async patch). None of them has been merged yet: https://review.openstack.org/#/c/586050/ https://review.openstack.org/#/c/586716/ That'd be ok if at least there was some reviews. It looks like nobody cares but Debian & Ubuntu people... :( Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
> > I didn't have time to investigate these, but at least Glance was > affected, and a patch was sent (as well as an async patch). None of them > has been merged yet: > > https://review.openstack.org/#/c/586050/ > https://review.openstack.org/#/c/586716/ > > That'd be ok if at least there was some reviews. It looks like nobody > cares but Debian & Ubuntu people... :( > Keep in mind that your priorities are different than everyone elses. There are large parts of the community still working on Python 3.5 support (our officially supported Python 3 version), as well as smaller teams overall working on things like critical bugs. Unless and until we declare Python 3.7 as our new target (which I don't think we are ready to do yet), these kinds of patches will be on a best effort basis. Making sure that duplicate patches are not pushed up will also help increase the chances that they will eventually make it through as well. Sean __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
On 08/02/2018 10:43 AM, Andrey Kurilin wrote: > There's also some "raise StopIteration" issues in: > - ceilometer > - cinder > - designate > - glance > - glare > - heat > - karbor > - manila > - murano > - networking-ovn > - neutron-vpnaas > - nova > - rally > > > Can you provide any traceback or steps to reproduce the issue for Rally > project ? I'm not sure there's any. The only thing I know is that it has stop StopIteration stuff, but I'm not sure if they are part of generators, in which case they should simply be replaced by "return" if you want it to be py 3.7 compatible. I didn't have time to investigate these, but at least Glance was affected, and a patch was sent (as well as an async patch). None of them has been merged yet: https://review.openstack.org/#/c/586050/ https://review.openstack.org/#/c/586716/ That'd be ok if at least there was some reviews. It looks like nobody cares but Debian & Ubuntu people... :( Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
Hi Thomas! On Thu, 2 Aug 2018 at 06:13, Thomas Goirand wrote: > On 07/12/2018 10:38 PM, Thomas Goirand wrote: > > Hi everyone! > > > > [...] > Here's more examples that shows why we should be gating earlier with > newer Python versions: > > Nova: > https://review.openstack.org/#/c/584365/ > > Glance: > https://review.openstack.org/#/c/586716/ > > Murano: > https://bugs.debian.org/904581 > > Pyghmi: > https://bugs.debian.org/905213 > > There's also some "raise StopIteration" issues in: > - ceilometer > - cinder > - designate > - glance > - glare > - heat > - karbor > - manila > - murano > - networking-ovn > - neutron-vpnaas > - nova > - rally Can you provide any traceback or steps to reproduce the issue for Rally project ? > > - zaqar > > It'd be nice to have these addressed ASAP. > > Cheers, > > Thomas Goirand (zigo) > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Best regards, Andrey Kurilin. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
On 07/12/2018 10:38 PM, Thomas Goirand wrote: > Hi everyone! > > [...] Here's more examples that shows why we should be gating earlier with newer Python versions: Nova: https://review.openstack.org/#/c/584365/ Glance: https://review.openstack.org/#/c/586716/ Murano: https://bugs.debian.org/904581 Pyghmi: https://bugs.debian.org/905213 There's also some "raise StopIteration" issues in: - ceilometer - cinder - designate - glance - glare - heat - karbor - manila - murano - networking-ovn - neutron-vpnaas - nova - rally - zaqar It'd be nice to have these addressed ASAP. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
On 07/18/2018 06:42 AM, Ian Wienand wrote: > While I'm reserved about the > idea of full platform functional tests, essentially having a > wide-variety of up-to-date tox environments using some of the methods > discussed there is, I think, a very practical way to be cow-catching > some of the bigger issues with Python version updates. If we are to > expend resources, my 2c worth is that pushing in that direction gives > the best return on effort. > > -i Hi Ian, Thanks a lot for your reply, that's very useful. I very much agree that testing the latest Qemu / libvirt could be a problem if it fails too often, and same with other components, however, these needs to be addressed anyway at some point. If we can't do it this way, then we have to define a mechanism to find out. Maybe a dvsm periodic task unrelated to a specific project would do? Anyway, my post was *not* about functional testing, so let's not talk about this. What I would love to get addressed is catching problems with newer language updates. Having them early avoids downstream distribution doing the heavy work, which is not sustainable considering the amount of people (which is about 1 or 2 guys per distro), and that's what I would like to be addressed. For example, "async" becoming a keyword in Python 3.7 is something I would have very much like to be caught by some kind of upstream CI running unit tests, rather than Debian and Ubuntu package maintainers fixing the problems as we get FTBFS (Fails To Build From Source) bugs filed in the BTS, and when we find out by ourselves that some package cannot be installed or built. This happened with oslo.messaging, taskflow, etc. This is just the new Python 3.7 things, though there was numerous problems with Python 3.6. Currently, it looks like Heat also has unit test failures in Sid (not sure yet what the issue is). Waiting for Bionic to be released to start gating unit tests on Python 3.6 is IMO a way too late, as for example Debian Sid was running Python 3.6 about a year before that, and that's what I would like to be fixed. Using either Fedora or SuSE is fine to me, as long as it gets latest Python language fast enough (does it go as fast as Debian testing?). If it's for doing unit testing only (ie: no functional tests using Qemu, libvirt and other component of this type) looks like a good plan. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
On Thu, Jul 12, 2018, at 1:38 PM, Thomas Goirand wrote: > Hi everyone! > > It's yet another of these emails where I'm going to complain out of > frustration because of OpenStack having bugs when running with the > newest stuff... Sorry in advance ! :) > > tl;dr: It's urgent, we need Python 3.7 uwsgi + SSL gate jobs. > > Longer version: > > When Python 3.6 reached Debian, i already forwarded a few patches. It > went quite ok, but still... When switching services to Python 3 for > Newton, I discover that many services still had issues with uwsgi / > mod_wsgi, and I spent a large amount of time trying to figure out ways > to fix the situation. Some patches are still not yet merged, even though > it was a community goal to have this support for Newton: > > Neutron: > https://review.openstack.org/#/c/555608/ > https://review.openstack.org/#/c/580049/ > > Neutron FWaaS: > https://review.openstack.org/#/c/580327/ > https://review.openstack.org/#/c/579433/ > > Horizon tempest plugin: > https://review.openstack.org/#/c/575714/ > > Oslotet (clearly, the -1 is for someone considering only Devstack / > venv, not understanding packaging environment): > https://review.openstack.org/#/c/571962/ > > Designate: > As much as I know, it still doesn't support uwsgi / mod_wsgi (please let > me know if this changed recently). > > There may be more, I didn't have much time investigating some projects > which are less important to me. > > Now, both Debian and Ubuntu have Python 3.7. Every package which I > upload in Sid need to support that. Yet, OpenStack's CI is still lagging > with Python 3.5. And there's lots of things currently broken. We've > fixed most "async" stuff, though we are failing to rebuild > oslo.messaging (from Queens) with Python 3.7: unit tests are just > hanging doing nothing. > > I'm very happy to do small contributions to each and every component > here and there whenever it's possible, but this time, it's becoming a > little bit frustrating. I sometimes even got replies like "hum ... > OpenStack only supports Python 3.5" a few times. That's not really > acceptable, unfortunately. > > So moving forward, what I think needs to happen is: > > - Get each and every project to actually gate using uwsgi for the API, > using both Python 3 and SSL (any other test environment is *NOT* a real > production environment). > > - The gating has to happen with whatever is the latest Python 3 version > available. Best would even be if we could have that *BEFORE* it reaches > distributions like Debian and Ubuntu. I'm aware that there's been some > attempts in the OpenStack infra to have Debian Sid (which is probably > the distribution getting the updates the faster). This effort needs to > be restarted, and some (non-voting ?) gate jobs needs to be setup using > whatever the latest thing is. If it cannot happen with Sid, then I don't > know, choose another platform, and do the Python 3-latest gating... When you asked about this last month I suggested Tumbleweed as an option. You get rolling release packages that are almost always up to date. I'd still suggest that now as a place to start. http://lists.openstack.org/pipermail/openstack-dev/2018-June/131302.html > > The current situation with the gate still doing Python 3.5 only jobs is > just not sustainable anymore. Moving forward, Python 2.7 will die. When > this happens, moving faster with Python 3 versions will be mandatory for > everyone, not only for fools like me who made the switch early. > > :) > > Cheers, > > Thomas Goirand (zigo) > > P.S: A big thanks to everyone who where helpful for making the switch to > Python 3 in Debian, especially Annp and the rest of the Neutron team. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
On 07/18/2018 12:42 AM, Ian Wienand wrote: The ideal is that a (say) Neutron dev gets a clear traceback from a standard Python error in their change and happily fixes it. The reality is probably more like this developer gets a tempest failure due to nova failing to boot a cirros image, stemming from a detached volume due to a qemu bug that manifests due to a libvirt update (I'm exaggerating, I know :). Not really exaggerating. :) -jay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
On 07/13/2018 06:38 AM, Thomas Goirand wrote: > Now, both Debian and Ubuntu have Python 3.7. Every package which I > upload in Sid need to support that. Yet, OpenStack's CI is still > lagging with Python 3.5. OpenStack's CI is rather broad -- I'm going to assume we're talking about whole-system devstack-ish based functional tests. Yes most testing is on Xenial and hence Python 3.5 We have Python 3.6 available via Bionic nodes. I think current talk is to look at mass-updates after the next release. Such updates, from history, are fairly disruptive. > I'm aware that there's been some attempts in the OpenStack infra to > have Debian Sid (which is probably the distribution getting the > updates the faster). We do not currently build Debian sid images, or mirror the unstable repos or do wheel builds for Debian. diskimage-builder also doesn't test it in CI. This is not to say it can't be done. > If it cannot happen with Sid, then I don't know, choose another > platform, and do the Python 3-latest gating... Fedora has been consistently updated in OpenStack Infra for many years. IMO, and from my experience, six-monthly-ish updates are about as frequent as can be practically handled. The ideal is that a (say) Neutron dev gets a clear traceback from a standard Python error in their change and happily fixes it. The reality is probably more like this developer gets a tempest failure due to nova failing to boot a cirros image, stemming from a detached volume due to a qemu bug that manifests due to a libvirt update (I'm exaggerating, I know :). That sort of deeply tangled platform issue always exists; however it is armortised across the lifespan of the testing. So several weeks after we update all these key components, a random Neutron dev can be pretty sure that submitting their change is actually testing *their* change, and not really a defacto test of every other tangentially related component. A small, but real example; uwsgi wouldn't build with the gcc/glibc combo on Fedora 28 for two months after its release until uwsgi's 2.0.17.1. Fedora carried patches; but of course there were a lot previously unconsidered assumptions in devstack around deployment that made using the packaged versions difficult [1] (that stack still hasn't received any reviews). Nobody would claim diskimage-builder is the greatest thing ever, but it does produce our customised images in a wide variety of formats that runs in our very heterogeneous clouds. It's very reactive -- we don't know about package updates until they hit the distro, and sometimes that breaks assumptions. It's largely taken for granted in our CI, but it takes a constant sustained effort across the infra team to make sure we have somewhere to test. I hear myself sounding negative, but I think it's a fundamental problem. You can't be dragging in the latest of everything AND expect that you won't be constantly running off fixing weird things you never even knew existed. We can (and do) get to the bottom of these things, but if the platform changes again before you've even fixed the current issue, things start piling up. If the job is constantly broken it gets ignored -- if a non-voting job fails in the woods, does it make a sound? :) > When this happens, moving faster with Python 3 versions will be > mandatory for everyone, not only for fools like me who made the > switch early. This is a long way of saying that - IMO - the idea of putting out a Debian sid image daily (to a lesser degree Buster images) and throwing a project's devstack runs against it is unlikely to produce a good problems-avoided : development-resources ratio. However, prove me wrong :) If people would like to run their master against Fedora (note OpenStack's stable branch lifespan is generally longer than a given Fedora release is supported, so it is not much good there) you have later packages, but still a fairly practical 6-month-ish stability cadence. I'm happy to help (some projects do already). > With my rant done :) ... there's already discussion around multiple python versions, containers, etc in [2]. While I'm reserved about the idea of full platform functional tests, essentially having a wide-variety of up-to-date tox environments using some of the methods discussed there is, I think, a very practical way to be cow-catching some of the bigger issues with Python version updates. If we are to expend resources, my 2c worth is that pushing in that direction gives the best return on effort. -i [1] https://review.openstack.org/#/c/565923/ [2] http://lists.openstack.org/pipermail/openstack-dev/2018-July/132152.html __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate
Hi everyone! It's yet another of these emails where I'm going to complain out of frustration because of OpenStack having bugs when running with the newest stuff... Sorry in advance ! :) tl;dr: It's urgent, we need Python 3.7 uwsgi + SSL gate jobs. Longer version: When Python 3.6 reached Debian, i already forwarded a few patches. It went quite ok, but still... When switching services to Python 3 for Newton, I discover that many services still had issues with uwsgi / mod_wsgi, and I spent a large amount of time trying to figure out ways to fix the situation. Some patches are still not yet merged, even though it was a community goal to have this support for Newton: Neutron: https://review.openstack.org/#/c/555608/ https://review.openstack.org/#/c/580049/ Neutron FWaaS: https://review.openstack.org/#/c/580327/ https://review.openstack.org/#/c/579433/ Horizon tempest plugin: https://review.openstack.org/#/c/575714/ Oslotet (clearly, the -1 is for someone considering only Devstack / venv, not understanding packaging environment): https://review.openstack.org/#/c/571962/ Designate: As much as I know, it still doesn't support uwsgi / mod_wsgi (please let me know if this changed recently). There may be more, I didn't have much time investigating some projects which are less important to me. Now, both Debian and Ubuntu have Python 3.7. Every package which I upload in Sid need to support that. Yet, OpenStack's CI is still lagging with Python 3.5. And there's lots of things currently broken. We've fixed most "async" stuff, though we are failing to rebuild oslo.messaging (from Queens) with Python 3.7: unit tests are just hanging doing nothing. I'm very happy to do small contributions to each and every component here and there whenever it's possible, but this time, it's becoming a little bit frustrating. I sometimes even got replies like "hum ... OpenStack only supports Python 3.5" a few times. That's not really acceptable, unfortunately. So moving forward, what I think needs to happen is: - Get each and every project to actually gate using uwsgi for the API, using both Python 3 and SSL (any other test environment is *NOT* a real production environment). - The gating has to happen with whatever is the latest Python 3 version available. Best would even be if we could have that *BEFORE* it reaches distributions like Debian and Ubuntu. I'm aware that there's been some attempts in the OpenStack infra to have Debian Sid (which is probably the distribution getting the updates the faster). This effort needs to be restarted, and some (non-voting ?) gate jobs needs to be setup using whatever the latest thing is. If it cannot happen with Sid, then I don't know, choose another platform, and do the Python 3-latest gating... The current situation with the gate still doing Python 3.5 only jobs is just not sustainable anymore. Moving forward, Python 2.7 will die. When this happens, moving faster with Python 3 versions will be mandatory for everyone, not only for fools like me who made the switch early. :) Cheers, Thomas Goirand (zigo) P.S: A big thanks to everyone who where helpful for making the switch to Python 3 in Debian, especially Annp and the rest of the Neutron team. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
