Excerpts from Thomas Goirand's message of 2018-06-07 22:17:11 +0200: > On 06/06/2018 10:04 PM, Doug Hellmann wrote: > > I have started submitting a series of patches to fix up the tox.ini > > settings for projects as a step towards running "python3 first" > > [1]. The point of doing this now is to give teams a head start on > > understanding the work involved as we consider whether to make this > > a community goal. > > > > The current patches are all mechanically generated changes to the > > basepython value for environments that seem to be likely candidates. > > They're basically the "easy" part of the transition. I've left any > > changes that will need more discussion alone for now. > > > > In particular, I've skipped over any tox environments with "functional" > > in the name, since I thought those ran functional tests. Teams will > > need to decide whether to change those job definitions, or duplicate > > them and run them under python 2 and 3. Since we are not dropping > > python 2 support until the U cycle, I suggest going ahead and running > > the jobs twice. > > > > Note that changing the tox settings won't actually change some of the > > jobs. For example, with our current PTI definition, the documentation > > and releasenotes jobs do not run under tox. That means those will need > > to be changed by editing the zuul configuration for the repository. > > > > I have started to make notes for tracking the work in > > https://etherpad.openstack.org/p/python3-first -- including some notes > > about taking the next step to update the zuul job definitions and common > > issues we've already encountered to help folks debug job failures. > > > > I could use some help keeping an eye on these changes and getting > > them through the gate. If you are interested in helping, please > > leave a comment on the review you are willing to shepherd. > > > > Doug > > > > [1] > > https://review.openstack.org/#/q/topic:python3-first+(status:open+OR+status:merged) > > tl;dr: all projects must move their functional tests under Py3, uwsgi > and SSL, otherwise there's undetected issues. > > Doug, > > Attempting to make puppet-openstack fully work for Debian (more on this > soon as hopefully I'll be able to announce soon the general availability > and success with testing), and having all Debian packages running in > Python 3, what I have discovered is that there's a general issue when > running tests: they do not take into account the fact that we cannot run > with Eventlet, because Python 3 + Eventlet + SSL = handshake crash. > > In other words, all projects need to run their API functional tests > using either mod_wsgi or uwsgi (and more likely the later), using Python > 3 *AND* SSL, as this is how operators will run. Unfortunately, this is > far from being the case at the moment, and I have been able to find > defects in at least 3 services, and sometimes, helped, or contributed > patches myself. > > More in details, projects after projects, just so you understand what I > mean. No finger pointing here, these are just to illustrate my > argumentation and show the kind of problems we're having. > > sahara-api looks completely broken under Python 3 (ie: error 500 on each > GET requests), unless this tiny 3 liner patch is applied > https://review.openstack.org/#/c/573179/ (hopefully, a better patch will > happen upstream, I'm just a package maintainer not having enough time to > get involved deep enough in each individual projects...) > > Cinder-volume fails to load with Ceph driver if using Luminous and > Python 3 unless this patch is applied: > https://review.openstack.org/#/c/568813/ > > neutron-server, under Python 3 and SSL, has to be transformed into > neutron-api served over uwsgi (probably also works using mod_wsgi). In > such case, you must run neutron-rpc-server to handle messages from the > bus. But then, the rpc-server doesn't understand plugin modules unless > this patch is added: https://review.openstack.org/555608 and also, > firewall rules aren't received by the openvswitch-agent (and therefore, > security group rules aren't being applied to iptables) unless this patch > is reverted: https://review.openstack.org/434678 and then it works for > openvswitch agent, though for linuxbridge, I still couldn't ping > floating IPs (I'm still investigating the problem, I'm not sure yet if > it's a routing problem or what...). And that's not even over, as it > looks like tempest's scenario > test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops is > still broken with openvswitch for me: > http://logs.openstack.org/28/564328/48/check/puppet-openstack-integration-4-scenario001-tempest-debian-stable/5e64694/job-output.txt.gz#_2018-06-07_12_42_50_760463 > Help is desperately wanted here as I spent a really long time on this > already, it's still failing, and I don't understand what's going on! > > All this to say, the more general issue here is that projects, while > unit testing under Python 3, aren't running functional tests the same > way that operators will (ie: with SSL and uwsgi/mod_wsgi), and because > of that, some problems aren't being detected. Just a hint, to run > something like neutron-api with ipv4, 6 and SSL, you'd run it this way: > > /usr/bin/uwsgi_python35 --https-socket > [::]:9696,/usr/local/share/ca-certificates/puppet_openstack.crt,/etc/neutron/ssl/private/debian-stretch-ovh-gra1-0004341140.pem > --ini /etc/neutron/neutron-api-uwsgi.ini --pyargv > "--config-file=/etc/neutron/neutron.conf > --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini > --config-file=/etc/neutron/fwaas_driver.ini" > > Notice the [::] meaning ipv4 AND ipv6 at the same time, and how --pyargv > needs a quoted list of arguments. > > Hopefully, this is slowly being addressed, though I would very much like > if there was a general move to using uwsgi, preferably with SSL and ipv6 > turned on and tested. > > It'd be also nice if projects could ship the uwsgi ini file that they > use for functional tests, as it looks like it sometimes depend on the > project (for example, I had to add rem-header: content-length for > nova-placement-api to avoid connection reset by peer, but I'm not sure > if it even is harmful for other projects). Here's an example from the > Debian packaging: > > https://salsa.debian.org/openstack-team/services/neutron/blob/debian/queens/debian/neutron-api-uwsgi.ini > > See how I'm deliberately *not* setting "pyargv" there, and prefer it to > be set by the init script / systemd service (so that it can be dynamic > and load the configuration file needed for the activated plugin), and > that http-socket / https-socket is also dynamic, ie: --https-socket is > used on the command line if a pair of certificate + private key is found > on the hard disk under /etc/neutron/ssl. See > https://salsa.debian.org/openstack-team/services/neutron/blob/debian/queens/debian/neutron-api.init.in > and > https://salsa.debian.org/openstack-team/debian/openstack-pkg-tools/tree/debian/queens/init-template > to understand how it's built. > > Hoping that these packaging-related insights are helpful for the project > at large, cheers, > > Thomas Goirand (zigo) >
Thanks for all of these details, Thomas, I will work on summarizing these notes in https://etherpad.openstack.org/p/python3-first so project teams can consider which parts relate to their services. Doug __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
