On 06/06/2018 10:04 PM, Doug Hellmann wrote: > I have started submitting a series of patches to fix up the tox.ini > settings for projects as a step towards running "python3 first" > [1]. The point of doing this now is to give teams a head start on > understanding the work involved as we consider whether to make this > a community goal. > > The current patches are all mechanically generated changes to the > basepython value for environments that seem to be likely candidates. > They're basically the "easy" part of the transition. I've left any > changes that will need more discussion alone for now. > > In particular, I've skipped over any tox environments with "functional" > in the name, since I thought those ran functional tests. Teams will > need to decide whether to change those job definitions, or duplicate > them and run them under python 2 and 3. Since we are not dropping > python 2 support until the U cycle, I suggest going ahead and running > the jobs twice. > > Note that changing the tox settings won't actually change some of the > jobs. For example, with our current PTI definition, the documentation > and releasenotes jobs do not run under tox. That means those will need > to be changed by editing the zuul configuration for the repository. > > I have started to make notes for tracking the work in > https://etherpad.openstack.org/p/python3-first -- including some notes > about taking the next step to update the zuul job definitions and common > issues we've already encountered to help folks debug job failures. > > I could use some help keeping an eye on these changes and getting > them through the gate. If you are interested in helping, please > leave a comment on the review you are willing to shepherd. > > Doug > > [1] > https://review.openstack.org/#/q/topic:python3-first+(status:open+OR+status:merged)
tl;dr: all projects must move their functional tests under Py3, uwsgi and SSL, otherwise there's undetected issues. Doug, Attempting to make puppet-openstack fully work for Debian (more on this soon as hopefully I'll be able to announce soon the general availability and success with testing), and having all Debian packages running in Python 3, what I have discovered is that there's a general issue when running tests: they do not take into account the fact that we cannot run with Eventlet, because Python 3 + Eventlet + SSL = handshake crash. In other words, all projects need to run their API functional tests using either mod_wsgi or uwsgi (and more likely the later), using Python 3 *AND* SSL, as this is how operators will run. Unfortunately, this is far from being the case at the moment, and I have been able to find defects in at least 3 services, and sometimes, helped, or contributed patches myself. More in details, projects after projects, just so you understand what I mean. No finger pointing here, these are just to illustrate my argumentation and show the kind of problems we're having. sahara-api looks completely broken under Python 3 (ie: error 500 on each GET requests), unless this tiny 3 liner patch is applied https://review.openstack.org/#/c/573179/ (hopefully, a better patch will happen upstream, I'm just a package maintainer not having enough time to get involved deep enough in each individual projects...) Cinder-volume fails to load with Ceph driver if using Luminous and Python 3 unless this patch is applied: https://review.openstack.org/#/c/568813/ neutron-server, under Python 3 and SSL, has to be transformed into neutron-api served over uwsgi (probably also works using mod_wsgi). In such case, you must run neutron-rpc-server to handle messages from the bus. But then, the rpc-server doesn't understand plugin modules unless this patch is added: https://review.openstack.org/555608 and also, firewall rules aren't received by the openvswitch-agent (and therefore, security group rules aren't being applied to iptables) unless this patch is reverted: https://review.openstack.org/434678 and then it works for openvswitch agent, though for linuxbridge, I still couldn't ping floating IPs (I'm still investigating the problem, I'm not sure yet if it's a routing problem or what...). And that's not even over, as it looks like tempest's scenario test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops is still broken with openvswitch for me: http://logs.openstack.org/28/564328/48/check/puppet-openstack-integration-4-scenario001-tempest-debian-stable/5e64694/job-output.txt.gz#_2018-06-07_12_42_50_760463 Help is desperately wanted here as I spent a really long time on this already, it's still failing, and I don't understand what's going on! All this to say, the more general issue here is that projects, while unit testing under Python 3, aren't running functional tests the same way that operators will (ie: with SSL and uwsgi/mod_wsgi), and because of that, some problems aren't being detected. Just a hint, to run something like neutron-api with ipv4, 6 and SSL, you'd run it this way: /usr/bin/uwsgi_python35 --https-socket [::]:9696,/usr/local/share/ca-certificates/puppet_openstack.crt,/etc/neutron/ssl/private/debian-stretch-ovh-gra1-0004341140.pem --ini /etc/neutron/neutron-api-uwsgi.ini --pyargv "--config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --config-file=/etc/neutron/fwaas_driver.ini" Notice the [::] meaning ipv4 AND ipv6 at the same time, and how --pyargv needs a quoted list of arguments. Hopefully, this is slowly being addressed, though I would very much like if there was a general move to using uwsgi, preferably with SSL and ipv6 turned on and tested. It'd be also nice if projects could ship the uwsgi ini file that they use for functional tests, as it looks like it sometimes depend on the project (for example, I had to add rem-header: content-length for nova-placement-api to avoid connection reset by peer, but I'm not sure if it even is harmful for other projects). Here's an example from the Debian packaging: https://salsa.debian.org/openstack-team/services/neutron/blob/debian/queens/debian/neutron-api-uwsgi.ini See how I'm deliberately *not* setting "pyargv" there, and prefer it to be set by the init script / systemd service (so that it can be dynamic and load the configuration file needed for the activated plugin), and that http-socket / https-socket is also dynamic, ie: --https-socket is used on the command line if a pair of certificate + private key is found on the hard disk under /etc/neutron/ssl. See https://salsa.debian.org/openstack-team/services/neutron/blob/debian/queens/debian/neutron-api.init.in and https://salsa.debian.org/openstack-team/debian/openstack-pkg-tools/tree/debian/queens/init-template to understand how it's built. Hoping that these packaging-related insights are helpful for the project at large, cheers, Thomas Goirand (zigo) __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev