Eventlet released 0.16.1 on 2015-01-14 , which removed a deprecated API that nova stable/* still used. This caused nova-compute in stable/juno and stable/icehouse to crash thus breaking grenade on master. In 24 hours this bug caused 671 grenade jobs to fail!
After some quick debugging of this cryptic error suppressing stacktrace  we got to the real stacktrace: " File "nova/virt/libvirt/__init__.py", line 15, in <module> from nova.virt.libvirt import driver File "nova/virt/libvirt/driver.py", line 48, in <module> from eventlet import util as eventlet_util ImportError: cannot import name util"  and capped the eventlet version on stable/*  As dependency bugs go this was a pretty typical one, so why am I writing this? Because we knew about this bug before it hit the gate, yet it was still an issue. The util module was removed in 0.16.0 but 'sneaked into' the build  so 0.16.1 fixed that. Before 0.16.1 was released this bug only impacted people who installed eventlet 0.16.0 from source and not from pip. Luckily someone did this and filed a bug  and a fix for this was landed on master on January 7th, and by the 10th the fix was backported to stable/juno and ready . In the mean time, we had an unusually bad week dealing with new library versions; boto 2.35.0 was released and broke master and stable/juno  and sqlalchemy-migrate broke glance on master and stable as well . And by the time these issues were fixed, and the tests would pass, eventlet 0.16.1 was already out. Unfortunately, there are a very small number of people who work on fixing dependency issues, some of them were traveling and the rest were over worked, so figuring out what went wrong with all the new packages came down to a handful of overworked people. So how could we have avoided this problem? By capping stable branch requirements so we only have to worry about uncapped dependencies on master. Capping stable branches has been previous discussed but no action has been taken. So going forward I propose we pin all requirements, including transitive, on stable branches. This way the release of new dependencies cannot automatically break stable branches and thus break grenade on master. I think we can implement this using `pip freeze` to get a list of all the installed libraries and `pip install -r --no-deps` to install the specific package versions. I am not sure how best to handle package versions being removed from pypi though. Or at the very least we can cap requirements in global requirements and at least reduce the number of packages we install uncapped versions of. Thoughts? best, Joe  https://pypi.python.org/pypi/eventlet/0.16.1  http://status.openstack.org/elastic-recheck/#1410626  http://logs.openstack.org/98/115998/59/check/check-grenade-dsvm/e57eb40/logs/old/screen-n-cpu.txt.gz  http://paste.openstack.org/show/157558/  https://review.openstack.org/#/q/I4bbbeb5bf9c22ed36f5c9a74fec6b487d2c15697,n,z  https://github.com/eventlet/eventlet/releases/tag/v0.16.1  https://bugs.launchpad.net/nova/+bug/1407685  https://review.openstack.org/#/q/Idbb9d2b53829dae0e807cd1260dee3dce155d5f3,n,z  https://bugs.launchpad.net/nova/+bug/1408987  https://bugs.launchpad.net/glance/+bug/1410494
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev