+1 how about the deep dependency? for example, we depends package A, and pin it, but A->B->C, then B and C are not pined since they are not directly depended, then what should we do? pin everything?
On Thu, Jan 15, 2015 at 5:35 PM, Joe Gordon <joe.gord...@gmail.com> wrote: > Eventlet released 0.16.1 on 2015-01-14 [0], which removed a deprecated > API that nova stable/* still used. This caused nova-compute in stable/juno > and stable/icehouse to crash thus breaking grenade on master. In 24 hours > this bug caused 671 grenade jobs to fail[1]! > > After some quick debugging of this cryptic error suppressing stacktrace > [2] we got to the real stacktrace: > " File "nova/virt/libvirt/__init__.py", line 15, in <module> > > from nova.virt.libvirt import driver > File "nova/virt/libvirt/driver.py", line 48, in <module> > from eventlet import util as eventlet_util > > ImportError: cannot import name util" [3] > and capped the eventlet version on stable/* [4] > > As dependency bugs go this was a pretty typical one, so why am I writing > this? Because we knew about this bug before it hit the gate, yet it was > still an issue. The util module was removed in 0.16.0 but 'sneaked into' > the build [5] so 0.16.1 fixed that. Before 0.16.1 was released this bug > only impacted people who installed eventlet 0.16.0 from source and not from > pip. Luckily someone did this and filed a bug [6] and a fix for this was > landed on master on January 7th, and by the 10th the fix was backported to > stable/juno and ready [7]. In the mean time, we had an unusually bad week > dealing with new library versions; boto 2.35.0 was released and broke > master and stable/juno [8] and sqlalchemy-migrate broke glance on master > and stable as well [9]. And by the time these issues were fixed, and the > tests would pass, eventlet 0.16.1 was already out. Unfortunately, there are > a very small number of people who work on fixing dependency issues, some of > them were traveling and the rest were over worked, so figuring out what > went wrong with all the new packages came down to a handful of overworked > people. > > So how could we have avoided this problem? By capping stable branch > requirements so we only have to worry about uncapped dependencies on > master. Capping stable branches has been previous discussed but no action > has been taken. So going forward I propose we pin all requirements, > including transitive, on stable branches. This way the release of new > dependencies cannot automatically break stable branches and thus break > grenade on master. > > I think we can implement this using `pip freeze` to get a list of all the > installed libraries and `pip install -r --no-deps` to install the specific > package versions. I am not sure how best to handle package versions being > removed from pypi though. Or at the very least we can cap requirements in > global requirements and at least reduce the number of packages we install > uncapped versions of. > > Thoughts? > > best, > Joe > > [0] https://pypi.python.org/pypi/eventlet/0.16.1 > [1] http://status.openstack.org/elastic-recheck/#1410626 > [2] > http://logs.openstack.org/98/115998/59/check/check-grenade-dsvm/e57eb40/logs/old/screen-n-cpu.txt.gz > [3] http://paste.openstack.org/show/157558/ > [4] > https://review.openstack.org/#/q/I4bbbeb5bf9c22ed36f5c9a74fec6b487d2c15697,n,z > [5] https://github.com/eventlet/eventlet/releases/tag/v0.16.1 > [6] https://bugs.launchpad.net/nova/+bug/1407685 > [7] > https://review.openstack.org/#/q/Idbb9d2b53829dae0e807cd1260dee3dce155d5f3,n,z > [8] https://bugs.launchpad.net/nova/+bug/1408987 > [9] https://bugs.launchpad.net/glance/+bug/1410494 > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- blog: zqfan.github.com git: github.com/zqfan
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev