-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/12/2013 09:39 AM, Anita Kuno wrote: > On 12/12/2013 08:20 AM, Sean Dague wrote: >> Current Gate Length: 12hrs*, 41 deep > >> (top of gate entered 12hrs ago) > >> It's been an *exciting* week this week. For people not paying >> attention we had 2 external events which made things terrible >> earlier in the week. > >> ========================== Event 1: sphinx 1.2 complete breakage >> - MOSTLY RESOLVED ========================== > >> It turns out sphinx 1.2 + distutils (which pbr magic call >> through) means total sadness. The fix for this was a requirements >> pin to sphinx < 1.2, and until a project has taken that they will >> fail in the gate. > >> It also turns out that tox installs pre-released software by >> default (a terrible default behavior), so you also need a >> tox.ini change like this - >> https://github.com/openstack/nova/blob/master/tox.ini#L9 >> otherwise local users will install things like sphinx 1.2b3. They >> will also break in other ways. > >> Not all projects have merged this. If you are a project that >> hasn't, please don't send any other jobs to the gate until you >> do. A lot of delay was added to the gate yesterday by Glance >> patches being pushed to the gate before their doc jobs were >> done. > >> ========================== Event 2: apt.puppetlabs.com outage - >> RESOLVED ========================== > >> We use that apt repository to setup the devstack nodes in >> nodepool with puppet. We were triggering an issue with grenade >> where it's apt-get calls were failing, because it does apt-get >> update once to make sure life is good. This only triggered in >> grenade (noth other devstack runs) because we do set -o errexit >> aggressively. > >> A fix in grenade to ignore these errors was merged yesterday >> afternoon (the purple line - >> http://status.openstack.org/elastic-recheck/ you can see where >> it showed up). > >> ========================== Top Gate Bugs >> ========================== > >> We normally do this as a list, and you can see the whole list >> here - http://status.openstack.org/elastic-recheck/ (now sorted >> by number of FAILURES in the last 2 weeks) > >> That being said, our bigs race bug is currently this one bug - >> https://bugs.launchpad.net/tempest/+bug/1253896 - and if you >> want to merge patches, fixing that one bug will be huge. > >> Basically, you can't ssh into guests that get created. That's >> sort of a fundamental property of a cloud. It shows up more >> frequently on neutron jobs, possibly due to actually testing the >> metadata server path. There have been many attempts on retry >> logic on this, we actually retry for 196 seconds to get in and >> only fail once we can't get in, so waiting isn't helping. It >> doesn't seem like the env is under that much load. > >> Until we resolve this, life will not be good in landing patches. > >> -Sean > > > >> _______________________________________________ OpenStack-dev >> mailing list [email protected] >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > Thanks Sean: > > This is a terrific summary which really makes my task of > confirming and following up much more manageable. > > Just by way of preempting the "its neutron's fault" pile-on, just > in case anyone is tempted, a few facts: > > We were paying attention, as it happens to the sphinx pin. Patches > to neutron and neutronclient have merged: > http://git.openstack.org/cgit/openstack/neutron/tree/test-requirements.txt#n9 > > http://git.openstack.org/cgit/openstack/python-neutronclient/tree/test-requirements.txt#n9 > A post to the mailing list isn't complete it appears unless I make a mistake in it. Here is our patch for neutronclient: https://review.openstack.org/#/c/61508/
> The addition of the -U flag for pip install in tox.ini for > neutronclient https://review.openstack.org/#/c/60825/4 is in the > check queue, it tripped on the sphinx pin for neutronclient. Here > is the one for neutron: https://review.openstack.org/#/c/60825/4 > I've just alerted all neutron-core to refrain from +A'ing until > these are merged. > > We had been tracking 1253896 quite closely and I at least was > working wtih the belief we had done the work we needed to do for > that bug. Since it now comes to light that I am in error with > regards to neutron's responsibility to 1253896, I welcome all > interested parties to #openstack-neutron so that we can again work > together to submit a patch that addresses this issue. > > Thanks Sean, Anita. > > > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJSqdRnAAoJELmKyZugNFU0H80IAMnjI4r3ulePeZ3eenAwOYlY TEC6hAClr374oq1B7Tk3JYP0dl+qMed7TaCSYI32wB9sVbYKsjKrjsNXARDpqqUe r1Mb05jknDZtwfwBYNsoPtsn/sf9mtXm8T+Czk7ojMVogwjng8ps4juyG4ZveUNl +zDkPtFgoWrzKA/pmUIfue5RgaH0MfX14ftojUmSroYfTuMK5fxSPc9mhV//2ZBw JyNowczOdhOVf4pBZ7JFDDd1potoi+CtrOHxW13aUC5XyPLN/HU3ZoZRoWVeLzDT FWgHgWo1DoK49+R12KsjzX0xhUVs2YV3oVrraPnXsMSqiWzWOU1xNglGwa6qWdg= =PIFo -----END PGP SIGNATURE----- _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
