Re: [openstack-dev] Top Gate Reseting issues that need attention - Tuesday morning update

2014-01-21 Thread Sean Dague
Brief update on where we stand on the gate (still not great) - gate is currently 126 deep - top of queue entered 51hrs ago Bug 1270680 - v3 extensions api inherently racey wrt instances - patch landed (seems to have helped though the exception is still showing up quite a bit, so don't know if

Re: [openstack-dev] Top Gate Reseting issues that need attention - Tuesday morning update

2014-01-21 Thread Russell Bryant
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/21/2014 07:14 AM, Sean Dague wrote: Brief update on where we stand on the gate (still not great) - gate is currently 126 deep - top of queue entered 51hrs ago Bug 1270680 - v3 extensions api inherently racey wrt instances - patch landed

Re: [openstack-dev] Top Gate Reseting issues that need attention - Tuesday morning update

2014-01-21 Thread Salvatore Orlando
I have some hints which the people looking at neutron failures might find useful. # 1 - in [1] a weird thing happens with DHCP. A DHCPDISCOVER with for fa:16:3e:cc:d9:c7 is pretty much simultaneously received by two dnsmasq instances, which are listening on ports belonging to two distinct

[openstack-dev] Top Gate Reseting issues that need attention

2014-01-20 Thread Sean Dague
Anyone that's looked at the gate this morning... knows things aren't good. It turns out that a few new races got into OpenStack last week, which are causing a ton of pain, and have put us dramatically over the edge. We've not tracked down all of them, but 2 that are quite important to address

Re: [openstack-dev] Top Gate Reseting issues that need attention

2014-01-20 Thread Eugene Nikanorov
Hi Sean, I think the following 2 commits in neutron are essential for bringing neutron jobs back to acceptable level of failure rate: https://review.openstack.org/#/c/67537/ https://review.openstack.org/#/c/66670/ Thanks, Eugene. On Mon, Jan 20, 2014 at 6:33 PM, Sean Dague s...@dague.net

Re: [openstack-dev] Top Gate Reseting issues that need attention

2014-01-20 Thread Salvatore Orlando
I gave a -2 yesterday to all my Neutron patches. I did that because I thought something was wrong with them, but then I started to realize it's a general problem. It makes sense to give some priority to the patches Eugene linked, even if it would be better to have some people root causing the

Re: [openstack-dev] Top Gate Reseting issues that need attention

2014-01-20 Thread Jay Pipes
On Mon, 2014-01-20 at 18:43 +0400, Eugene Nikanorov wrote: Hi Sean, I think the following 2 commits in neutron are essential for bringing neutron jobs back to acceptable level of failure rate: https://review.openstack.org/#/c/67537/ https://review.openstack.org/#/c/66670/ That second patch

Re: [openstack-dev] Top Gate Reseting issues that need attention

2014-01-20 Thread Robert Collins
Joe Gordon points at http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiSW1wb3J0RXJyb3I6IE5vIG1vZHVsZSBuYW1lZCBwYXNzbGliLmhhc2hcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiMTcyODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5MDI2MjQwNjMzOH0=

Re: [openstack-dev] Top Gate Reseting issues that need attention

2014-01-20 Thread Joe Gordon
On Mon, Jan 20, 2014 at 5:51 PM, Jay Pipes jaypi...@gmail.com wrote: On Mon, 2014-01-20 at 18:43 +0400, Eugene Nikanorov wrote: Hi Sean, I think the following 2 commits in neutron are essential for bringing neutron jobs back to acceptable level of failure rate:

Re: [openstack-dev] Top Gate Reseting issues that need attention

2014-01-20 Thread Jay Pipes
I wasn't saying it was frequent :) Just that it was happening on one of the patches that Eugene said needed to go through :) -jay On Mon, 2014-01-20 at 19:04 -0500, Joe Gordon wrote: On Mon, Jan 20, 2014 at 5:51 PM, Jay Pipes jaypi...@gmail.com wrote: On Mon, 2014-01-20 at 18:43