[openstack-dev] [neutron] 40% failure on neutron python3.4 tests in the gate

2015-08-28 Thread Sean Dague
We're at a 18hr backup in the gate, which is really unusual given the amount of decoupling. Even under our current load that means we're seeing huge failure rates causing resets. It appears one of the major culprits is the python34 tests in neutron, which were over a 40% failure rate recently -

Re: [openstack-dev] [neutron] 40% failure on neutron python3.4 tests in the gate

2015-08-28 Thread Neil Jerram
On 28/08/15 13:39, Kevin Benton wrote: For the py34 failures, they seem to have started around the same time as a change was merged that adjusted the way they were ran so I proposed a revert for that patch here: https://review.openstack.org/218244 Which leads on to

Re: [openstack-dev] [neutron] 40% failure on neutron python3.4 tests in the gate

2015-08-28 Thread Sean Dague
On 08/28/2015 09:22 AM, Assaf Muller wrote: On Fri, Aug 28, 2015 at 9:12 AM, Neil Jerram neil.jer...@metaswitch.com mailto:neil.jer...@metaswitch.com wrote: On 28/08/15 13:39, Kevin Benton wrote: For the py34 failures, they seem to have started around the same time as a

Re: [openstack-dev] [neutron] 40% failure on neutron python3.4 tests in the gate

2015-08-28 Thread Sean Dague
On 08/28/2015 08:34 AM, Kevin Benton wrote: One of the patches that fixes one of the functional failures that has been hitting is here: https://review.openstack.org/#/c/217927/ However, it failed in the DVR job on the 'test_router_rescheduling' test.[1] This failure is because the logic to

Re: [openstack-dev] [neutron] 40% failure on neutron python3.4 tests in the gate

2015-08-28 Thread Kevin Benton
Why would that only impact py34 and not py27? Aren't the py27 run with testtools? On Fri, Aug 28, 2015 at 5:41 AM, Sean Dague s...@dague.net wrote: On 08/28/2015 08:34 AM, Kevin Benton wrote: One of the patches that fixes one of the functional failures that has been hitting is here:

Re: [openstack-dev] [neutron] 40% failure on neutron python3.4 tests in the gate

2015-08-28 Thread Sean Dague
On 08/28/2015 08:50 AM, Kevin Benton wrote: Why would that only impact py34 and not py27? Aren't the py27 run with testtools? py34 is only running some subset of tests, so there are a lot of ways this can go weird. It may be that the db tests that are failing assume some other tests which have

Re: [openstack-dev] [neutron] 40% failure on neutron python3.4 tests in the gate

2015-08-28 Thread Kevin Benton
One of the patches that fixes one of the functional failures that has been hitting is here: https://review.openstack.org/#/c/217927/ However, it failed in the DVR job on the 'test_router_rescheduling' test.[1] This failure is because the logic to skip when DVR is enabled is based on a check that

Re: [openstack-dev] [neutron] 40% failure on neutron python3.4 tests in the gate

2015-08-28 Thread Assaf Muller
On Fri, Aug 28, 2015 at 9:12 AM, Neil Jerram neil.jer...@metaswitch.com wrote: On 28/08/15 13:39, Kevin Benton wrote: For the py34 failures, they seem to have started around the same time as a change was merged that adjusted the way they were ran so I proposed a revert for that patch

Re: [openstack-dev] [neutron] 40% failure on neutron python3.4 tests in the gate

2015-08-28 Thread Assaf Muller
To recap, we had three issues impacting the gate queue: 1) The neutron functional job has had a high failure rate for a while now. Since it's impacting the gate, I've removed it from the gate queue but kept it in the Neutron check queue: https://review.openstack.org/#/c/218302/ If you'd like to

Re: [openstack-dev] [neutron] 40% failure on neutron python3.4 tests in the gate

2015-08-28 Thread Sean Dague
On 08/28/2015 11:20 AM, Assaf Muller wrote: To recap, we had three issues impacting the gate queue: 1) The neutron functional job has had a high failure rate for a while now. Since it's impacting the gate, I've removed it from the gate queue but kept it in the Neutron check queue:

Re: [openstack-dev] [neutron] 40% failure on neutron python3.4 tests in the gate

2015-08-28 Thread Salvatore Orlando
On 28 August 2015 at 16:57, Sean Dague s...@dague.net wrote: On 08/28/2015 11:20 AM, Assaf Muller wrote: To recap, we had three issues impacting the gate queue: 1) The neutron functional job has had a high failure rate for a while now. Since it's impacting the gate, I've removed it

Re: [openstack-dev] [neutron] 40% failure on neutron python3.4 tests in the gate

2015-08-28 Thread Assaf Muller
On Fri, Aug 28, 2015 at 1:50 PM, Salvatore Orlando salv.orla...@gmail.com wrote: On 28 August 2015 at 16:57, Sean Dague s...@dague.net wrote: On 08/28/2015 11:20 AM, Assaf Muller wrote: To recap, we had three issues impacting the gate queue: 1) The neutron functional job has had a high