Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

On 28/08/2017 21:20, Jeremy Stanley wrote: > On 2017-08-11 15:31:58 +0200 (+0200), Jakub Libosvar wrote: > [...] >> My best hope is they are gonna backport the fix to 4.4.0 and tag a >> new kernel so we can start running the tests again. > > Just to follow up on this thread, it looks like the

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

On 2017-08-11 15:31:58 +0200 (+0200), Jakub Libosvar wrote: [...] > My best hope is they are gonna backport the fix to 4.4.0 and tag a > new kernel so we can start running the tests again. Just to follow up on this thread, it looks like the Ubuntu kernel team have had the upstream fix patched

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

On 10/08/2017 21:16, Jeremy Stanley wrote: > On 2017-08-10 17:13:58 + (+), Mooney, Sean K wrote: > [...] >> so on that it would bre quite trivial to have disk image builder >> install The linux-image-virtual-hwe-16.04 >> linux-image-virtual-hwe-16.04-edge to pull in a 4.10 or 4.11 >>

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

On 2017-08-10 17:13:58 + (+), Mooney, Sean K wrote: [...] > so on that it would bre quite trivial to have disk image builder > install The linux-image-virtual-hwe-16.04 > linux-image-virtual-hwe-16.04-edge to pull in a 4.10 or 4.11 > kernel respctivly if the default 4.4 is broken. We just

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

From: Miguel Angel Ajo Pelayo [mailto:majop...@redhat.com] Sent: Thursday, August 10, 2017 8:55 AM To: OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org> Subject: Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

Good (amazing) job folks. :) El 10 ago. 2017 9:43, "Thierry Carrez" escribió: > Oh, that's good for us. Should still be fixed, if only so that we can > test properly :) > > Kevin Benton wrote: > > This is just the code simulating the conntrack entries that would be > >

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

Oh, that's good for us. Should still be fixed, if only so that we can test properly :) Kevin Benton wrote: > This is just the code simulating the conntrack entries that would be > created by real traffic in a production system, right? > > On Wed, Aug 9, 2017 at 11:46 AM, Jakub Libosvar

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

This is just the code simulating the conntrack entries that would be created by real traffic in a production system, right? On Wed, Aug 9, 2017 at 11:46 AM, Jakub Libosvar wrote: > On 09/08/2017 18:23, Jeremy Stanley wrote: > > On 2017-08-09 15:29:04 +0200 (+0200), Jakub

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

On 09/08/2017 18:23, Jeremy Stanley wrote: > On 2017-08-09 15:29:04 +0200 (+0200), Jakub Libosvar wrote: > [...] >> Is it possible to switch used image for jenkins machines to use >> back the older version? Any other ideas how to deal with the >> kernel bug? > > Making our images use non-current

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

On 2017-08-09 15:29:04 +0200 (+0200), Jakub Libosvar wrote: [...] > Is it possible to switch used image for jenkins machines to use > back the older version? Any other ideas how to deal with the > kernel bug? Making our images use non-current kernel packages isn't trivial, but as Thierry points

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

Thanks for this nice detective work, Jakub and Daniel ! This disabled test job was making me nervous wrt. Pike release. Now I suspect that this kernel regression is affecting Ubuntu Xenial users for previous releases of OpenStack, too, so I hope this will get fixed in Ubuntu soon enough. Daniel

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

Some more info added to Jakub's excellent report :) New kernel Ubuntu-4.4.0-89.112HEADUbuntu-4.4.0-89.112master was tagged 9 days ago (07/31/2017) [0]. >From a quick look, the only commit around this function is [1]. [0]

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

Daniel Alvarez and I spent some time looking at it and the culprit was finally found. tl;dr We updated a kernel on machines to one containing bug when creating conntrack entries which makes functional tests stuck. More info at [4]. For now, I sent a patch [5] to disable for now jobs that create