Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

2017-08-29 Thread Jakub Libosvar 
On 28/08/2017 21:20, Jeremy Stanley wrote:
> On 2017-08-11 15:31:58 +0200 (+0200), Jakub Libosvar wrote:
> [...]
>> My best hope is they are gonna backport the fix to 4.4.0 and tag a
>> new kernel so we can start running the tests again.
> 
> Just to follow up on this thread, it looks like the Ubuntu kernel
> team have had the upstream fix patched into xenial-proposed for a
> couple weeks already, awaiting confirmation from someone with the
> ability to reproduce the issue. Please see
> https://launchpad.net/bugs/1709032 if you can help close it out.

Thanks Jeremy for reminder. I replied on the LP bug that they can close
it. We enabled our functional job back a while ago and it works just fine.

PS. The new kernel also solved other issue we were facing, so we killed
two birds with one stone.

Jakub

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

2017-08-28 Thread Jeremy Stanley 
On 2017-08-11 15:31:58 +0200 (+0200), Jakub Libosvar wrote:
[...]
> My best hope is they are gonna backport the fix to 4.4.0 and tag a
> new kernel so we can start running the tests again.

Just to follow up on this thread, it looks like the Ubuntu kernel
team have had the upstream fix patched into xenial-proposed for a
couple weeks already, awaiting confirmation from someone with the
ability to reproduce the issue. Please see
https://launchpad.net/bugs/1709032 if you can help close it out.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

2017-08-11 Thread Jakub Libosvar 
On 10/08/2017 21:16, Jeremy Stanley wrote:
> On 2017-08-10 17:13:58 + (+), Mooney, Sean K wrote:
> [...]
>> so on that it would bre quite trivial to have disk image builder
>> install The linux-image-virtual-hwe-16.04
>> linux-image-virtual-hwe-16.04-edge to pull in a 4.10 or 4.11
>> kernel respctivly if the default 4.4 is broken. We just need a new
>> dib element to install the package And modify the  nodepool config
>> to include it when it rebuildes the image every night.
>> Alternitivly You can pull a vanilla kernel form
>> http://kernel.ubuntu.com/~kernel-ppa/mainline/ Follow the process
>> documented here https://wiki.ubuntu.com/Kernel/MainlineBuilds If
>> you want to maintain testing with 4.4.x
> [...]
> 
> Sure, your definition of "quite trivial" just differs a lot from
> mine. Given that the bug in question seems to have no further impact
> to the pace of development with the discussed test temporarily
> disabled, that strikes me as a lot more maintainable in the long run
> for a problem which started getting urgent attention from the distro
> as soon as it was reported to them and will likely be resolved over
> the course of the next few days at which point we'll automatically
> update to the fixed version and you can try reenabling that test
> again.

Yeah, I totally agree with Jeremy. Ubuntu folks are very active :)
Thanks for that. My best hope is they are gonna backport the fix to
4.4.0 and tag a new kernel so we can start running the tests again.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

2017-08-10 Thread Jeremy Stanley 
On 2017-08-10 17:13:58 + (+), Mooney, Sean K wrote:
[...]
> so on that it would bre quite trivial to have disk image builder
> install The linux-image-virtual-hwe-16.04
> linux-image-virtual-hwe-16.04-edge to pull in a 4.10 or 4.11
> kernel respctivly if the default 4.4 is broken. We just need a new
> dib element to install the package And modify the  nodepool config
> to include it when it rebuildes the image every night.
> Alternitivly You can pull a vanilla kernel form
> http://kernel.ubuntu.com/~kernel-ppa/mainline/ Follow the process
> documented here https://wiki.ubuntu.com/Kernel/MainlineBuilds If
> you want to maintain testing with 4.4.x
[...]

Sure, your definition of "quite trivial" just differs a lot from
mine. Given that the bug in question seems to have no further impact
to the pace of development with the discussed test temporarily
disabled, that strikes me as a lot more maintainable in the long run
for a problem which started getting urgent attention from the distro
as soon as it was reported to them and will likely be resolved over
the course of the next few days at which point we'll automatically
update to the fixed version and you can try reenabling that test
again.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

2017-08-10 Thread Mooney, Sean K 

From: Miguel Angel Ajo Pelayo [mailto:majop...@redhat.com]
Sent: Thursday, August 10, 2017 8:55 AM
To: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [neutron][infra] Functional job failure rate at 
100%

Good (amazing) job folks. :)

El 10 ago. 2017 9:43, "Thierry Carrez" 
<thie...@openstack.org<mailto:thie...@openstack.org>> escribió:
Oh, that's good for us. Should still be fixed, if only so that we can
test properly :)

Kevin Benton wrote:
> This is just the code simulating the conntrack entries that would be
> created by real traffic in a production system, right?
>
> On Wed, Aug 9, 2017 at 11:46 AM, Jakub Libosvar 
> <jlibo...@redhat.com<mailto:jlibo...@redhat.com>
> <mailto:jlibo...@redhat.com<mailto:jlibo...@redhat.com>>> wrote:
>
> On 09/08/2017 18:23, Jeremy Stanley wrote:
> > On 2017-08-09 15:29:04 +0200 (+0200), Jakub Libosvar wrote:
> > [...]
> >> Is it possible to switch used image for jenkins machines to use
> >> back the older version? Any other ideas how to deal with the
> >> kernel bug?
> >
> > Making our images use non-current kernel packages isn't trivial, but
[Mooney, Sean K]  so on that it would bre quite trivial to have disk image 
builder install
The linux-image-virtual-hwe-16.04 linux-image-virtual-hwe-16.04-edge to pull in 
a 4.10 or
4.11 kernel respctivly if the default 4.4 is broken. We just need a new dib 
element to install the package
And modify the  nodepool config to include it when it rebuildes the image every 
night. Alternitivly
You can pull a vanilla kernel form 
http://kernel.ubuntu.com/~kernel-ppa/mainline/
Follow the process documented here https://wiki.ubuntu.com/Kernel/MainlineBuilds
If you want to maintain testing with 4.4.x

> > as Thierry points out in his reply this is not just a problem for
> > our CI system. Basically Ubuntu has broken OpenStack (and probably a
> > variety of other uses of conntrack) for a lot of people following
> > kernel updates in 16.04 LTS so the fix needs to happen there
> > regardless. Right now, basically, Ubuntu Xenial is not a good
> > platform to be running OpenStack on until they get the kernel
> > regression addressed.
>
> True. Fortunately, the impact is not that catastrophic for Neutron as it
> might seem on the first look. Not sure about the other projects, though.
> Neutron doesn't create conntrack entries in production code - only in
> testing. That said, agents should work just fine even with the
> kernel bug.

--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

2017-08-10 Thread Miguel Angel Ajo Pelayo 
Good (amazing) job folks. :)

El 10 ago. 2017 9:43, "Thierry Carrez"  escribió:

> Oh, that's good for us. Should still be fixed, if only so that we can
> test properly :)
>
> Kevin Benton wrote:
> > This is just the code simulating the conntrack entries that would be
> > created by real traffic in a production system, right?
> >
> > On Wed, Aug 9, 2017 at 11:46 AM, Jakub Libosvar  > > wrote:
> >
> > On 09/08/2017 18:23, Jeremy Stanley wrote:
> > > On 2017-08-09 15:29:04 +0200 (+0200), Jakub Libosvar wrote:
> > > [...]
> > >> Is it possible to switch used image for jenkins machines to use
> > >> back the older version? Any other ideas how to deal with the
> > >> kernel bug?
> > >
> > > Making our images use non-current kernel packages isn't trivial,
> but
> > > as Thierry points out in his reply this is not just a problem for
> > > our CI system. Basically Ubuntu has broken OpenStack (and probably
> a
> > > variety of other uses of conntrack) for a lot of people following
> > > kernel updates in 16.04 LTS so the fix needs to happen there
> > > regardless. Right now, basically, Ubuntu Xenial is not a good
> > > platform to be running OpenStack on until they get the kernel
> > > regression addressed.
> >
> > True. Fortunately, the impact is not that catastrophic for Neutron
> as it
> > might seem on the first look. Not sure about the other projects,
> though.
> > Neutron doesn't create conntrack entries in production code - only in
> > testing. That said, agents should work just fine even with the
> > kernel bug.
>
> --
> Thierry Carrez (ttx)
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

2017-08-10 Thread Thierry Carrez 
Oh, that's good for us. Should still be fixed, if only so that we can
test properly :)

Kevin Benton wrote:
> This is just the code simulating the conntrack entries that would be
> created by real traffic in a production system, right?
> 
> On Wed, Aug 9, 2017 at 11:46 AM, Jakub Libosvar  > wrote:
> 
> On 09/08/2017 18:23, Jeremy Stanley wrote:
> > On 2017-08-09 15:29:04 +0200 (+0200), Jakub Libosvar wrote:
> > [...]
> >> Is it possible to switch used image for jenkins machines to use
> >> back the older version? Any other ideas how to deal with the
> >> kernel bug?
> >
> > Making our images use non-current kernel packages isn't trivial, but
> > as Thierry points out in his reply this is not just a problem for
> > our CI system. Basically Ubuntu has broken OpenStack (and probably a
> > variety of other uses of conntrack) for a lot of people following
> > kernel updates in 16.04 LTS so the fix needs to happen there
> > regardless. Right now, basically, Ubuntu Xenial is not a good
> > platform to be running OpenStack on until they get the kernel
> > regression addressed.
> 
> True. Fortunately, the impact is not that catastrophic for Neutron as it
> might seem on the first look. Not sure about the other projects, though.
> Neutron doesn't create conntrack entries in production code - only in
> testing. That said, agents should work just fine even with the
> kernel bug.

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

2017-08-09 Thread Kevin Benton 
This is just the code simulating the conntrack entries that would be
created by real traffic in a production system, right?

On Wed, Aug 9, 2017 at 11:46 AM, Jakub Libosvar  wrote:

> On 09/08/2017 18:23, Jeremy Stanley wrote:
> > On 2017-08-09 15:29:04 +0200 (+0200), Jakub Libosvar wrote:
> > [...]
> >> Is it possible to switch used image for jenkins machines to use
> >> back the older version? Any other ideas how to deal with the
> >> kernel bug?
> >
> > Making our images use non-current kernel packages isn't trivial, but
> > as Thierry points out in his reply this is not just a problem for
> > our CI system. Basically Ubuntu has broken OpenStack (and probably a
> > variety of other uses of conntrack) for a lot of people following
> > kernel updates in 16.04 LTS so the fix needs to happen there
> > regardless. Right now, basically, Ubuntu Xenial is not a good
> > platform to be running OpenStack on until they get the kernel
> > regression addressed.
>
> True. Fortunately, the impact is not that catastrophic for Neutron as it
> might seem on the first look. Not sure about the other projects, though.
> Neutron doesn't create conntrack entries in production code - only in
> testing. That said, agents should work just fine even with the kernel bug.
>
> >
> >
> >
> > 
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
> unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

2017-08-09 Thread Jakub Libosvar 
On 09/08/2017 18:23, Jeremy Stanley wrote:
> On 2017-08-09 15:29:04 +0200 (+0200), Jakub Libosvar wrote:
> [...]
>> Is it possible to switch used image for jenkins machines to use
>> back the older version? Any other ideas how to deal with the
>> kernel bug?
> 
> Making our images use non-current kernel packages isn't trivial, but
> as Thierry points out in his reply this is not just a problem for
> our CI system. Basically Ubuntu has broken OpenStack (and probably a
> variety of other uses of conntrack) for a lot of people following
> kernel updates in 16.04 LTS so the fix needs to happen there
> regardless. Right now, basically, Ubuntu Xenial is not a good
> platform to be running OpenStack on until they get the kernel
> regression addressed.

True. Fortunately, the impact is not that catastrophic for Neutron as it
might seem on the first look. Not sure about the other projects, though.
Neutron doesn't create conntrack entries in production code - only in
testing. That said, agents should work just fine even with the kernel bug.

> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

2017-08-09 Thread Jeremy Stanley 
On 2017-08-09 15:29:04 +0200 (+0200), Jakub Libosvar wrote:
[...]
> Is it possible to switch used image for jenkins machines to use
> back the older version? Any other ideas how to deal with the
> kernel bug?

Making our images use non-current kernel packages isn't trivial, but
as Thierry points out in his reply this is not just a problem for
our CI system. Basically Ubuntu has broken OpenStack (and probably a
variety of other uses of conntrack) for a lot of people following
kernel updates in 16.04 LTS so the fix needs to happen there
regardless. Right now, basically, Ubuntu Xenial is not a good
platform to be running OpenStack on until they get the kernel
regression addressed.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

2017-08-09 Thread Thierry Carrez 
Thanks for this nice detective work, Jakub and Daniel ! This disabled
test job was making me nervous wrt. Pike release.

Now I suspect that this kernel regression is affecting Ubuntu Xenial
users for previous releases of OpenStack, too, so I hope this will get
fixed in Ubuntu soon enough.

Daniel Alvarez Sanchez wrote:
> Some more info added to Jakub's excellent report :)
> 
> 
> New kernel Ubuntu-4.4.0-89.112HEADUbuntu-4.4.0-89.112master was
> tagged 9 days ago (07/31/2017) [0].
> 
> From a quick look, the only commit around this function is [1].
> 
> [0] 
> http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/commit/?id=64de31ed97a03ec1b86fd4f76e445506dce55b02
> [1] 
> http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/commit/?id=2ad4caea651e1cc0fc86111ece9f9d74de825b78
> 
> On Wed, Aug 9, 2017 at 3:29 PM, Jakub Libosvar  > wrote:
> 
> Daniel Alvarez and I spent some time looking at it and the culprit was
> finally found.
> 
> tl;dr
> 
> We updated a kernel on machines to one containing bug when creating
> conntrack entries which makes functional tests stuck. More info at [4].
> 
> For now, I sent a patch [5] to disable for now jobs that create
> conntrack entries manually, it needs update of commit message. Once it
> merges, we an enable back functional job to voting to avoid regressions.
> 
> Is it possible to switch used image for jenkins machines to use back the
> older version? Any other ideas how to deal with the kernel bug?
> 
> Thanks
> Jakub
> 
> [5] https://review.openstack.org/#/c/492068/1
> 
> 
> On 07/08/2017 11:52, Jakub Libosvar wrote:
> > Hi all,
> >
> > as per grafana [1] the functional job is broken. Looking at
> logstash [2]
> > it started happening consistently since 2017-08-03 16:27. I didn't
> find
> > any particular patch in Neutron that could cause it.
> >
> > The culprit is that ovsdb starts misbehaving [3] and then we retry
> calls
> > indefinitely. We still use 2.5.2 openvswitch as we had before. I
> opened
> > a bug [4] and started investigation, I'll update my findings there.
> >
> > I think at this point there is no reason to run "recheck" on your
> patches.
> >
> > Thanks,
> > Jakub
> >
> > [1]
> >
> 
> http://grafana.openstack.org/dashboard/db/neutron-failure-rate?panelId=7
> 
> 
> > [2] http://bit.ly/2vdKMwy
> > [3]
> >
> 
> http://logs.openstack.org/14/488914/8/check/gate-neutron-dsvm-functional-ubuntu-xenial/75d7482/logs/openvswitch/ovsdb-server.txt.gz
> 
> 
> > [4] https://bugs.launchpad.net/neutron/+bug/1709032
> 
> >
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

2017-08-09 Thread Daniel Alvarez Sanchez 
Some more info added to Jakub's excellent report :)


New kernel Ubuntu-4.4.0-89.112HEADUbuntu-4.4.0-89.112master was
tagged 9 days ago (07/31/2017) [0].

>From a quick look, the only commit around this function is [1].

[0]
http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/commit/?id=64de31ed97a03ec1b86fd4f76e445506dce55b02
[1]
http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/commit/?id=2ad4caea651e1cc0fc86111ece9f9d74de825b78

On Wed, Aug 9, 2017 at 3:29 PM, Jakub Libosvar  wrote:

> Daniel Alvarez and I spent some time looking at it and the culprit was
> finally found.
>
> tl;dr
>
> We updated a kernel on machines to one containing bug when creating
> conntrack entries which makes functional tests stuck. More info at [4].
>
> For now, I sent a patch [5] to disable for now jobs that create
> conntrack entries manually, it needs update of commit message. Once it
> merges, we an enable back functional job to voting to avoid regressions.
>
> Is it possible to switch used image for jenkins machines to use back the
> older version? Any other ideas how to deal with the kernel bug?
>
> Thanks
> Jakub
>
> [5] https://review.openstack.org/#/c/492068/1
>
> On 07/08/2017 11:52, Jakub Libosvar wrote:
> > Hi all,
> >
> > as per grafana [1] the functional job is broken. Looking at logstash [2]
> > it started happening consistently since 2017-08-03 16:27. I didn't find
> > any particular patch in Neutron that could cause it.
> >
> > The culprit is that ovsdb starts misbehaving [3] and then we retry calls
> > indefinitely. We still use 2.5.2 openvswitch as we had before. I opened
> > a bug [4] and started investigation, I'll update my findings there.
> >
> > I think at this point there is no reason to run "recheck" on your
> patches.
> >
> > Thanks,
> > Jakub
> >
> > [1]
> > http://grafana.openstack.org/dashboard/db/neutron-failure-
> rate?panelId=7
> > [2] http://bit.ly/2vdKMwy
> > [3]
> > http://logs.openstack.org/14/488914/8/check/gate-neutron-
> dsvm-functional-ubuntu-xenial/75d7482/logs/openvswitch/ovsdb-server.txt.gz
> > [4] https://bugs.launchpad.net/neutron/+bug/1709032
> >
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][infra] Functional job failure rate at 100%

2017-08-09 Thread Jakub Libosvar 
Daniel Alvarez and I spent some time looking at it and the culprit was
finally found.

tl;dr

We updated a kernel on machines to one containing bug when creating
conntrack entries which makes functional tests stuck. More info at [4].

For now, I sent a patch [5] to disable for now jobs that create
conntrack entries manually, it needs update of commit message. Once it
merges, we an enable back functional job to voting to avoid regressions.

Is it possible to switch used image for jenkins machines to use back the
older version? Any other ideas how to deal with the kernel bug?

Thanks
Jakub

[5] https://review.openstack.org/#/c/492068/1

On 07/08/2017 11:52, Jakub Libosvar wrote:
> Hi all,
> 
> as per grafana [1] the functional job is broken. Looking at logstash [2]
> it started happening consistently since 2017-08-03 16:27. I didn't find
> any particular patch in Neutron that could cause it.
> 
> The culprit is that ovsdb starts misbehaving [3] and then we retry calls
> indefinitely. We still use 2.5.2 openvswitch as we had before. I opened
> a bug [4] and started investigation, I'll update my findings there.
> 
> I think at this point there is no reason to run "recheck" on your patches.
> 
> Thanks,
> Jakub
> 
> [1]
> http://grafana.openstack.org/dashboard/db/neutron-failure-rate?panelId=7
> [2] http://bit.ly/2vdKMwy
> [3]
> http://logs.openstack.org/14/488914/8/check/gate-neutron-dsvm-functional-ubuntu-xenial/75d7482/logs/openvswitch/ovsdb-server.txt.gz
> [4] https://bugs.launchpad.net/neutron/+bug/1709032
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev