Re: [openstack-dev] [neutron] Gate-failure bugs spring cleaning

[Jakub Libosvar]

On 21/04/2018 14:16, Slawomir Kaplonski wrote:
> Hi Neutrinos,
> 
> There is time for some spring cleaning now so I went through list of Neutron 
> bugs with „gate-failure” tag https://tinyurl.com/y826rccx 
> I mark some of them as incomplete if there was not hits of same errors in 
> last 30 days. Please reopen them with proper comment if You think that it is 
> still valid bug or if You will spot similar error in some recent test runs.
> 
> About some of them I’m not sure if are still valid so please check it and 
> maybe update comment or close it if it’s already fixed somehow :)
> 
> Below detailed summary of bugs which I checked:
> 
> I removed Neutron from affected projects:
> * https://bugs.launchpad.net/tempest/+bug/1660612 
> 
> I marked as incomplete:
> * https://bugs.launchpad.net/neutron/+bug/1687027
> * https://bugs.launchpad.net/neutron/+bug/1693931
> * https://bugs.launchpad.net/neutron/+bug/1676966
> 
> Bugs which needs check of owner:
> * https://bugs.launchpad.net/neutron/+bug/1711463 - @Miguel, is it still 
> valid? Can we close it?
> * https://bugs.launchpad.net/neutron/+bug/1717302 - @Brian, no action since 
> 2017-12-12, is it failing still?
> 
> Bug which IMO should be reported against Cinder instead of Neutron, can 
> someone check and confirm that:
> * https://bugs.launchpad.net/neutron/+bug/1726462 - Is it related to Neutron 
> really, IMO it look like error with Cinder and it happens also in other than 
> neutron jobs, like „devstack-platform-opensuse-tumbleweed” and 
> „nova-multiattach” for example
> 
> Still valid bugs probably:
> * https://bugs.launchpad.net/neutron/+bug/1693950 - not exactly same error 
> but same tests failures I found recently so I think it is still valid to check
> * https://bugs.launchpad.net/neutron/+bug/1756301 - @Miguel: Can You check 
> and confirm that this is still valid
> * https://bugs.launchpad.net/neutron/+bug/1569621 - should be fixed by 
> https://review.openstack.org/#/c/562220/ - @Jakub can You confirm that?

That is correct - it's tracked as a fix for
https://bugs.launchpad.net/neutron/+bug/1708731
> 
> — 
> Best regards
> Slawek Kaplonski
> skapl...@redhat.com
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Gate failure

[Armando M.]

On 26 February 2016 at 11:43, Doug Wiegley 
wrote:

> This has merged, so you can resume your previously scheduled
> mad-merge-for-Mitaka and break it all again.
>

Assume this is the default behavior unless told otherwise. So, yes go nuts!

Cheers,
Armando


>
> Thanks,
> doug
>
> On Feb 25, 2016, at 2:46 PM, Armando M.  wrote:
>
> Folks,
>
> The API job recent breakage prevents us from merging code. Please refrain
> from pushing patches to the merge queue until [1] completes going through
> the pipeline.
>
> A.
>
> [1] https://review.openstack.org/#/c/284911/
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Gate failure

[Doug Wiegley]

This has merged, so you can resume your previously scheduled 
mad-merge-for-Mitaka and break it all again.

Thanks,
doug

> On Feb 25, 2016, at 2:46 PM, Armando M.  wrote:
> 
> Folks,
> 
> The API job recent breakage prevents us from merging code. Please refrain 
> from pushing patches to the merge queue until [1] completes going through the 
> pipeline.
> 
> A.
> 
> [1] https://review.openstack.org/#/c/284911/ 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Gate failure

[Ihar Hrachyshka]

UPD both fixes merged, should now be safe to recheck neutron patches.

Doug Wiegley  wrote:

This fix failed to merge, due to a new regression in how another job is  
using dib.  Here's a non voting patch for that, until it gets debugged:


https://review.openstack.org/267223

The fail stack is now two deep.

Doug


On Jan 13, 2016, at 11:41 AM, Armando M.  wrote:


It's the usual time of the week where I submit the dreaded email

Please do not push anything in the queue until change [1] merges.

Cheers,
Armando

[1] https://review.openstack.org/#/c/266885/
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Gate failure

[Doug Wiegley]

This fix failed to merge, due to a new regression in how another job is using 
dib.  Here's a non voting patch for that, until it gets debugged:

https://review.openstack.org/267223

The fail stack is now two deep.

Doug


> On Jan 13, 2016, at 11:41 AM, Armando M.  wrote:
> 
> It's the usual time of the week where I submit the dreaded email
> 
> Please do not push anything in the queue until change [1] merges.
> 
> Cheers,
> Armando
> 
> [1] https://review.openstack.org/#/c/266885/
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Gate failure

[Kevin Benton]

Ugh, this test is racey. I should have paid closer attention before I
+2'ed, my bad.

It just iterates external ports and uses an IP not in that list so if
another test grabbed the same IP, the floating IP creation request
specifying that IP will fail.

Revert proposed here: https://review.openstack.org/#/c/264573/

On Wed, Jan 6, 2016 at 9:00 PM, Clark Boylan  wrote:

> On Wed, Jan 6, 2016, at 08:48 PM, Henry Gessau wrote:
> > Armando M.  wrote:
> > > Hi folks,
> > >
> > > Due to [1], Neutron related jobs (api, and lbaas) are failing. Please
> hold
> > > your +A button until the issue is resolved.
> > >
> > > Thanks,
> > > Armando
> > >
> > > [1] https://review.openstack.org/#/c/256164/
> >
> > That fix has merged, but now we have a new issue [2] in the api job. :(
> >
> > [2] https://bugs.launchpad.net/neutron/+bug/1531706
> >
> It appears that that test class at least is allocating unused floating
> IPs in resource_setup(). At the very least you can probably stop
> overallocating to give yourself some breathing room. I do not think this
> explains how an entire /24 of IPs gets used up so there may be a leak
> elsewhere.
>
> Clark
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Kevin Benton
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Gate failure

[Clark Boylan]

On Wed, Jan 6, 2016, at 08:48 PM, Henry Gessau wrote:
> Armando M.  wrote:
> > Hi folks,
> > 
> > Due to [1], Neutron related jobs (api, and lbaas) are failing. Please hold
> > your +A button until the issue is resolved.
> > 
> > Thanks,
> > Armando 
> > 
> > [1] https://review.openstack.org/#/c/256164/
> 
> That fix has merged, but now we have a new issue [2] in the api job. :(
> 
> [2] https://bugs.launchpad.net/neutron/+bug/1531706
> 
It appears that that test class at least is allocating unused floating
IPs in resource_setup(). At the very least you can probably stop
overallocating to give yourself some breathing room. I do not think this
explains how an entire /24 of IPs gets used up so there may be a leak
elsewhere.

Clark

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Gate failure

[Kevin Benton]

Ah, that was a bit misleading. We're not leaking any floating IPs. The
create method for them adds them to a list that is cleaned up on teardown.

The issue was that the get_unused_ip method was iterating over v4 and v6
subnets. So if it happened to pick a v6 subnet, it would cause the test to
request a floating IP address with a v6 IP address, which isn't valid.

Fix is up here: https://review.openstack.org/264599

On Wed, Jan 6, 2016 at 10:03 PM, Kevin Benton  wrote:

> I spoke too soon. The test is racey and needs to be fixed, but that wasn't
> the cause of this failure. Floating IPs are being leaked as Clark
> suggested.  I'm looking into it now.
>
> On Wed, Jan 6, 2016 at 9:09 PM, Kevin Benton  wrote:
>
>> Ugh, this test is racey. I should have paid closer attention before I
>> +2'ed, my bad.
>>
>> It just iterates external ports and uses an IP not in that list so if
>> another test grabbed the same IP, the floating IP creation request
>> specifying that IP will fail.
>>
>> Revert proposed here: https://review.openstack.org/#/c/264573/
>>
>> On Wed, Jan 6, 2016 at 9:00 PM, Clark Boylan 
>> wrote:
>>
>>> On Wed, Jan 6, 2016, at 08:48 PM, Henry Gessau wrote:
>>> > Armando M.  wrote:
>>> > > Hi folks,
>>> > >
>>> > > Due to [1], Neutron related jobs (api, and lbaas) are failing.
>>> Please hold
>>> > > your +A button until the issue is resolved.
>>> > >
>>> > > Thanks,
>>> > > Armando
>>> > >
>>> > > [1] https://review.openstack.org/#/c/256164/
>>> >
>>> > That fix has merged, but now we have a new issue [2] in the api job. :(
>>> >
>>> > [2] https://bugs.launchpad.net/neutron/+bug/1531706
>>> >
>>> It appears that that test class at least is allocating unused floating
>>> IPs in resource_setup(). At the very least you can probably stop
>>> overallocating to give yourself some breathing room. I do not think this
>>> explains how an entire /24 of IPs gets used up so there may be a leak
>>> elsewhere.
>>>
>>> Clark
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>>
>> --
>> Kevin Benton
>>
>
>
>
> --
> Kevin Benton
>



-- 
Kevin Benton
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Gate failure with grenade

[Armando M.]

On 14 December 2015 at 10:18, Paul Michali  wrote:

> Thanks Sean!
>
> On Mon, Dec 14, 2015 at 12:58 PM Armando M.  wrote:
>
>> On 14 December 2015 at 09:51, Sean Dague  wrote:
>>
>>> On 12/14/2015 12:32 PM, Armando M. wrote:
>>> > Hi folks,
>>> >
>>> > Something snuck in past the gate last night [1]. Please stop rechecking
>>> > and pushing in the merge queue until the matter is resolved.
>>> >
>>> > I will follow up with details, if someone knows more, please find me
>>> on IRC.
>>> >
>>> > Thanks,
>>> > Armando
>>> >
>>> > [1]
>>> >
>>> http://logs.openstack.org/00/254900/4/gate/gate-grenade-dsvm-neutron/a9216c9/logs/grenade.sh.txt.gz#_2015-12-14_12_24_12_561
>>>
>>> https://review.openstack.org/#/c/257303/ is the fix, it's top of gate
>>> right now. Apparently it wasn't noticed that those were deprecated
>>> during the liberty cycle, and fixed accordingly.
>>>
>>
This has merged.


>
>> Thanks Sean!
>>
>> Cheers,
>> Armando
>>
>>
>>>
>>> -Sean
>>>
>>> --
>>> Sean Dague
>>> http://dague.net
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Gate failure with grenade

[Armando M.]

On 14 December 2015 at 09:51, Sean Dague  wrote:

> On 12/14/2015 12:32 PM, Armando M. wrote:
> > Hi folks,
> >
> > Something snuck in past the gate last night [1]. Please stop rechecking
> > and pushing in the merge queue until the matter is resolved.
> >
> > I will follow up with details, if someone knows more, please find me on
> IRC.
> >
> > Thanks,
> > Armando
> >
> > [1]
> >
> http://logs.openstack.org/00/254900/4/gate/gate-grenade-dsvm-neutron/a9216c9/logs/grenade.sh.txt.gz#_2015-12-14_12_24_12_561
>
> https://review.openstack.org/#/c/257303/ is the fix, it's top of gate
> right now. Apparently it wasn't noticed that those were deprecated
> during the liberty cycle, and fixed accordingly.
>

Thanks Sean!

Cheers,
Armando


>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Gate failure with grenade

[Sean Dague]

On 12/14/2015 12:32 PM, Armando M. wrote:
> Hi folks,
> 
> Something snuck in past the gate last night [1]. Please stop rechecking
> and pushing in the merge queue until the matter is resolved.
> 
> I will follow up with details, if someone knows more, please find me on IRC.
> 
> Thanks,
> Armando
> 
> [1]
> http://logs.openstack.org/00/254900/4/gate/gate-grenade-dsvm-neutron/a9216c9/logs/grenade.sh.txt.gz#_2015-12-14_12_24_12_561

https://review.openstack.org/#/c/257303/ is the fix, it's top of gate
right now. Apparently it wasn't noticed that those were deprecated
during the liberty cycle, and fixed accordingly.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Gate failure with grenade

[Paul Michali]

Thanks Sean!

On Mon, Dec 14, 2015 at 12:58 PM Armando M.  wrote:

> On 14 December 2015 at 09:51, Sean Dague  wrote:
>
>> On 12/14/2015 12:32 PM, Armando M. wrote:
>> > Hi folks,
>> >
>> > Something snuck in past the gate last night [1]. Please stop rechecking
>> > and pushing in the merge queue until the matter is resolved.
>> >
>> > I will follow up with details, if someone knows more, please find me on
>> IRC.
>> >
>> > Thanks,
>> > Armando
>> >
>> > [1]
>> >
>> http://logs.openstack.org/00/254900/4/gate/gate-grenade-dsvm-neutron/a9216c9/logs/grenade.sh.txt.gz#_2015-12-14_12_24_12_561
>>
>> https://review.openstack.org/#/c/257303/ is the fix, it's top of gate
>> right now. Apparently it wasn't noticed that those were deprecated
>> during the liberty cycle, and fixed accordingly.
>>
>
> Thanks Sean!
>
> Cheers,
> Armando
>
>
>>
>> -Sean
>>
>> --
>> Sean Dague
>> http://dague.net
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev