subject:"\[openstack\-dev\] gate jobs \- papercuts"

Re: [openstack-dev] gate jobs - papercuts

2017-02-02 Thread Jordan Pittier

On Wed, Feb 1, 2017 at 6:57 PM, Davanum Srinivas  wrote:
> Thanks Melanie! Since the last report 4 hours ago, the
> ServersNegativeTestJSON failed 8 more times.
>
> Is the following one of the libvirt ones?
>
> http://logs.openstack.org/24/425924/2/gate/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/f1b9229/logs/testr_results.html.gz
> tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2
No, it's something else. Probably a random network glitch: failure is
due to "13:19:01,294 12963 ERROR [paramiko.transport] Socket
exception: Connection reset by peer (104)" while executing a SSH
command on a fresh test VM.
>
> Thanks,
> Dims
>
>
> On Wed, Feb 1, 2017 at 12:29 PM, melanie witt  wrote:
>> On Wed, 1 Feb 2017 08:06:41 -0500, Davanum Srinivas wrote:
>>>
>>> Three more from this morning, at least the first one looks new:
>>>
>>>
>>> http://logs.openstack.org/04/426604/3/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/708771d/logs/testr_results.html.gz
>>> tempest.api.compute.images.test_list_image_filters
>>>
>>>
>>> http://logs.openstack.org/99/420299/3/gate/gate-tempest-dsvm-neutron-dvr-ubuntu-xenial/6e8d208/logs/testr_results.html.gz
>>>
>>> http://logs.openstack.org/23/427223/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/18e635a/logs/testr_results.html.gz
>>> tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON
>>
>>
>> The last two are https://launchpad.net/bugs/1660878 being worked on here
>> https://review.openstack.org/#/c/427775
>>
>> -melanie
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Davanum Srinivas :: https://twitter.com/dims
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 
 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] gate jobs - papercuts

2017-02-01 Thread Davanum Srinivas

Thanks Melanie! Since the last report 4 hours ago, the
ServersNegativeTestJSON failed 8 more times.

Is the following one of the libvirt ones?

http://logs.openstack.org/24/425924/2/gate/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/f1b9229/logs/testr_results.html.gz
tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2

Thanks,
Dims


On Wed, Feb 1, 2017 at 12:29 PM, melanie witt  wrote:
> On Wed, 1 Feb 2017 08:06:41 -0500, Davanum Srinivas wrote:
>>
>> Three more from this morning, at least the first one looks new:
>>
>>
>> http://logs.openstack.org/04/426604/3/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/708771d/logs/testr_results.html.gz
>> tempest.api.compute.images.test_list_image_filters
>>
>>
>> http://logs.openstack.org/99/420299/3/gate/gate-tempest-dsvm-neutron-dvr-ubuntu-xenial/6e8d208/logs/testr_results.html.gz
>>
>> http://logs.openstack.org/23/427223/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/18e635a/logs/testr_results.html.gz
>> tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON
>
>
> The last two are https://launchpad.net/bugs/1660878 being worked on here
> https://review.openstack.org/#/c/427775
>
> -melanie
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] gate jobs - papercuts

2017-02-01 Thread melanie witt


On Wed, 1 Feb 2017 08:06:41 -0500, Davanum Srinivas wrote:

Three more from this morning, at least the first one looks new:

http://logs.openstack.org/04/426604/3/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/708771d/logs/testr_results.html.gz
tempest.api.compute.images.test_list_image_filters

http://logs.openstack.org/99/420299/3/gate/gate-tempest-dsvm-neutron-dvr-ubuntu-xenial/6e8d208/logs/testr_results.html.gz
http://logs.openstack.org/23/427223/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/18e635a/logs/testr_results.html.gz
tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON


The last two are https://launchpad.net/bugs/1660878 being worked on here 
https://review.openstack.org/#/c/427775


-melanie

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] gate jobs - papercuts

2017-02-01 Thread Andrea Frittoli

I actually ended-up proposing an update to the other bug, as the signature
was much closed [0]

andrea

[0] https://bugs.launchpad.net/nova/+bug/1643911
[1] https://review.openstack.org/427754

On Wed, Feb 1, 2017 at 1:50 PM Andrea Frittoli 
wrote:

> The first one is yet another flavour of libvirtd crash [0]:
>
> >> *** Error in `/usr/sbin/libvirtd': realloc(): invalid next size:
> 0x56108d7a7450 ***
>
> which adds to the list of three libvirt crash signatures listed by Jordan
> in [1].
> I guess this new (?) signature could be incorporated in bug [2] - I can
> update the E-R query for that.
>
> andrea
>
> [0]
> http://logs.openstack.org/04/426604/3/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/708771d/logs/syslog.txt.gz#_Feb_01_11_47_53
> [1]
> http://lists.openstack.org/pipermail/openstack-dev/2017-February/111347.html
>
> [2] https://bugs.launchpad.net/tempest/+bug/1646779
>
>
> On Wed, Feb 1, 2017 at 1:09 PM Davanum Srinivas  wrote:
>
> MattR,
>
> Three more from this morning, at least the first one looks new:
>
>
> http://logs.openstack.org/04/426604/3/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/708771d/logs/testr_results.html.gz
> tempest.api.compute.images.test_list_image_filters
>
>
> http://logs.openstack.org/99/420299/3/gate/gate-tempest-dsvm-neutron-dvr-ubuntu-xenial/6e8d208/logs/testr_results.html.gz
>
> http://logs.openstack.org/23/427223/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/18e635a/logs/testr_results.html.gz
> tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON
>
> Thanks,
> Dims
>
>
> On Tue, Jan 31, 2017 at 10:23 PM, Matt Riedemann 
> wrote:
> > On 1/31/2017 11:49 AM, Davanum Srinivas wrote:
> >>
> >> Folks,
> >>
> >> Here's the list of job failures that failed in the gate queue.
> >> captured with my script[1][2] since around 10:00 AM today. All jobs
> >> failed with just one bad test.
> >>
> >>
> >>
> http://logs.openstack.org/48/426448/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/ecb3d0a/
> >>-
> >>
> tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON
> >>
> >>
> http://logs.openstack.org/48/426448/2/gate/gate-tempest-dsvm-neutron-full-ssh/71f6c8c/
> >>  -
> tempest.api.compute.admin.test_servers.ServersAdminTestJSON
> >>
> >>
> http://logs.openstack.org/48/376548/8/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/cf3028b/
> >>-
> tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON
> >>
> >>
> http://logs.openstack.org/68/417668/8/gate/gate-tempest-dsvm-neutron-full-ssh/27bda02/
> >>  -
> >>
> tempest.api.compute.volumes.test_attach_volume.AttachVolumeShelveTestJSON
> >>
> >>
> http://logs.openstack.org/48/423548/11/gate/gate-keystone-python27-db-ubuntu-xenial/a1f55ca/
> >>- keystone.tests.unit.test_v3_auth.TestMFARules
> >>
> >>
> http://logs.openstack.org/61/424961/1/gate/gate-tempest-dsvm-cells-ubuntu-xenial/8a1f9e7/
> >>   - tempest.api.compute.admin.test_servers.ServersAdminTestJSON
> >>
> >>
> http://logs.openstack.org/23/426823/3/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0204168/
> >>-
> >>
> tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps
> >>
> >> So our gate is now 36 deep with stuff running for little more than 4
> >> hours repeatedly Can folks look deeper please?
> >>
> >> Thanks,
> >> Dims
> >>
> >> [1] https://gist.github.com/dims/54b391bd5964d3d208113b16766ea85e
> >> [2] http://paste.openstack.org/show/597071/
> >>
> >
> > I've identified a regression in nova here:
> >
> > https://bugs.launchpad.net/nova/+bug/1660878
> >
> > --
> >
> > Thanks,
> >
> > Matt Riedemann
> >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Davanum Srinivas :: https://twitter.com/dims
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] gate jobs - papercuts

2017-02-01 Thread Andrea Frittoli

The first one is yet another flavour of libvirtd crash [0]:

>> *** Error in `/usr/sbin/libvirtd': realloc(): invalid next size:
0x56108d7a7450 ***

which adds to the list of three libvirt crash signatures listed by Jordan
in [1].
I guess this new (?) signature could be incorporated in bug [2] - I can
update the E-R query for that.

andrea

[0]
http://logs.openstack.org/04/426604/3/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/708771d/logs/syslog.txt.gz#_Feb_01_11_47_53
[1]
http://lists.openstack.org/pipermail/openstack-dev/2017-February/111347.html

[2] https://bugs.launchpad.net/tempest/+bug/1646779


On Wed, Feb 1, 2017 at 1:09 PM Davanum Srinivas  wrote:

MattR,

Three more from this morning, at least the first one looks new:

http://logs.openstack.org/04/426604/3/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/708771d/logs/testr_results.html.gz
tempest.api.compute.images.test_list_image_filters

http://logs.openstack.org/99/420299/3/gate/gate-tempest-dsvm-neutron-dvr-ubuntu-xenial/6e8d208/logs/testr_results.html.gz
http://logs.openstack.org/23/427223/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/18e635a/logs/testr_results.html.gz
tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON

Thanks,
Dims


On Tue, Jan 31, 2017 at 10:23 PM, Matt Riedemann 
wrote:
> On 1/31/2017 11:49 AM, Davanum Srinivas wrote:
>>
>> Folks,
>>
>> Here's the list of job failures that failed in the gate queue.
>> captured with my script[1][2] since around 10:00 AM today. All jobs
>> failed with just one bad test.
>>
>>
>>
http://logs.openstack.org/48/426448/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/ecb3d0a/
>>-
>> tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON
>>
>>
http://logs.openstack.org/48/426448/2/gate/gate-tempest-dsvm-neutron-full-ssh/71f6c8c/
>>  -
tempest.api.compute.admin.test_servers.ServersAdminTestJSON
>>
>>
http://logs.openstack.org/48/376548/8/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/cf3028b/
>>- tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON
>>
>>
http://logs.openstack.org/68/417668/8/gate/gate-tempest-dsvm-neutron-full-ssh/27bda02/
>>  -
>> tempest.api.compute.volumes.test_attach_volume.AttachVolumeShelveTestJSON
>>
>>
http://logs.openstack.org/48/423548/11/gate/gate-keystone-python27-db-ubuntu-xenial/a1f55ca/
>>- keystone.tests.unit.test_v3_auth.TestMFARules
>>
>>
http://logs.openstack.org/61/424961/1/gate/gate-tempest-dsvm-cells-ubuntu-xenial/8a1f9e7/
>>   - tempest.api.compute.admin.test_servers.ServersAdminTestJSON
>>
>>
http://logs.openstack.org/23/426823/3/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0204168/
>>-
>>
tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps
>>
>> So our gate is now 36 deep with stuff running for little more than 4
>> hours repeatedly Can folks look deeper please?
>>
>> Thanks,
>> Dims
>>
>> [1] https://gist.github.com/dims/54b391bd5964d3d208113b16766ea85e
>> [2] http://paste.openstack.org/show/597071/
>>
>
> I've identified a regression in nova here:
>
> https://bugs.launchpad.net/nova/+bug/1660878
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] gate jobs - papercuts

2017-02-01 Thread Davanum Srinivas

MattR,

Three more from this morning, at least the first one looks new:

http://logs.openstack.org/04/426604/3/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/708771d/logs/testr_results.html.gz
tempest.api.compute.images.test_list_image_filters

http://logs.openstack.org/99/420299/3/gate/gate-tempest-dsvm-neutron-dvr-ubuntu-xenial/6e8d208/logs/testr_results.html.gz
http://logs.openstack.org/23/427223/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/18e635a/logs/testr_results.html.gz
tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON

Thanks,
Dims


On Tue, Jan 31, 2017 at 10:23 PM, Matt Riedemann  wrote:
> On 1/31/2017 11:49 AM, Davanum Srinivas wrote:
>>
>> Folks,
>>
>> Here's the list of job failures that failed in the gate queue.
>> captured with my script[1][2] since around 10:00 AM today. All jobs
>> failed with just one bad test.
>>
>>
>> http://logs.openstack.org/48/426448/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/ecb3d0a/
>>-
>> tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON
>>
>> http://logs.openstack.org/48/426448/2/gate/gate-tempest-dsvm-neutron-full-ssh/71f6c8c/
>>  - tempest.api.compute.admin.test_servers.ServersAdminTestJSON
>>
>> http://logs.openstack.org/48/376548/8/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/cf3028b/
>>- tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON
>>
>> http://logs.openstack.org/68/417668/8/gate/gate-tempest-dsvm-neutron-full-ssh/27bda02/
>>  -
>> tempest.api.compute.volumes.test_attach_volume.AttachVolumeShelveTestJSON
>>
>> http://logs.openstack.org/48/423548/11/gate/gate-keystone-python27-db-ubuntu-xenial/a1f55ca/
>>- keystone.tests.unit.test_v3_auth.TestMFARules
>>
>> http://logs.openstack.org/61/424961/1/gate/gate-tempest-dsvm-cells-ubuntu-xenial/8a1f9e7/
>>   - tempest.api.compute.admin.test_servers.ServersAdminTestJSON
>>
>> http://logs.openstack.org/23/426823/3/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0204168/
>>-
>> tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps
>>
>> So our gate is now 36 deep with stuff running for little more than 4
>> hours repeatedly Can folks look deeper please?
>>
>> Thanks,
>> Dims
>>
>> [1] https://gist.github.com/dims/54b391bd5964d3d208113b16766ea85e
>> [2] http://paste.openstack.org/show/597071/
>>
>
> I've identified a regression in nova here:
>
> https://bugs.launchpad.net/nova/+bug/1660878
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] gate jobs - papercuts

2017-01-31 Thread Matt Riedemann


On 1/31/2017 11:49 AM, Davanum Srinivas wrote:

Folks,

Here's the list of job failures that failed in the gate queue.
captured with my script[1][2] since around 10:00 AM today. All jobs
failed with just one bad test.

http://logs.openstack.org/48/426448/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/ecb3d0a/
   - tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON
http://logs.openstack.org/48/426448/2/gate/gate-tempest-dsvm-neutron-full-ssh/71f6c8c/
 - tempest.api.compute.admin.test_servers.ServersAdminTestJSON
http://logs.openstack.org/48/376548/8/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/cf3028b/
   - tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON
http://logs.openstack.org/68/417668/8/gate/gate-tempest-dsvm-neutron-full-ssh/27bda02/
 - 
tempest.api.compute.volumes.test_attach_volume.AttachVolumeShelveTestJSON
http://logs.openstack.org/48/423548/11/gate/gate-keystone-python27-db-ubuntu-xenial/a1f55ca/
   - keystone.tests.unit.test_v3_auth.TestMFARules
http://logs.openstack.org/61/424961/1/gate/gate-tempest-dsvm-cells-ubuntu-xenial/8a1f9e7/
  - tempest.api.compute.admin.test_servers.ServersAdminTestJSON
http://logs.openstack.org/23/426823/3/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0204168/
   - tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps

So our gate is now 36 deep with stuff running for little more than 4
hours repeatedly Can folks look deeper please?

Thanks,
Dims

[1] https://gist.github.com/dims/54b391bd5964d3d208113b16766ea85e
[2] http://paste.openstack.org/show/597071/



I've identified a regression in nova here:

https://bugs.launchpad.net/nova/+bug/1660878

--

Thanks,

Matt Riedemann

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] gate jobs - papercuts

2017-01-31 Thread Morgan Fainberg

On Tue, Jan 31, 2017 at 1:55 PM, Morgan Fainberg 
wrote:

>
>
> On Tue, Jan 31, 2017 at 10:37 AM, Matthew Treinish 
> wrote:
>
>> On Tue, Jan 31, 2017 at 01:19:41PM -0500, Steve Martinelli wrote:
>> > On Tue, Jan 31, 2017 at 12:49 PM, Davanum Srinivas 
>> > wrote:
>> >
>> > > Folks,
>> > >
>> > > Here's the list of job failures that failed in the gate queue.
>> > > captured with my script[1][2] since around 10:00 AM today. All jobs
>> > > failed with just one bad test.
>> > >
>> > > http://logs.openstack.org/48/423548/11/gate/gate-keystone-
>> > > python27-db-ubuntu-xenial/a1f55ca/
>> > >- keystone.tests.unit.test_v3_auth.TestMFARules
>> > >
>> > > > m-cells-ubuntu-xenial/8a1f9e7/>
>> >
>> >
>> > This was due to a race condition between token issuance and validation,
>> > should be fixed.
>>
>> Is there a bug open for this? If so lets get an elastic-recheck query up
>> for it
>> so we can track it and get it off the uncategorized page:
>>
>>
> No bug. Also this is not really fixable because time resolution within
> tokens and revocations is 1 second. The answer is
> to use freezegun and freeze time when doing things that can cause
> revocations at the same time as issuance (usually can only really be hit
> within keystone's unit tests). It is also unlikely to be something that can
> easily be searched for in elastic search as it revolves around a "token
> cannot be validated" message (token Not found/revoked/etc), which is used
> in many cases where tokens cannot be validated (both correctly and in cases
> like this).
>
> The other case(es) that hit this actually were so bad they only passed at
> a ~5% rate.
>

Meaning it didn't get to the point where it could gate that was less than
5% and it was hit in multiple tests at once.

>
> So in short, an elastic-recheck-query would be pointless here short of
> looking specifically for the test name as a failure.
>
>
>> http://status.openstack.org/elastic-recheck/data/integrated_gate.html
>>
>> Our categorization rate is quite low right now and it'll only make things
>> harder
>> to debug other failures if we've got a bunch of unknown races going on.
>>
>> We have a lot of tools to make debugging the gate easier and making
>> everyone more
>> productive. But, it feels like we haven't been utilizing them fully
>> lately which
>> makes gate backups more likely and digging out of the hole harder.
>>
>> Thanks,
>>
>> Matt Treinish
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] gate jobs - papercuts

2017-01-31 Thread Morgan Fainberg

On Tue, Jan 31, 2017 at 10:37 AM, Matthew Treinish 
wrote:

> On Tue, Jan 31, 2017 at 01:19:41PM -0500, Steve Martinelli wrote:
> > On Tue, Jan 31, 2017 at 12:49 PM, Davanum Srinivas 
> > wrote:
> >
> > > Folks,
> > >
> > > Here's the list of job failures that failed in the gate queue.
> > > captured with my script[1][2] since around 10:00 AM today. All jobs
> > > failed with just one bad test.
> > >
> > > http://logs.openstack.org/48/423548/11/gate/gate-keystone-
> > > python27-db-ubuntu-xenial/a1f55ca/
> > >- keystone.tests.unit.test_v3_auth.TestMFARules
> > >
> > >  dsvm-cells-ubuntu-xenial/8a1f9e7/>
> >
> >
> > This was due to a race condition between token issuance and validation,
> > should be fixed.
>
> Is there a bug open for this? If so lets get an elastic-recheck query up
> for it
> so we can track it and get it off the uncategorized page:
>
>
No bug. Also this is not really fixable because time resolution within
tokens and revocations is 1 second. The answer is
to use freezegun and freeze time when doing things that can cause
revocations at the same time as issuance (usually can only really be hit
within keystone's unit tests). It is also unlikely to be something that can
easily be searched for in elastic search as it revolves around a "token
cannot be validated" message (token Not found/revoked/etc), which is used
in many cases where tokens cannot be validated (both correctly and in cases
like this).

The other case(es) that hit this actually were so bad they only passed at a
~5% rate.

So in short, an elastic-recheck-query would be pointless here short of
looking specifically for the test name as a failure.


> http://status.openstack.org/elastic-recheck/data/integrated_gate.html
>
> Our categorization rate is quite low right now and it'll only make things
> harder
> to debug other failures if we've got a bunch of unknown races going on.
>
> We have a lot of tools to make debugging the gate easier and making
> everyone more
> productive. But, it feels like we haven't been utilizing them fully lately
> which
> makes gate backups more likely and digging out of the hole harder.
>
> Thanks,
>
> Matt Treinish
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] gate jobs - papercuts

2017-01-31 Thread Davanum Srinivas

Thanks MattT, MattR and Steve. Since that last update 4 runs failed

http://logs.openstack.org/20/396620/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/e82ace8/
* tempest.api.compute.admin.test_migrations.MigrationsAdminTest -
test_resize_server_revert_deleted_flavor
* tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON
- test_create_list_show_delete_interfaces

http://logs.openstack.org/45/423645/19/gate/gate-grenade-dsvm-neutron-dvr-multinode-ubuntu-xenial/61dbd0e/
http://logs.openstack.org/45/423645/19/gate/gate-grenade-dsvm-neutron-dvr-multinode-ubuntu-xenial/61dbd0e/
* Both runs failed with the following
  "Failed to fetch
http://mirror.regionone.osic-cloud1.openstack.org/ubuntu/pool/main/o/openssl/openssl_1.0.2g-1ubuntu4.6_amd64.deb;

* 
http://logs.openstack.org/04/427004/2/gate/gate-keystone-python35-db/1502dbe/console.html
  35 mins of zero logs and then timed out

Thanks,
Dims

On Tue, Jan 31, 2017 at 3:32 PM, Matt Riedemann  wrote:
> On 1/31/2017 11:49 AM, Davanum Srinivas wrote:
>>
>> Folks,
>>
>> Here's the list of job failures that failed in the gate queue.
>> captured with my script[1][2] since around 10:00 AM today. All jobs
>> failed with just one bad test.
>>
>>
>> http://logs.openstack.org/48/426448/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/ecb3d0a/
>>-
>> tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON
>>
>> http://logs.openstack.org/48/426448/2/gate/gate-tempest-dsvm-neutron-full-ssh/71f6c8c/
>>  - tempest.api.compute.admin.test_servers.ServersAdminTestJSON
>>
>> http://logs.openstack.org/48/376548/8/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/cf3028b/
>>- tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON
>>
>> http://logs.openstack.org/68/417668/8/gate/gate-tempest-dsvm-neutron-full-ssh/27bda02/
>>  -
>> tempest.api.compute.volumes.test_attach_volume.AttachVolumeShelveTestJSON
>>
>> http://logs.openstack.org/48/423548/11/gate/gate-keystone-python27-db-ubuntu-xenial/a1f55ca/
>>- keystone.tests.unit.test_v3_auth.TestMFARules
>>
>> http://logs.openstack.org/61/424961/1/gate/gate-tempest-dsvm-cells-ubuntu-xenial/8a1f9e7/
>>   - tempest.api.compute.admin.test_servers.ServersAdminTestJSON
>>
>> http://logs.openstack.org/23/426823/3/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0204168/
>>-
>> tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps
>>
>> So our gate is now 36 deep with stuff running for little more than 4
>> hours repeatedly Can folks look deeper please?
>>
>> Thanks,
>> Dims
>>
>> [1] https://gist.github.com/dims/54b391bd5964d3d208113b16766ea85e
>> [2] http://paste.openstack.org/show/597071/
>>
>
> I know of two issues impacting the cells v1 job, one of which is fixed, the
> other has a patch recently posted.
>
> The first was one I posted about last night, total blocker for the cells v1
> job which was kicking things out of the gate for Nova, but that is fixed:
>
> https://review.openstack.org/#/c/427009/
>
> The other one that's not fixed yet (was just identified today) has a patch
> up now:
>
> https://review.openstack.org/#/c/427394/
>
> --
>
> Thanks,
>
> Matt Riedemann
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] gate jobs - papercuts

2017-01-31 Thread Matt Riedemann


On 1/31/2017 11:49 AM, Davanum Srinivas wrote:

Folks,

Here's the list of job failures that failed in the gate queue.
captured with my script[1][2] since around 10:00 AM today. All jobs
failed with just one bad test.

http://logs.openstack.org/48/426448/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/ecb3d0a/
   - tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON
http://logs.openstack.org/48/426448/2/gate/gate-tempest-dsvm-neutron-full-ssh/71f6c8c/
 - tempest.api.compute.admin.test_servers.ServersAdminTestJSON
http://logs.openstack.org/48/376548/8/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/cf3028b/
   - tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON
http://logs.openstack.org/68/417668/8/gate/gate-tempest-dsvm-neutron-full-ssh/27bda02/
 - 
tempest.api.compute.volumes.test_attach_volume.AttachVolumeShelveTestJSON
http://logs.openstack.org/48/423548/11/gate/gate-keystone-python27-db-ubuntu-xenial/a1f55ca/
   - keystone.tests.unit.test_v3_auth.TestMFARules
http://logs.openstack.org/61/424961/1/gate/gate-tempest-dsvm-cells-ubuntu-xenial/8a1f9e7/
  - tempest.api.compute.admin.test_servers.ServersAdminTestJSON
http://logs.openstack.org/23/426823/3/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0204168/
   - tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps

So our gate is now 36 deep with stuff running for little more than 4
hours repeatedly Can folks look deeper please?

Thanks,
Dims

[1] https://gist.github.com/dims/54b391bd5964d3d208113b16766ea85e
[2] http://paste.openstack.org/show/597071/



I know of two issues impacting the cells v1 job, one of which is fixed, 
the other has a patch recently posted.


The first was one I posted about last night, total blocker for the cells 
v1 job which was kicking things out of the gate for Nova, but that is fixed:


https://review.openstack.org/#/c/427009/

The other one that's not fixed yet (was just identified today) has a 
patch up now:


https://review.openstack.org/#/c/427394/

--

Thanks,

Matt Riedemann

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] gate jobs - papercuts

2017-01-31 Thread Matthew Treinish

On Tue, Jan 31, 2017 at 01:19:41PM -0500, Steve Martinelli wrote:
> On Tue, Jan 31, 2017 at 12:49 PM, Davanum Srinivas 
> wrote:
> 
> > Folks,
> >
> > Here's the list of job failures that failed in the gate queue.
> > captured with my script[1][2] since around 10:00 AM today. All jobs
> > failed with just one bad test.
> >
> > http://logs.openstack.org/48/423548/11/gate/gate-keystone-
> > python27-db-ubuntu-xenial/a1f55ca/
> >- keystone.tests.unit.test_v3_auth.TestMFARules
> >
> > 
> 
> 
> This was due to a race condition between token issuance and validation,
> should be fixed.

Is there a bug open for this? If so lets get an elastic-recheck query up for it
so we can track it and get it off the uncategorized page:

http://status.openstack.org/elastic-recheck/data/integrated_gate.html

Our categorization rate is quite low right now and it'll only make things harder
to debug other failures if we've got a bunch of unknown races going on.

We have a lot of tools to make debugging the gate easier and making everyone 
more
productive. But, it feels like we haven't been utilizing them fully lately which
makes gate backups more likely and digging out of the hole harder.

Thanks,

Matt Treinish

signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] gate jobs - papercuts

2017-01-31 Thread Matthew Treinish

On Tue, Jan 31, 2017 at 12:49:13PM -0500, Davanum Srinivas wrote:
> Folks,
> 
> Here's the list of job failures that failed in the gate queue.
> captured with my script[1][2] since around 10:00 AM today. All jobs
> failed with just one bad test.
> 
> http://logs.openstack.org/48/426448/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/ecb3d0a/
>- tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON
> http://logs.openstack.org/48/426448/2/gate/gate-tempest-dsvm-neutron-full-ssh/71f6c8c/
>  - tempest.api.compute.admin.test_servers.ServersAdminTestJSON
> http://logs.openstack.org/48/376548/8/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/cf3028b/
>- tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON
> http://logs.openstack.org/68/417668/8/gate/gate-tempest-dsvm-neutron-full-ssh/27bda02/
>  - 
> tempest.api.compute.volumes.test_attach_volume.AttachVolumeShelveTestJSON
> http://logs.openstack.org/48/423548/11/gate/gate-keystone-python27-db-ubuntu-xenial/a1f55ca/
>- keystone.tests.unit.test_v3_auth.TestMFARules
> http://logs.openstack.org/61/424961/1/gate/gate-tempest-dsvm-cells-ubuntu-xenial/8a1f9e7/
>   - tempest.api.compute.admin.test_servers.ServersAdminTestJSON
> http://logs.openstack.org/23/426823/3/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0204168/
>- 
> tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps
> 
> So our gate is now 36 deep with stuff running for little more than 4
> hours repeatedly Can folks look deeper please?
> 
> Thanks,
> Dims
> 
> [1] https://gist.github.com/dims/54b391bd5964d3d208113b16766ea85e
> [2] http://paste.openstack.org/show/597071/

Just as an aside this basic view is integrated into the home page on
openstack-health:

http://status.openstack.org/openstack-health/#/

under the section "Failed Tests in Last 10 Failed Runs". It also hooks into
elastic-recheck and will point out e-r hits there too. So, people don't need
to run this script manually to see what is failing.

Thanks,

Matt Treinish


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] gate jobs - papercuts

2017-01-31 Thread Steve Martinelli

On Tue, Jan 31, 2017 at 12:49 PM, Davanum Srinivas 
wrote:

> Folks,
>
> Here's the list of job failures that failed in the gate queue.
> captured with my script[1][2] since around 10:00 AM today. All jobs
> failed with just one bad test.
>
> http://logs.openstack.org/48/423548/11/gate/gate-keystone-
> python27-db-ubuntu-xenial/a1f55ca/
>- keystone.tests.unit.test_v3_auth.TestMFARules
>
> 

This was due to a race condition between token issuance and validation,
should be fixed.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] gate jobs - papercuts

2017-01-31 Thread Davanum Srinivas

Folks,

Here's the list of job failures that failed in the gate queue.
captured with my script[1][2] since around 10:00 AM today. All jobs
failed with just one bad test.

http://logs.openstack.org/48/426448/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/ecb3d0a/
   - tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON
http://logs.openstack.org/48/426448/2/gate/gate-tempest-dsvm-neutron-full-ssh/71f6c8c/
 - tempest.api.compute.admin.test_servers.ServersAdminTestJSON
http://logs.openstack.org/48/376548/8/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/cf3028b/
   - tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON
http://logs.openstack.org/68/417668/8/gate/gate-tempest-dsvm-neutron-full-ssh/27bda02/
 - 
tempest.api.compute.volumes.test_attach_volume.AttachVolumeShelveTestJSON
http://logs.openstack.org/48/423548/11/gate/gate-keystone-python27-db-ubuntu-xenial/a1f55ca/
   - keystone.tests.unit.test_v3_auth.TestMFARules
http://logs.openstack.org/61/424961/1/gate/gate-tempest-dsvm-cells-ubuntu-xenial/8a1f9e7/
  - tempest.api.compute.admin.test_servers.ServersAdminTestJSON
http://logs.openstack.org/23/426823/3/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0204168/
   - tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps

So our gate is now 36 deep with stuff running for little more than 4
hours repeatedly Can folks look deeper please?

Thanks,
Dims

[1] https://gist.github.com/dims/54b391bd5964d3d208113b16766ea85e
[2] http://paste.openstack.org/show/597071/

-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] gate jobs - papercuts

Re: [openstack-dev] gate jobs - papercuts

Re: [openstack-dev] gate jobs - papercuts

Re: [openstack-dev] gate jobs - papercuts

Re: [openstack-dev] gate jobs - papercuts

Re: [openstack-dev] gate jobs - papercuts

Re: [openstack-dev] gate jobs - papercuts

Re: [openstack-dev] gate jobs - papercuts

Re: [openstack-dev] gate jobs - papercuts

Re: [openstack-dev] gate jobs - papercuts

Re: [openstack-dev] gate jobs - papercuts

Re: [openstack-dev] gate jobs - papercuts

Re: [openstack-dev] gate jobs - papercuts

Re: [openstack-dev] gate jobs - papercuts

[openstack-dev] gate jobs - papercuts

15 matches

Site Navigation

Mail list logo

Footer information