On Tue, Jan 31, 2017 at 1:55 PM, Morgan Fainberg <[email protected]> wrote:
> > > On Tue, Jan 31, 2017 at 10:37 AM, Matthew Treinish <[email protected]> > wrote: > >> On Tue, Jan 31, 2017 at 01:19:41PM -0500, Steve Martinelli wrote: >> > On Tue, Jan 31, 2017 at 12:49 PM, Davanum Srinivas <[email protected]> >> > wrote: >> > >> > > Folks, >> > > >> > > Here's the list of job failures that failed in the gate queue. >> > > captured with my script[1][2] since around 10:00 AM today. All jobs >> > > failed with just one bad test. >> > > >> > > http://logs.openstack.org/48/423548/11/gate/gate-keystone- >> > > python27-db-ubuntu-xenial/a1f55ca/ >> > > - keystone.tests.unit.test_v3_auth.TestMFARules >> > > >> > > <http://logs.openstack.org/61/424961/1/gate/gate-tempest-dsv >> m-cells-ubuntu-xenial/8a1f9e7/> >> > >> > >> > This was due to a race condition between token issuance and validation, >> > should be fixed. >> >> Is there a bug open for this? If so lets get an elastic-recheck query up >> for it >> so we can track it and get it off the uncategorized page: >> >> > No bug. Also this is not really fixable because time resolution within > tokens and revocations is 1 second. The answer is > to use freezegun and freeze time when doing things that can cause > revocations at the same time as issuance (usually can only really be hit > within keystone's unit tests). It is also unlikely to be something that can > easily be searched for in elastic search as it revolves around a "token > cannot be validated" message (token Not found/revoked/etc), which is used > in many cases where tokens cannot be validated (both correctly and in cases > like this). > > The other case(es) that hit this actually were so bad they only passed at > a ~5% rate. > Meaning it didn't get to the point where it could gate that was less than 5% and it was hit in multiple tests at once. > > So in short, an elastic-recheck-query would be pointless here short of > looking specifically for the test name as a failure. > > >> http://status.openstack.org/elastic-recheck/data/integrated_gate.html >> >> Our categorization rate is quite low right now and it'll only make things >> harder >> to debug other failures if we've got a bunch of unknown races going on. >> >> We have a lot of tools to make debugging the gate easier and making >> everyone more >> productive. But, it feels like we haven't been utilizing them fully >> lately which >> makes gate backups more likely and digging out of the hole harder. >> >> Thanks, >> >> Matt Treinish >> >> ____________________________________________________________ >> ______________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: [email protected]?subject:unsubscrib >> e >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
