Re: Intermittent oranges and when to disable the related test case - a simplified policy

2017-09-12 Thread Steve Fink
On 9/12/17 7:02 AM, James Graham wrote: On 12/09/17 14:55, Andrew Halberstadt wrote: On Mon, Sep 11, 2017 at 10:33 PM Robert O'Callahan wrote: On Tue, Sep 12, 2017 at 11:38 AM, Andrew Halberstadt < ahalberst...@mozilla.com> wrote: I don't think so, that data already

Re: Intermittent oranges and when to disable the related test case - a simplified policy

2017-09-12 Thread James Graham
On 12/09/17 14:55, Andrew Halberstadt wrote: On Mon, Sep 11, 2017 at 10:33 PM Robert O'Callahan wrote: On Tue, Sep 12, 2017 at 11:38 AM, Andrew Halberstadt < ahalberst...@mozilla.com> wrote: I don't think so, that data already exists and is query-able from ActiveData:

Re: Intermittent oranges and when to disable the related test case - a simplified policy

2017-09-12 Thread Andrew Halberstadt
On Mon, Sep 11, 2017 at 10:33 PM Robert O'Callahan wrote: > On Tue, Sep 12, 2017 at 11:38 AM, Andrew Halberstadt < > ahalberst...@mozilla.com> wrote: > >> I don't think so, that data already exists and is query-able from >> ActiveData: >>

Re: Intermittent oranges and when to disable the related test case - a simplified policy

2017-09-11 Thread Robert O'Callahan
On Tue, Sep 12, 2017 at 11:38 AM, Andrew Halberstadt < ahalberst...@mozilla.com> wrote: > I don't think so, that data already exists and is query-able from > ActiveData: > https://activedata.allizom.org/tools/query.html#query_id=8pDOpeni That query tells you about disabled tests, but doesn't

Re: Intermittent oranges and when to disable the related test case - a simplified policy

2017-09-11 Thread Andrew Halberstadt
On Fri, Sep 8, 2017 at 7:10 PM Gregory Szorc wrote: > I know we've topic in this topic in the past but I can't recall outcomes. > Is it worthwhile to define and use a richer test manifest "schema" that > will facilitate querying and building dashboards so we have better >

Re: Intermittent oranges and when to disable the related test case - a simplified policy

2017-09-08 Thread Robert O'Callahan
On Sat, Sep 9, 2017 at 11:09 AM, Gregory Szorc wrote: > Is it worthwhile to define and use a richer test manifest "schema" that > will facilitate querying and building dashboards so we have better > visibility into disabled tests? > It would be great if there was a way to run

Re: Intermittent oranges and when to disable the related test case - a simplified policy

2017-09-08 Thread Gregory Szorc
On Wed, Sep 6, 2017 at 2:10 PM, wrote: > Over the last 9 months a few of us have really watched intermittent test > failures almost daily and done a lot to pester people as well as fix many. > While there are over 420 bugs that have been fixed since the beginning of > the

Re: Intermittent oranges and when to disable the related test case - a simplified policy

2017-09-06 Thread Daniel Veditz
On Wed, Sep 6, 2017 at 4:53 PM, Emma Humphries wrote: > This begs the question, why was that whiteboard tag being used that way? > ​Surely there are other reasons to disable tests, and people might want to track those too. If you want to restrict your new keyword to just

Re: Intermittent oranges and when to disable the related test case - a simplified policy

2017-09-06 Thread Emma Humphries
Andrew Swan noticed that whiteboard tag was showing up on bugs not filed by the intermittent-test-failure bot (thanks!) Here's the queries restricted to just bot filed bugs: https://mzl.la/2eMnMRz (add ons) https://mzl.la/2eMaPqQ (all core/ffx/toolkit) This begs the question, why was that

Re: Intermittent oranges and when to disable the related test case - a simplified policy

2017-09-06 Thread Emma Humphries
In order to be more consistent, marking a failing test as disabled should be done with a keyword instead of a whiteboard tag. Proposed keyword: test-disabled Description: Used by test automation team to indicate the failing test associated with this bug has a failure rate above a threshold level

Re: Intermittent oranges and when to disable the related test case - a simplified policy

2017-09-06 Thread Andrew McKay
Cool, thanks. On 6 September 2017 at 16:32, Emma Humphries wrote: > Andy > > To start: > >

Re: Intermittent oranges and when to disable the related test case - a simplified policy

2017-09-06 Thread Emma Humphries
Andy To start: https://bugzilla.mozilla.org/buglist.cgi?product=Core=Firefox=Firefox%20for%20Android=Firefox%20for%20iOS=Toolkit_format=advanced=---_whiteboard=test%20disabled%2Ctest-disabled%2Ctestdisabled_whiteboard_type=anywordssubstr=bug_id=0 Add-ons, Web Extensions, and Plugins related:

Re: Intermittent oranges and when to disable the related test case - a simplified policy

2017-09-06 Thread Andrew McKay
Is there an easy way for me to track what tests have been disabled as result of intermittent issues we haven't been able to fix? On 6 September 2017 at 14:10, wrote: > Over the last 9 months a few of us have really watched intermittent test > failures almost daily and done

Intermittent oranges and when to disable the related test case - a simplified policy

2017-09-06 Thread jmaher
Over the last 9 months a few of us have really watched intermittent test failures almost daily and done a lot to pester people as well as fix many. While there are over 420 bugs that have been fixed since the beginning of the year, there are half that many (211+) which have been disabled in