Andrew Swan noticed that whiteboard tag was showing up on bugs not filed by the intermittent-test-failure bot (thanks!)
Here's the queries restricted to just bot filed bugs: https://mzl.la/2eMnMRz (add ons) https://mzl.la/2eMaPqQ (all core/ffx/toolkit) This begs the question, why was that whiteboard tag being used that way? On Wed, Sep 6, 2017 at 4:32 PM, Emma Humphries <e...@mozilla.com> wrote: > Andy > > To start: > > https://bugzilla.mozilla.org/buglist.cgi?product=Core&; > product=Firefox&product=Firefox%20for%20Android& > product=Firefox%20for%20iOS&product=Toolkit&query_format= > advanced&resolution=---&status_whiteboard=test%20disabled%2Ctest-disabled% > 2Ctestdisabled&status_whiteboard_type=anywordssubstr&order=bug_id&limit=0 > > Add-ons, Web Extensions, and Plugins related: > > https://bugzilla.mozilla.org/buglist.cgi?resolution=---&; > status_whiteboard_type=anywordssubstr&query_format= > advanced&status_whiteboard=test%20disabled%2Ctest-disabled%2Ctestdisabled& > component=Add-on%20Manager&component=Add-ons%20Manager& > component=Plug-ins&component=WebExtensions%3A%20Android& > component=WebExtensions%3A%20Compatibility&component= > WebExtensions%3A%20Developer%20Tools&component=WebExtensions%3A% > 20Experiments&component=WebExtensions%3A%20Frontend& > component=WebExtensions%3A%20General&component=WebExtensions%3A%20Request% > 20Handling&component=WebExtensions%3A%20Untriaged& > product=Core&product=Firefox&product=Firefox%20for% > 20Android&product=Firefox%20for%20iOS&product=Toolkit > > On Wed, Sep 6, 2017 at 3:46 PM, Andrew McKay <amc...@mozilla.com> wrote: > >> Is there an easy way for me to track what tests have been disabled as >> result of intermittent issues we haven't been able to fix? >> >> On 6 September 2017 at 14:10, <jma...@mozilla.com> wrote: >> > Over the last 9 months a few of us have really watched intermittent >> test failures almost daily and done a lot to pester people as well as fix >> many. While there are over 420 bugs that have been fixed since the >> beginning of the year, there are half that many (211+) which have been >> disabled in some form (including turning off the jobs). >> > >> > We don't like to disable and have been pretty relaxed in recommending >> disabling a test. Overall we have tried to adhere to a policy of: >> > * >=30 failures/week- ask for owner to look at failure and fix it, if >> this persists for a few weeks with no real traction we would go ahead [and >> recommend] disabling it. >> > * >= 75 failures/week- ask for people to fix this in a shorter time >> frame and recommend disabling the test in a week or so >> > * >= 150 failures/week- often just disable the test >> > >> > This is confusing and hard to manage. Since then we have started >> adjusting triage queries and some teams are doing their own triage and we >> are ignoring those bugs (while they are getting prioritized properly). >> > >> > What we are looking to start doing this month is adopting a simpler >> policy: >> > * any bug that has >=200 instances in the last 30 days will be disabled >> > ** this will be a manual process, so it will happen a couple times/week >> > >> > We expect the outcome of this to be a similar amount of disabling, just >> an easier method for doing so. It is very possible we might recommend >> disabling a test before it hits the threshold- keep in mind a disabled test >> is easy to re-enable (so feel free to disable for that one platform until >> you have time to look at fixing it) >> > >> > To be clear we (and some component owners) will continue triaging bugs >> and trying to get fixes in place as often as possible and prefer a fix, not >> a disabled test! >> > >> > Please raise any concerns, otherwise we will move forward with this in >> the coming weeks. >> > _______________________________________________ >> > dev-platform mailing list >> > dev-platform@lists.mozilla.org >> > https://lists.mozilla.org/listinfo/dev-platform >> _______________________________________________ >> dev-platform mailing list >> dev-platform@lists.mozilla.org >> https://lists.mozilla.org/listinfo/dev-platform >> > > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
