On Tuesday, April 15, 2014 9:42:25 AM UTC-4, Kyle Huey wrote: > On Tue, Apr 15, 2014 at 6:21 AM, jmaher <joel.ma...@gmail.com> wrote: > > > This policy will define an escalation path for when a single test case is > > identified to be leaking or failing and is causing enough disruption on the > > trees. Disruption is defined as: > > > 1) Test case is on the list of top 20 intermittent failures on Orange > > Factor (http://brasstacks.mozilla.com/orangefactor/index.html) > > > 2) It is causing oranges >=8% of the time > > > 3) We have >100 instances of this failure in the bug in the last 30 days > > > > Are these conditions joined by 'and' or by 'or'? If 'or', there will > > always be at least 20 tests meeting this set of criteria ... > > > > - Kyle
Great question Kyle- The top 20 doesn't always include specific tests- sometimes it is infrastructure, hardware/vm, or test harness, mozharness, etc. related. If a test meets any of the above criteria and is escalated, then we should expect to follow some basic criteria about either working on fixing it or disabling it as spelled out in the escalation path. For the large majority of the cases, bugs filed for specific test cases will meet all 3 conditions. We have had some cases where we have thousands of stars over years, but it isn't on the top 20 list all the time. Likewise when we have 10 infra bugs, a frequent orange on the trees won't be in the top 20. -Joel _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
