Mozilla - 2015 Talos performance regression policy Over the last year and a half the Talos tests have been rewritten to be more useful and meaningful. This means we need to take them seriously and cannot just ignore real issues when we don't have time. This does not mean we need to fix or backout every changeset that caused a regression.
Starting in 2015, when a regression is identified to be related to a specific changeset, the patch author will be ask for information via the needinfo flag. We expect a response and reasonable dialog within 72 hours (3 business days) of requesting information. If no response is given we will backout the patch(es) in question and the patch author can investigate when they have time and reland. Some requirements before requesting needinfo: * On integration branches (higher volume), a talos sheriff will have verified the root cause within 1 week of the patch landing * a patch or set of patches from a bug must be identified as the root cause. This can take place through retriggers on the tree or in the case of many patches landing at once this would take place through a push to try backing out the suspected patch(es) * links in the bug to document the regression (and any related regressions/improvements) * if we are confident this is the root cause and it meets a 3% regression threshold, then the needinfo request will mention that this policy will be enforced Acceptable outcomes: * A promise to attempt a fix at the bug is agreed upon, the bug is assigned to someone and put in a queue. * The bug will contain enough details and evidence to support accepting this regression, we will mark it as wontfix * It is agreed that this should be backed out Other scenarios: * A bug related to the alert is not filed within 1 week of the patch landing. This removes the urgency and required action. * We only caught a regression at uplift time. There is a chance this isn't easily determined, this will be documented and identified patch authors will use their judgement to fix the bug * Regression is unrelated to code (say pgo issue) - this should be documented in the bug and closed as wontfix. * When we uplift to Aurora or Beta, all regressions filed before the uplift that show up on the upstream branch will have a needinfo flag set and require action to be taken. Please take a moment to look over this and outline any concerns you might have. Thanks, Joel _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
