Steve A. created JENKINS-13120:
----------------------------------
Summary: Extend criteria for current regression/improvement
triggers
Key: JENKINS-13120
URL: https://issues.jenkins-ci.org/browse/JENKINS-13120
Project: Jenkins
Issue Type: Improvement
Components: email-ext
Reporter: Steve A.
The current criteria for the regression/improvement triggers is based on
failure stats only (e.g., regression triggers if number of failures has
increased since last build). However, would like to extend this set of criteria
to optionally include successes and total tests run. For example:
regression = # of failures has increased OR # of successes has decreased OR #
of total tests run has decreased
improvement = # of failures has decreased OR # of successes has increased OR #
of total tests run has increased
The background behind this request is a unit test "timeout" (mstest) which was
not reported as a failure. Looking at the normal Jenkins test result stats,
there was no indication that anything was wrong: there were no failures and the
total number of tests was the same. If the test result stats reported number of
successes, then I might have noticed this number had dropped, but it's a moot
point since this is not displayed. The regression trigger would not have kicked
in since the number of failures did not increase. For this specific scenario,
the "# of successes has decreased" regression trigger applies. The "total tests
run" trigger would come in handy in the situation where developers
inadvertently (or intentionally) enable/disable tests. Thanks, Steve
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
