https://bugzilla.wikimedia.org/show_bug.cgi?id=65773
Bug ID: 65773
Summary: investigate re-trying each failed test within a build
Product: Wikimedia
Version: wmf-deployment
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: Unprioritized
Component: Quality Assurance
Assignee: [email protected]
Reporter: [email protected]
CC: [email protected], [email protected]
Web browser: ---
Mobile Platform: ---
Many browser tests fail for reasons other than an issue with the code or the
test environment: a flaky timeout, a momentary network glitch, etc. etc.
It would be convenient to build into the command that calls Cucumber to run the
tests the ability to:
* track which test is being run
* evaluate the return status of each test, either per Feature or possibly even
per Secenario
* if the return value for the individual test is non-zero, do not record the
test result. Instead run the single test with the single failure.
* if the second run of the single test returns a non-zero result, record that
result as usual
* if the test passes on the second run, record that result and continue running
the tests in the build.
Some years ago I wrote a harness like this in Ruby. If it is not convenient to
do this directly in a shell script, it might be possible to use Ruby and shell
out for the appropriate commands.
--
You are receiving this mail because:
You are the assignee for the bug.
You are on the CC list for the bug.
_______________________________________________
Wikibugs-l mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
