Status: New
Owner: ----
Labels: Type-Enhancement Priority-Medium
New issue 837 by [email protected]: request for a new attribute for
(executed) test-cases / suites: fail-classification
http://code.google.com/p/robotframework/issues/detail?id=837
1) Background
I'm working on firmware development for embedded systems. Firmware gets
tests via RobotFramework. The test host communicates via TCP/IP with the
SUT (embedded system).
I've recently made an analysis of the whole history of regression tests
done by the continuous integration server in the environment described
above.
Going through the failed tests and the generated messages there showed up
some typical types of failures and causes:
a) test execution errors (suite setup/teardown errors, initialization
errors, sometimes keyword / tests data errors)
b) communication errors (communication between test host and SUT broken)
c) real regression errors - wrong behavior of the system and or software
under test
The message for the various types of error differ very frequently and can't
that easily be used to map them into one of the categories.
2) Goal
During post processing of the test logs (by the CI server) it should be
possible to automatically determine if real regression failures occurred.
Regression failures should be treated as "hard errors", other types of
failures should be treated more as some kind of a warning.
In a CI scenario, developers are responsible for caring about "hard errors"
(regression errors) and test maintainers need to take care about "warnings"
and fix up issues like communication errors.
So finally the "noise" coming from not yet stable enough test environments
or host <-> SUT communication should not be passed to the firmware
developers.
3) Request for new feature
RF should provide an ability to plug-in a classification function which
takes - in case of a failed test - the failure message and transforms into
into a fail classification ID (string).
This fail classification should be added as independent attribute to the
test cases and also be displayed in the reports.
4) Additional thoughts
a) Similar to the statistics by tags, there should be a statistic by
fail-classification
b) Depending from the fail-classification the test status should be set to
FAIL (hard error) or "WARN" (instable, wanted checks of the test case not
done because of generic problems not directly caused by the SUT)
c) It should be configurable, whether for the exit code of robot runner
tests completed with WARN status are considered in the same way as with
FAIL status.
In a CI scenario a build can be considered as successful too, if there a
only tests with PASS and some with WARN status. Some tests might not be
executed like specified, but there is no need for developers to check code
changes. Rather it's the job for test maintainers to stabilize tests and
test environment.
If anybody has similar needs, I'd like to discuss about and find the
best "model" to handle that kind of test execution tracking.
