On 05/09/2016 03:46 PM, Dylan Baker wrote:
In piglit when a test generates subtests we treat the "worst" subtest as
the test status, and in most cases this works as expected. There is at
least one case where this is not correct, and that is the case of crash.
There are a couple of reasons that crash should not be masked. One is
that it is generated by the framework when the test binary hits an
assert or segfaults, or any number of similar cases. The second is that
it may mean that all of the subtests did not run, as such we don't want
the status to be masked by the "worst" subtest which would be fail at
worst.
Signed-off-by: Dylan Baker <[email protected]>
---
framework/results.py | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/framework/results.py b/framework/results.py
index 469abeb..2095d90 100644
--- a/framework/results.py
+++ b/framework/results.py
@@ -173,10 +173,13 @@ class TestResult(object):
"""Return the result of the test.
If there are subtests return the "worst" value of those subtests. If
- there are not return the stored value of the test.
+ there are not return the stored value of the test. There is an
+ exception to this rule, and that's if the status is crash; since this
+ status is set by the framework, and can be generated even when some or
+ all unit tests pass.
"""
- if self.subtests:
+ if self.subtests and self.__result != status.CRASH:
return max(six.itervalues(self.subtests))
return self.__result
This looks OK to me, and it fixes the incorrect "pass" result in the
json file.
Reviewed-by: Brian Paul <[email protected]>
Tested-by: Brian Paul <[email protected]>
However, when I run piglit-summary.py -s on the results file I get:
name: new
---- ------
pass: 70
fail: 2
crash: 0
skip: 19
timeout: 0
warn: 0
incomplete: 0
dmesg-warn: 0
dmesg-fail: 0
changes: 0
fixes: 0
regressions: 0
total: 91
The "crash" result in the json file doesn't show up here. I believe
when there's subtests, we're counting the subtest results but not the
overall test result. Is there a way to make sure that the crash result
shows up in the summary?
-Brian
_______________________________________________
Piglit mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/piglit
