No objections from me (I have enough crashes and fails to worry about,
so "warns" are rarely something I have time for.)
It might make sense to have an option like GCC's "-Werror" that makes
warnings errors. Though I suppose anybody interesting in finding out
warnings can also run the json backend.
Jose
On 26/06/15 19:11, Mark Janes wrote:
Ilia Mirkin <[email protected]> writes:
So a test that otherwise passes (i.e. has a "pass" result code) but
generates errors in dmesg will get converted to a "dmesg-warn" code.
You were previously treating those as failures, but now will treat
them as success.
Yes. I used the comment in framework/status.py to determine what
severities of piglit status correspond to JUnit failures:
Status ordering from best to worst:
pass
dmesg-warn
warn
dmesg-fail
fail
timeout
crash
If warn doesn't trigger a failure, then dmesg-warn shouldn't either.
I don't feel strongly either way, but just wanted to point it out for
your consideration. Also adding Jose who IIRC is also a junit user.
[I, btw, am not.]
Thanks, I should have remembered to do that. In my own tests, I found
the following tests that were warning:
HSW, IVB, SNB, BDW:
ext_transform_feedback.tessellation quads flat_last
ext_transform_feedback.tessellation quad_strip flat_last
ext_transform_feedback.tessellation polygon flat_last
The tests that provided output indicated pixel color accuracy was beyond
the warning threshold.
G45, G965:
!opengl 1_1.teximage-colors gl_r32f
!opengl 1_1.teximage-colors gl_rg32f
!opengl 1_1.teximage-colors gl_rgb32f
!opengl 1_1.teximage-colors gl_rgba32f
No output was provided for these warnings, so they may have have
previously been dmesg-warn.
On Fri, Jun 26, 2015 at 1:40 PM, Mark Janes <[email protected]> wrote:
JUnit has no concept of "warning". It supports the following
statuses:
- skip
- success
- fail
- error
dEQP has been found to intermittently emit warnings for passed tests,
and this status is accurately represented in piglit json. However,
current JUnit transforms them into failures.
A test which emits a warning is more accurately represented as
"success" in JUnit.
---
framework/backends/junit.py | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/framework/backends/junit.py b/framework/backends/junit.py
index 632e516..7499829 100644
--- a/framework/backends/junit.py
+++ b/framework/backends/junit.py
@@ -142,8 +142,7 @@ class JUnitBackend(FileBackend):
if data['result'] == 'skip':
res = etree.SubElement(element, 'skipped')
- elif data['result'] in ['warn', 'fail', 'dmesg-warn',
- 'dmesg-fail']:
+ elif data['result'] in ['fail', 'dmesg-fail']:
if expected_result == "failure":
err.text += "\n\nWARN: passing test as an expected
failure"
res = etree.SubElement(element, 'skipped',
--
2.1.4
_______________________________________________
Piglit mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/piglit
