On Fri, Jun 26, 2015 at 2:11 PM, Mark Janes <[email protected]> wrote: > Ilia Mirkin <[email protected]> writes: > >> So a test that otherwise passes (i.e. has a "pass" result code) but >> generates errors in dmesg will get converted to a "dmesg-warn" code. >> You were previously treating those as failures, but now will treat >> them as success. > > Yes. I used the comment in framework/status.py to determine what > severities of piglit status correspond to JUnit failures: > > Status ordering from best to worst: > > pass > dmesg-warn > warn > dmesg-fail > fail > timeout > crash
Pretty sure that dmesg-warn is worse than warn, and quite sure that dmesg-fail is worse than fail. I think that comment is out of date. Dylan? > > If warn doesn't trigger a failure, then dmesg-warn shouldn't either. >> >> I don't feel strongly either way, but just wanted to point it out for >> your consideration. Also adding Jose who IIRC is also a junit user. >> [I, btw, am not.] > > Thanks, I should have remembered to do that. In my own tests, I found > the following tests that were warning: > > HSW, IVB, SNB, BDW: > ext_transform_feedback.tessellation quads flat_last > ext_transform_feedback.tessellation quad_strip flat_last > ext_transform_feedback.tessellation polygon flat_last Yep, the test verifies something questionable and returns a 'warn' when that questionable thing is not satisifed. (The order of tessellation of complex primitives when doing transform feedback is unspecified.) > > The tests that provided output indicated pixel color accuracy was beyond > the warning threshold. > > G45, G965: > !opengl 1_1.teximage-colors gl_r32f > !opengl 1_1.teximage-colors gl_rg32f > !opengl 1_1.teximage-colors gl_rgb32f > !opengl 1_1.teximage-colors gl_rgba32f > > No output was provided for these warnings, so they may have have > previously been dmesg-warn. > >> >> On Fri, Jun 26, 2015 at 1:40 PM, Mark Janes <[email protected]> wrote: >>> JUnit has no concept of "warning". It supports the following >>> statuses: >>> >>> - skip >>> - success >>> - fail >>> - error >>> >>> dEQP has been found to intermittently emit warnings for passed tests, >>> and this status is accurately represented in piglit json. However, >>> current JUnit transforms them into failures. >>> >>> A test which emits a warning is more accurately represented as >>> "success" in JUnit. >>> --- >>> framework/backends/junit.py | 3 +-- >>> 1 file changed, 1 insertion(+), 2 deletions(-) >>> >>> diff --git a/framework/backends/junit.py b/framework/backends/junit.py >>> index 632e516..7499829 100644 >>> --- a/framework/backends/junit.py >>> +++ b/framework/backends/junit.py >>> @@ -142,8 +142,7 @@ class JUnitBackend(FileBackend): >>> if data['result'] == 'skip': >>> res = etree.SubElement(element, 'skipped') >>> >>> - elif data['result'] in ['warn', 'fail', 'dmesg-warn', >>> - 'dmesg-fail']: >>> + elif data['result'] in ['fail', 'dmesg-fail']: >>> if expected_result == "failure": >>> err.text += "\n\nWARN: passing test as an expected >>> failure" >>> res = etree.SubElement(element, 'skipped', >>> -- >>> 2.1.4 >>> _______________________________________________ Piglit mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/piglit
