On Fri, Jun 26, 2015 at 2:27 PM, Daniel Vetter <[email protected]> wrote: > On Fri, Jun 26, 2015 at 10:40:28AM -0700, Mark Janes wrote: >> JUnit has no concept of "warning". It supports the following >> statuses: >> >> - skip >> - success >> - fail >> - error >> >> dEQP has been found to intermittently emit warnings for passed tests, >> and this status is accurately represented in piglit json. However, >> current JUnit transforms them into failures. > > Isn't that something the dEQP runner should take care of? If warnings > aren't valid there, then just don't flag any test results as warning. At > least for other cases I'd count gpu hangs in dmesg very much as a failure. > Hence this result frobbing shouldn't be done here imo.
IMHO there are two separate and unrelated issues which are being conflated here. The two issues are: (a) junit conversion of piglit statuses is wrong (b) deqp tests return random statuses sometimes for no good reason [I haven't personally verified this, just going by word-of-mouth here.] Both are issues, and both ought to be addressed. > -Daniel > >> >> A test which emits a warning is more accurately represented as >> "success" in JUnit. >> --- >> framework/backends/junit.py | 3 +-- >> 1 file changed, 1 insertion(+), 2 deletions(-) >> >> diff --git a/framework/backends/junit.py b/framework/backends/junit.py >> index 632e516..7499829 100644 >> --- a/framework/backends/junit.py >> +++ b/framework/backends/junit.py >> @@ -142,8 +142,7 @@ class JUnitBackend(FileBackend): >> if data['result'] == 'skip': >> res = etree.SubElement(element, 'skipped') >> >> - elif data['result'] in ['warn', 'fail', 'dmesg-warn', >> - 'dmesg-fail']: >> + elif data['result'] in ['fail', 'dmesg-fail']: >> if expected_result == "failure": >> err.text += "\n\nWARN: passing test as an expected >> failure" >> res = etree.SubElement(element, 'skipped', >> -- >> 2.1.4 >> >> _______________________________________________ >> Piglit mailing list >> [email protected] >> http://lists.freedesktop.org/mailman/listinfo/piglit > > -- > Daniel Vetter > Software Engineer, Intel Corporation > http://blog.ffwll.ch > _______________________________________________ > Piglit mailing list > [email protected] > http://lists.freedesktop.org/mailman/listinfo/piglit _______________________________________________ Piglit mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/piglit
