Ilia Mirkin <[email protected]> writes: > On Fri, Jun 26, 2015 at 2:44 PM, Dylan Baker <[email protected]> wrote: >>> That's fine, but then that's the justification, not "junit interprets >>> warn as fail". Note that some (regular) piglit tests will also emit >>> "warn", and they don't mean "fail". I believe that junit needs to be >>> fixed to not interpret warn as fail irrespective of what happens in >>> deqp. But for the record, I don't use junit. >>> >>> I'm not sure how I feel about the "warn" output in piglit in general, >>> it seems a bit weird. Bogus and intermittent warnings certainly sound >>> like a problem, and perhaps that's reason enough to just ignore them. >> >> I'm fine with either solution. >> >> Warn is kind of an odd status. Basically if a test passes, but there is >> anything unexpected in stderr, the test will get marked warn. (I think >> some other suites use it differently, maybe igt?) > > Well, with the GL suite, a test can on its own decide to return a > "warn" status. The framework doesn't automatically add that in (or at > least didn't before). > > -ilia
And this is why "warn status" is problematic in a test suite. Consider the case where a commit introduces a new warning. Should a bug be written up? It depends: as with compiler warnings, some projects may want to be more strict, and disallow warnings. Other projects may not mind warnings, with the result that test writers emit warnings that are not actionable. At least with piglit the warnings are stable, so any CI displaying JUnit trends will not report spurious regressions. Additionally, the developers can easily remedy warnings which are intermittent. For this reason, warnings-as-errors provides a sharper edge with with to catch regressions in the core piglit test suite (caveat: I'm not sure how dmesg-warn is handled for tests executed in parallel). dEQP has 2 issues that work against using warnings as errors: * warnings are intermittent, at least on intel hardware. Minimal investigation by Chad indicated that the warnings are bogus. * developers don't have easy access to upstream, to improve the tests. There may be no optimal, consistent warning policy for both test suites. I'm inclined to turn down the warning noise emitted by dEQP, because it is external to piglit and can't be easily fixed. Even at Google, dEQP runners blacklist thousands of tests because they are noisy. -Mark _______________________________________________ Piglit mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/piglit
