On Fri, Jun 26, 2015 at 4:02 PM, Mark Janes <[email protected]> wrote: > Ilia Mirkin <[email protected]> writes: > >> On Fri, Jun 26, 2015 at 2:44 PM, Dylan Baker <[email protected]> wrote: >>>> That's fine, but then that's the justification, not "junit interprets >>>> warn as fail". Note that some (regular) piglit tests will also emit >>>> "warn", and they don't mean "fail". I believe that junit needs to be >>>> fixed to not interpret warn as fail irrespective of what happens in >>>> deqp. But for the record, I don't use junit. >>>> >>>> I'm not sure how I feel about the "warn" output in piglit in general, >>>> it seems a bit weird. Bogus and intermittent warnings certainly sound >>>> like a problem, and perhaps that's reason enough to just ignore them. >>> >>> I'm fine with either solution. >>> >>> Warn is kind of an odd status. Basically if a test passes, but there is >>> anything unexpected in stderr, the test will get marked warn. (I think >>> some other suites use it differently, maybe igt?) >> >> Well, with the GL suite, a test can on its own decide to return a >> "warn" status. The framework doesn't automatically add that in (or at >> least didn't before). >> >> -ilia > > And this is why "warn status" is problematic in a test suite. Consider > the case where a commit introduces a new warning. Should a bug be > written up? It depends: as with compiler warnings, some projects may > want to be more strict, and disallow warnings. Other projects may not > mind warnings, with the result that test writers emit warnings that are > not actionable.
Defaults are important. The default is "warn is ok". That's how I see it from piglit. I'd do the split between warn and dmesg-warn. The latter indicates a real issue, whereas warn indicates a highly theoretical one. If you add a flag to treat "warn" as error optionally (-Werror style), I certainly wouldn't object. > > At least with piglit the warnings are stable, so any CI displaying JUnit > trends will not report spurious regressions. Additionally, the > developers can easily remedy warnings which are intermittent. For this > reason, warnings-as-errors provides a sharper edge with with to catch > regressions in the core piglit test suite (caveat: I'm not sure how > dmesg-warn is handled for tests executed in parallel). > > dEQP has 2 issues that work against using warnings as errors: > > * warnings are intermittent, at least on intel hardware. Minimal > investigation by Chad indicated that the warnings are bogus. > > * developers don't have easy access to upstream, to improve the tests. > > There may be no optimal, consistent warning policy for both test suites. > I'm inclined to turn down the warning noise emitted by dEQP, because it > is external to piglit and can't be easily fixed. Even at Google, dEQP > runners blacklist thousands of tests because they are noisy. > > -Mark If warns from deqp are useless, then those should also be disabled. But that has no bearing on what junit does with warns... -ilia _______________________________________________ Piglit mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/piglit
