On Tue, May 10, 2016 at 8:05 PM, Dylan Baker <[email protected]> wrote: > Quoting Ilia Mirkin (2016-05-10 14:32:59) >> On Tue, May 10, 2016 at 5:32 PM, Dylan Baker <[email protected]> wrote: >> > Quoting Ilia Mirkin (2016-05-10 06:41:33) >> >> Shouldn't the test's result just be included into the "max" no matter >> >> what? >> >> >> > >> > Well, the problem is that a lot of tests were retrofitted with subtests, >> > and now the overall status is bunk, and for new tests we generally don't >> > add an overall result when there's subtests. >> >> It's surprising if the status of a test says fail but piglit-summary >> reports it as pass. In the case where one of the subtests fails but >> the overall status is pass, this would still say fail, no? >> >> What situation are you protecting against by only incorporating the >> overall status on crash? > > I have this gut feeling that says skip or notrun might play havok here. > But I guess I should look test it and see if that's the case. > > The other thing I guess is that there's only about 100 tests (or, c/c++ > files, really) using subtests. So maybe it would just be easier to audit > them to make sure that they do the right thing and just add the overall > result to the max call.
I dunno... I think in at least some cases some "stuff" happens between subtests that can fail. I don't think it's correct to look at a test as a shell for a bunch of disconnected subtests, and thus anything the shell itself does is meaningless. The shell can also fail. Feel free to audit everything :) -ilia _______________________________________________ Piglit mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/piglit
