Apparently I was somewhat unclear. Let me restate. We have the following mechanisms available when a test fails:
1) Check in a new -expected.* file. 2) Modify the test. 3) Modify a TestExpectations file. 4) Add the test to a Skipped file. 5) Remove the test entirely. I have no problem with (1) unless it is intended to mark the test as expected-to-fail-but-not-crash. I agree that using -expected.* to accomplish what TestExpectations accomplishes is not valuable, but I further believe that even TestExpectations is not valuable. I broadly prefer (2) whenever possible. I believe that (3) and (4) are redundant, and I don't buy the value of (3). I don't like (5) but we should probably do more of it for tests that have a chronically low signal-to-noise ratio. You're proposing a new mechanism. I'm arguing that given the sheer number of tests, and the overheads associated with maintaining them, (4) is the broadly more productive strategy in terms of bugs-fixed/person-hours. And, increasing the number of mechanisms for dealing with tests by 20% is likely to reduce overall productivity rather than helping anyone. -F On Aug 15, 2012, at 12:40 PM, Dirk Pranke <dpra...@chromium.org> wrote: > On Wed, Aug 15, 2012 at 12:27 PM, Filip Pizlo <fpi...@apple.com> wrote: >> This sounds like it's adding even more complication to an already >> complicated system. > > In some ways, yes. In other ways, perhaps it will allow us to simplify > things; e.g., if we are checking in failing tests, there is much less > of a need for multiple failure keywords in the TestExpectations file > (so perhaps we can simplify them back to something closer to Skipped > files). > >> Given how many tests we currently have, I also don't buy that continuing to >> run a test that is already known to fail provides much benefit. > > I'm not sure I understand your feedback here? It's common practice (on > all the ports as far as I know today) to rebaseline tests that are > currently failing so that they fail differently. Of course, we also > skip some tests while they are failing as well. However, common > objections to skipping tests are that we can lose coverage for a > feature and/or miss when a test starts failing worse (e.g. crashing)? > And of course, a test might start passing again, but if we're skipping > it we wouldn't know that ... > >> So, I'd rather not continue to institutionalize this notion that we should >> have loads of incomprehensible machinery to reason about tests that have >> already given us all the information they were meant to give (i.e. they >> failed, end of story). > > Are you suggesting that, rather than checking in new baselines at all > or having lots of logic to manage different kinds of failures, we > should just let failing tests fail (and keep the tree red) until a > change is either reverted or the test is fixed? > > -- Dirk _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
