I understand, all valid points. Don't people currently fix tests that are failing and that they didn't author? If so, isn't that similar as generating a new ref file in that you first have to understand what is being tested? There would be the old pixel file to go from, so in many cases this may not be too hard.
On the other hand, having the test author create the ref file is definitely preferable. What would we do in the case where a test author is no longer available? From: Ryosuke Niwa <rn...@webkit.org<mailto:rn...@webkit.org>> Date: Wed, 11 Apr 2012 16:00:44 -0700 To: Jacob Goldstein <jac...@adobe.com<mailto:jac...@adobe.com>> Cc: Dirk Pranke <dpra...@chromium.org<mailto:dpra...@chromium.org>>, Robert Hogan <li...@roberthogan.net<mailto:li...@roberthogan.net>>, "webkit-dev@lists.webkit.org<mailto:webkit-dev@lists.webkit.org>" <webkit-dev@lists.webkit.org<mailto:webkit-dev@lists.webkit.org>> Subject: Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite) On Wed, Apr 11, 2012 at 3:52 PM, Jacob Goldstein <jac...@adobe.com<mailto:jac...@adobe.com>> wrote: +1 on not introducing new pixel tests and allowing someone other than the test author to create the -expected file. We may also be able to streamline some of this process by implementing some helper scripts. Ultimately, someone will still have to review new files manually, but scripts should be able to speed up the process. -1 on that. As I said on other threads about this topic, determining whether a reference file adequately detect all bugs a test is intended to test is hard, and losing the test coverage at the cost of lowering maintenance cost is not necessary a good thing. Also, adding a reference file would mean that either we're adding -ref.html / -noref.html files or modifying reftest.list. If doing the former, then we can't use this approach in any directory where we use reftest.list at the moment because we explicitly prohibit mixing naming convention and reftest.list. Modifying reftest.list is essentially modifying the test suite, and it seems like there is a consensus that we don't want to do it. - Ryosuke
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
