On Wed, May 23, 2012 at 2:59 PM, Jacob Goldstein <jac...@adobe.com> wrote: > As a side note to this discussion, there is talk in the W3C community > regarding their test approval process. At the recent working group > meetings in Germany the idea was floated to simply approve all tests that > are currently waiting for review (and doing this going forward, e.g. no > longer requiring approval upon submission). > > Apparently, not enough people are reviewing tests, and as a result, tests > can linger for months (or longer) before ever being looked at. Once > browser vendors start implementing features, associated tests will be > revisited for that area. > > This has by no means been decided, but something we should consider if it > ultimately does come to fruition. If W3C tests are no longer > reviewed, this would mean importing tests without any knowledge of their > accuracy - though that will still allow us to catch regressions.
Ick :(. I suppose if the test suites are mostly coming from established browser vendors or other trusted sources (who have been running them on their own browsers for a while) this isn't so bad, but it would be nice to see the suites get vetted somehow prior to them getting blessed by the w3c. Levels of approval, or something? > Dirk, I've updated the process on the Wiki page with the feedback you > provided. I hope I captured it all - I included everything that appeared > to have agreement between you and Ryosuke. Feel free to modify it > directly if I missed anything, or let me know and I can refine it further. It looks much closer to what I'd like, thanks! I'll probably modify it a bit and/or send you some suggestions offlist for some minor things. The one major issue I still have that I wouldn't want to just add/sneak into the wiki, is that I still don't see much of a discussion for how we identify duplicate tests. The mindset I would prefer would be that, for a given test suite, the person or persons importing the test suite should have to start by identifying which existing tests we have for similar functionality, and produce lists of which existing tests look like they are duplicates and should be remvoed, which tests we have that should be submitted back to the w3c for future inclusion, and which tests are clearly webkit-specific and should be kept but identified as clearly webkit-specific. For example, if we import a "flexbox" test suite from css3, we should default to removing fast/flexbox (and possibly also ietestcenter/css3/flexbox). [ This is a purely hypothetical example (I know nothing about these test suites). I wish to re-emphasize that we will have to look at these things on a case-by-case basis. ] I have no idea how much work we'd be asking the importer to undertake, but that's kinda the point. If we don't ask the importer to de-dup things, who is realistically going to de-dup them later? What do others thinks, is this too much to ask? If it is, how do we avoid incurring further "test debt" over time? -- Dirk _______________________________________________ webkit-dev mailing list email@example.com http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev