Re: [webkit-dev] sharing more test references
On Wed, Nov 14, 2012 at 12:01 AM, Dirk Pranke wrote: > > The way ref tests are currently implemented, there is no direct way to > say "use test X as a reference for test Y" (although this is in the > W3C specs for ref tests, which suggests using tags in the test > html). Previous conversations on this topic have concluded with us > thinking that we don't want to give up the convention of having an > "-expected" file next to each test, and I still think this is a good > idea. > > Can I +1 supporting the W3C method for using one reference result to support multiple tests rather than come up with our own way? https://bugs.webkit.org/show_bug.cgi?id=83048 supported the W3C way but had to be rolled out because it didn't work on Windows for reasons no one fully understood because no one was able to test it on a Windows build. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] sharing more test references
On Tue, Nov 13, 2012 at 5:06 PM, Eric Seidel wrote: > > > > > Does seem pretty simple. > > > > > > is even shorter. :) > > I support getting rid of pixel tests. I suspect that some very dumb > scripts could turn large chunks of these existing pixel-tests into > ref-tests. I doubt that those would be the interesting ones though > (where platforms have divergent results). > I've been spending a fair amount of time working on this, actually. I think it's harder than you might think. I'm happy to talk further about it. >From what I can tell, we get most of divergence between platforms from the fact that we render text differently everywhere and we tend to render controls differently everywhere. Most of the time the differences are unrelated to what's actually being tested, unfortunately :(. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] sharing more test references
On Tue, Nov 13, 2012 at 4:59 PM, Darin Adler wrote: > On Nov 13, 2012, at 4:56 PM, Dirk Pranke wrote: > >> Wouldn't the fact that there are a large set of tests with the same result >> be an argument *for* doing the iframe thing? > > The simple hand-coded green square in upper left corner should be simple, > perhaps even simpler than the iframe thing. > >> What is the advantage to having 50 copies of a hand-coded "green square in >> upper left corner" reference test? > > Tests standing alone and being independent, easy to move around, revise, and > understand individually rather than as part of a suite. > Got it. It seems like referencing a well-known result makes things easier to understand, not harder, once you see it at least once, but I imagine it certainly depends on the complexity of the result. E.g., "PASSED" is better than "". Past about four lines it seems like the iframe would win. > I don’t have a strong objection to your iframe technique, but I’d start > simpler and do it only if it’s really needed. > I will also note that there are a large number of tests where we seem to have duplicate results, e.g., dom/html/level1 and dom/xhtml/level1 where the results are basically the same between the two suites, and having the xhtml results just be iframe'd versions of the html one seems like it would make sense. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] sharing more test references
Does seem pretty simple. is even shorter. :) I support getting rid of pixel tests. I suspect that some very dumb scripts could turn large chunks of these existing pixel-tests into ref-tests. I doubt that those would be the interesting ones though (where platforms have divergent results). On Tue, Nov 13, 2012 at 4:59 PM, Darin Adler wrote: > On Nov 13, 2012, at 4:56 PM, Dirk Pranke wrote: > >> Wouldn't the fact that there are a large set of tests with the same result >> be an argument *for* doing the iframe thing? > > The simple hand-coded green square in upper left corner should be simple, > perhaps even simpler than the iframe thing. > >> What is the advantage to having 50 copies of a hand-coded "green square in >> upper left corner" reference test? > > Tests standing alone and being independent, easy to move around, revise, and > understand individually rather than as part of a suite. > > I don’t have a strong objection to your iframe technique, but I’d start > simpler and do it only if it’s really needed. > > -- Darin > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] sharing more test references
On Nov 13, 2012, at 4:56 PM, Dirk Pranke wrote: > Wouldn't the fact that there are a large set of tests with the same result be > an argument *for* doing the iframe thing? The simple hand-coded green square in upper left corner should be simple, perhaps even simpler than the iframe thing. > What is the advantage to having 50 copies of a hand-coded "green square in > upper left corner" reference test? Tests standing alone and being independent, easy to move around, revise, and understand individually rather than as part of a suite. I don’t have a strong objection to your iframe technique, but I’d start simpler and do it only if it’s really needed. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] sharing more test references
On Tue, Nov 13, 2012 at 4:33 PM, Darin Adler wrote: > On Nov 13, 2012, at 4:01 PM, Dirk Pranke wrote: > >> It turns out that we have a reasonably large number of tests that produce >> the exact same pixel results. On chromium-mac on 10.8, for example, there >> are 2048 tests that share a result with some other test. 50 of them, for >> example, draw a green square in the upper left corner of the page (e.g., for >> svg/custom/root-element.html). > > It seems to me that when we find a pattern like this, we should create a > hand-coded reference test result. The fact that so many tests produce the > same pixels means that each reference can be used to move a lot of tests from > the pixel test to the reference test category. > > I don’t think we need to do that thing you said, as long as there > are large sets of tests with the same result, since if the payoff for each > reference is big enough, it should be affordable to hand-write the reference > file. > I don't think I'm understanding you. Wouldn't the fact that there are a large set of tests with the same result be an argument *for* doing the iframe thing? What is the advantage to having 50 copies of a hand-coded "green square in upper left corner" reference test? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] sharing more test references
On Nov 13, 2012, at 4:01 PM, Dirk Pranke wrote: > It turns out that we have a reasonably large number of tests that produce the > exact same pixel results. On chromium-mac on 10.8, for example, there are > 2048 tests that share a result with some other test. 50 of them, for example, > draw a green square in the upper left corner of the page (e.g., for > svg/custom/root-element.html). It seems to me that when we find a pattern like this, we should create a hand-coded reference test result. The fact that so many tests produce the same pixels means that each reference can be used to move a lot of tests from the pixel test to the reference test category. I don’t think we need to do that thing you said, as long as there are large sets of tests with the same result, since if the payoff for each reference is big enough, it should be affordable to hand-write the reference file. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] sharing more test references
Hi all, Currently, we have ~8000 pixel tests in the tree, and ~800 reference tests. I would like to make that first number a lot smaller. It turns out that we have a reasonably large number of tests that produce the exact same pixel results. On chromium-mac on 10.8, for example, there are 2048 tests that share a result with some other test. 50 of them, for example, draw a green square in the upper left corner of the page (e.g., for svg/custom/root-element.html). The way ref tests are currently implemented, there is no direct way to say "use test X as a reference for test Y" (although this is in the W3C specs for ref tests, which suggests using tags in the test html). Previous conversations on this topic have concluded with us thinking that we don't want to give up the convention of having an "-expected" file next to each test, and I still think this is a good idea. However, we can probably have our cake and eat it too in many cases by simply doing something like creating an -expected.html file that contains solely "". So, I would like to : 1) document that as an accepted pattern somewhere 2) start building up a directory of shared references, e.g., "LayoutTests/references/green-square.html" 3) start converting existing pixel tests to use these. Anyone think this won't, work, otherwise object, or have better ideas? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
