Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
Correction: the person from Opera who tried to add reference files is Geoffrey Sneddon as Anne pointed out. +Ms2ger, +ehsan from Mozilla, and +jgraham from Opera. Also note that both Ehsan and James have pointed out that my concern that a test and the corresponding ref will match for wrong reasons (e.g. the entire page is blank) could be be mitigated by having good noref (expected-mismatch in WebKit convention). James also noted that they haven't had such problems in practice. Given their experience and knowledge working with reftest, I change my position and will support writing our own reference files as well. But I'm still interested in moving forward with upstreaming our reference files to the W3C repository and making clear distinction between ones we import and ones we write ourselves. - Ryosuke On Thu, Apr 12, 2012 at 1:45 PM, Ryosuke Niwa wrote: > I've done more digging and talked with folks from Mozilla and Opera about > this topic. Here's a summary. > > Daniel Glazman from Opera has tried adding reference files for W3C's: > http://lists.w3.org/Archives/Public/public-css-testsuite/2010Sep/0030.html > > However, some people in the CSS working group opposed to make this change > in last minute because it would have delayed REC, and other parties did not > support the proposal enough to convince the WG as a whole to import the > reference files. > (See the member only mailing list around September 2010 if you have > access). > > Ms2ger also has more reference files available at > https://bitbucket.org/ms2ger/css-tests > > On the other hand, CSS3 test suite consists entirely of reference tests > and testharness.js tests (js text based tests). Given that and TabAtkins > and other folks at the CSS WG now prefer reference tests over pixel tests > in addition to the fact CSS2.1 has already been REC'ed, it might be > possible for us to convince the WG to convert the existing tests now. > > In my opinion, we should definitely try to make changes upstream because > then virtually all browser vendors will benefit. > > - Ryosuke > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
I've done more digging and talked with folks from Mozilla and Opera about this topic. Here's a summary. Daniel Glazman from Opera has tried adding reference files for W3C's: http://lists.w3.org/Archives/Public/public-css-testsuite/2010Sep/0030.html However, some people in the CSS working group opposed to make this change in last minute because it would have delayed REC, and other parties did not support the proposal enough to convince the WG as a whole to import the reference files. (See the member only mailing list around September 2010 if you have access). Ms2ger also has more reference files available at https://bitbucket.org/ms2ger/css-tests On the other hand, CSS3 test suite consists entirely of reference tests and testharness.js tests (js text based tests). Given that and TabAtkins and other folks at the CSS WG now prefer reference tests over pixel tests in addition to the fact CSS2.1 has already been REC'ed, it might be possible for us to convince the WG to convert the existing tests now. In my opinion, we should definitely try to make changes upstream because then virtually all browser vendors will benefit. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
On Thu, Apr 12, 2012 at 12:50 PM, Robert Hogan wrote: > On Thursday 12 April 2012 00:58:47 Ryosuke Niwa wrote: > > On Wed, Apr 11, 2012 at 4:45 PM, Ojan Vafai wrote: > > > I agree with the sentiment that we should be upstreaming these to the > > > W3C, but I don't see why we would require upstreaming them first > > > instead of committing them locally and then upstreaming them. > > > > How do we know whether a reference file came from W3C repository or not. > > (Maybe by the fact it's named *-expected.html?) > > Yes, that is it exactly. Presumably that's by design - as NRWT currently > won't use anything like *-ref.htm unless it's in a manifest. > > > > > Also, there are directories with reftest.list but without reference > > files for some tests. The last time I checked, you were opposed to > > having bot reftest.list and *-expected.html / *-expected-mismatch.html > > files. Have you changed your opinion on this? > > It would be good if this was allowed. It would allow us to import the > reftest.list from the CSS test suite while keeping our own reference > results in there for tests in the suite that don't have them yet. > I'm not opposed to changing this if it gives us a path forward. I still firmly believe we should not use manifest files for non-imported test suites, but we don't need to enforce that with the tooling if it makes things more complicated. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
On Thursday 12 April 2012 00:58:47 Ryosuke Niwa wrote: > On Wed, Apr 11, 2012 at 4:45 PM, Ojan Vafai wrote: > > I agree with the sentiment that we should be upstreaming these to the > > W3C, but I don't see why we would require upstreaming them first > > instead of committing them locally and then upstreaming them. > > How do we know whether a reference file came from W3C repository or not. > (Maybe by the fact it's named *-expected.html?) Yes, that is it exactly. Presumably that's by design - as NRWT currently won't use anything like *-ref.htm unless it's in a manifest. > > Also, there are directories with reftest.list but without reference > files for some tests. The last time I checked, you were opposed to > having bot reftest.list and *-expected.html / *-expected-mismatch.html > files. Have you changed your opinion on this? It would be good if this was allowed. It would allow us to import the reftest.list from the CSS test suite while keeping our own reference results in there for tests in the suite that don't have them yet. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
On Wed, Apr 11, 2012 at 4:45 PM, Ojan Vafai wrote: > > I agree with the sentiment that we should be upstreaming these to the W3C, > but I don't see why we would require upstreaming them first instead of > committing them locally and then upstreaming them. > How do we know whether a reference file came from W3C repository or not. (Maybe by the fact it's named *-expected.html?) Also, there are directories with reftest.list but without reference files for some tests. The last time I checked, you were opposed to having bot reftest.list and *-expected.html / *-expected-mismatch.html files. Have you changed your opinion on this? - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
On Wed, Apr 11, 2012 at 3:48 PM, Dirk Pranke wrote: > On Wed, Apr 11, 2012 at 3:46 PM, Dirk Pranke wrote: > > On Fri, Mar 9, 2012 at 2:49 PM, Sam Weinig wrote: > >> > >> On Mar 7, 2012, at 4:41 PM, Ojan Vafai wrote: > >> > >> I just did a first pass a greening the Chromium Lion > >> bot: http://trac.webkit.org/changeset/110096. Of these hundreds of > tests, > >> ~99% of them are perfect candidates for being reftests (e.g. they > contain > >> one line of text and a solid box or two under the text), but most of > them > >> are in the CSS imported test suites. > >> > >> Is it kosher to convert them to reftests or should we leave pixel tests > from > >> imported test suites alone? > >> > >> > >> If we want to make these ref tests, it probably makes more sense to do > that > >> work with the CSS WG, so that they can be part of the standard test > suite. > >> Until then, I think we should keep them regular pixel tests. > >> > > > > Note that this thread (to resurrect it just-after-easter because it's > > timely) is directly relevant to the note I just sent out about > > importing test suites. > > > > I, at least, would like some clarification ... if we are importing > > tests that have no accompanying "expected result" (and are expected to > > be inspected manually for correctness), is it acceptable to write > > reference html for the tests, or do they have to be imported as pixel > > tests? > > > > Put differently, we either need to add an -expected html or an > -expected png. Does it have to be the latter? > I strongly prefer adding -expected.html files. While there is a chance the reference may not cover all the code paths the original test intended, I believe our overall test coverage will be better with more reftests and fewer pixel tests, not to mention our project-wide sanity from spending less time doing expectations file management. Pixel tests accidentally let bugs through all the time because not all ports run them and because it's often hard to tell if a new result is correct. I agree with the sentiment that we should be upstreaming these to the W3C, but I don't see why we would require upstreaming them first instead of committing them locally and then upstreaming them. If there are people willing to do the work, lets allow them do it either way. > > I personally think it's acceptable, but I understand that there might > > be a difference of opinion. > > > > https://bugs.webkit.org/show_bug.cgi?id=72987 is an example of this. > > > > -- Dirk > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
On Wed, Apr 11, 2012 at 3:46 PM, Dirk Pranke wrote: > On Fri, Mar 9, 2012 at 2:49 PM, Sam Weinig wrote: >> >> On Mar 7, 2012, at 4:41 PM, Ojan Vafai wrote: >> >> I just did a first pass a greening the Chromium Lion >> bot: http://trac.webkit.org/changeset/110096. Of these hundreds of tests, >> ~99% of them are perfect candidates for being reftests (e.g. they contain >> one line of text and a solid box or two under the text), but most of them >> are in the CSS imported test suites. >> >> Is it kosher to convert them to reftests or should we leave pixel tests from >> imported test suites alone? >> >> >> If we want to make these ref tests, it probably makes more sense to do that >> work with the CSS WG, so that they can be part of the standard test suite. >> Until then, I think we should keep them regular pixel tests. >> > > Note that this thread (to resurrect it just-after-easter because it's > timely) is directly relevant to the note I just sent out about > importing test suites. > > I, at least, would like some clarification ... if we are importing > tests that have no accompanying "expected result" (and are expected to > be inspected manually for correctness), is it acceptable to write > reference html for the tests, or do they have to be imported as pixel > tests? > Put differently, we either need to add an -expected html or an -expected png. Does it have to be the latter? > I personally think it's acceptable, but I understand that there might > be a difference of opinion. > > https://bugs.webkit.org/show_bug.cgi?id=72987 is an example of this. > > -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
On Fri, Mar 9, 2012 at 2:49 PM, Sam Weinig wrote: > > On Mar 7, 2012, at 4:41 PM, Ojan Vafai wrote: > > I just did a first pass a greening the Chromium Lion > bot: http://trac.webkit.org/changeset/110096. Of these hundreds of tests, > ~99% of them are perfect candidates for being reftests (e.g. they contain > one line of text and a solid box or two under the text), but most of them > are in the CSS imported test suites. > > Is it kosher to convert them to reftests or should we leave pixel tests from > imported test suites alone? > > > If we want to make these ref tests, it probably makes more sense to do that > work with the CSS WG, so that they can be part of the standard test suite. > Until then, I think we should keep them regular pixel tests. > Note that this thread (to resurrect it just-after-easter because it's timely) is directly relevant to the note I just sent out about importing test suites. I, at least, would like some clarification ... if we are importing tests that have no accompanying "expected result" (and are expected to be inspected manually for correctness), is it acceptable to write reference html for the tests, or do they have to be imported as pixel tests? I personally think it's acceptable, but I understand that there might be a difference of opinion. https://bugs.webkit.org/show_bug.cgi?id=72987 is an example of this. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
On Mar 7, 2012, at 4:41 PM, Ojan Vafai wrote: > I just did a first pass a greening the Chromium Lion bot: > http://trac.webkit.org/changeset/110096. Of these hundreds of tests, ~99% of > them are perfect candidates for being reftests (e.g. they contain one line of > text and a solid box or two under the text), but most of them are in the CSS > imported test suites. > > Is it kosher to convert them to reftests or should we leave pixel tests from > imported test suites alone? If we want to make these ref tests, it probably makes more sense to do that work with the CSS WG, so that they can be part of the standard test suite. Until then, I think we should keep them regular pixel tests. -Sam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
On 08.03.12 01:57, Levi Weintraub wrote: On Wed, Mar 7, 2012 at 4:50 PM, Ryosuke Niwa mailto:rn...@webkit.org>> wrote: On Wed, Mar 7, 2012 at 4:44 PM, Darin Fisher mailto:da...@chromium.org>> wrote: Hrm, if the test expectations are customized already for different ports of WebKit, then why not support replacing a PNG file with a HTML file that is intended to generate exactly the same result? How does this impair our ability to update the tests? (I realize that our current reftest system may not work like this. I'm not familiar with the details of how it works in fact, but it seems like it could be as simple as having an expected result that is a HTML file instead of a PNG file.) How do we know that we are testing what the test is intending to test after the conversion? e.g. it's possible to create a reference file that fails to catch certain bugs. This sums up my worry as well. I can imagine a bug causing a CSS test and its reference to fail in the same way, masking the failure. What if the reference is one line of text + image that represents the expected result? That way ports can share the associated png. tor arne ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
I too am mildly concerned about references not being sufficiently independent of the tests, which is why I hoped we could get the WG in the business of reviewing references along with tests. However, another possibility is looking at what Mozilla uses for reference for these tests, since those would presumably have a track record of being good enough. Regards, Maciej On Mar 7, 2012, at 4:58 PM, Ojan Vafai wrote: > On Wed, Mar 7, 2012 at 4:50 PM, Ryosuke Niwa wrote: > On Wed, Mar 7, 2012 at 4:44 PM, Darin Fisher wrote: > Hrm, if the test expectations are customized already for different ports of > WebKit, then why not support replacing a PNG file with a HTML file that is > intended to generate exactly the same result? How does this impair our > ability to update the tests? > > (I realize that our current reftest system may not work like this. I'm not > familiar with the details of how it works in fact, but it seems like it could > be as simple as having an expected result that is a HTML file instead of a > PNG file.) > > How do we know that we are testing what the test is intending to test after > the conversion? e.g. it's possible to create a reference file that fails to > catch certain bugs. > > It's not obvious to me how one would figure out how many reference files are > needed for a given test to make sure we're not making the test more > permissible than the author intended it to be. > > I'm not suggesting that we convert 100% of these tests to reftests, but a > very very large percentage of them can easily be verified to be testing the > correct thing and would only need a single reference file (it's just a line > of text with a colored box in it after all). > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
On Wed, Mar 7, 2012 at 4:50 PM, Ryosuke Niwa wrote: > On Wed, Mar 7, 2012 at 4:44 PM, Darin Fisher wrote: > >> Hrm, if the test expectations are customized already for different ports >> of WebKit, then why not support replacing a PNG file with a HTML file that >> is intended to generate exactly the same result? How does this impair our >> ability to update the tests? >> >> (I realize that our current reftest system may not work like this. I'm >> not familiar with the details of how it works in fact, but it seems like it >> could be as simple as having an expected result that is a HTML file instead >> of a PNG file.) >> > > How do we know that we are testing what the test is intending to test > after the conversion? e.g. it's possible to create a reference file that > fails to catch certain bugs. > > It's not obvious to me how one would figure out how many reference files > are needed for a given test to make sure we're not making the test more > permissible than the author intended it to be. > I'm not suggesting that we convert 100% of these tests to reftests, but a very very large percentage of them can easily be verified to be testing the correct thing and would only need a single reference file (it's just a line of text with a colored box in it after all). ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
On Wed, Mar 7, 2012 at 4:50 PM, Ryosuke Niwa wrote: > On Wed, Mar 7, 2012 at 4:44 PM, Darin Fisher wrote: > >> Hrm, if the test expectations are customized already for different ports >> of WebKit, then why not support replacing a PNG file with a HTML file that >> is intended to generate exactly the same result? How does this impair our >> ability to update the tests? >> >> (I realize that our current reftest system may not work like this. I'm >> not familiar with the details of how it works in fact, but it seems like it >> could be as simple as having an expected result that is a HTML file instead >> of a PNG file.) >> > > How do we know that we are testing what the test is intending to test > after the conversion? e.g. it's possible to create a reference file that > fails to catch certain bugs. > This sums up my worry as well. I can imagine a bug causing a CSS test and its reference to fail in the same way, masking the failure. > > It's not obvious to me how one would figure out how many reference files > are needed for a given test to make sure we're not making the test more > permissible than the author intended it to be. > > - Ryosuke > > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
On Wed, Mar 7, 2012 at 4:44 PM, Darin Fisher wrote: > Hrm, if the test expectations are customized already for different ports > of WebKit, then why not support replacing a PNG file with a HTML file that > is intended to generate exactly the same result? How does this impair our > ability to update the tests? > > (I realize that our current reftest system may not work like this. I'm > not familiar with the details of how it works in fact, but it seems like it > could be as simple as having an expected result that is a HTML file instead > of a PNG file.) > How do we know that we are testing what the test is intending to test after the conversion? e.g. it's possible to create a reference file that fails to catch certain bugs. It's not obvious to me how one would figure out how many reference files are needed for a given test to make sure we're not making the test more permissible than the author intended it to be. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
That hadn't occurred to me. You're right, we wouldn't actually modify the test itself. We would just replace the -expected.txt/png with a -expected.html file. Maciej, does that change your opinion on this? On Wed, Mar 7, 2012 at 4:44 PM, Darin Fisher wrote: > Hrm, if the test expectations are customized already for different ports > of WebKit, then why not support replacing a PNG file with a HTML file that > is intended to generate exactly the same result? How does this impair our > ability to update the tests? > > (I realize that our current reftest system may not work like this. I'm > not familiar with the details of how it works in fact, but it seems like it > could be as simple as having an expected result that is a HTML file instead > of a PNG file.) > > -Darin > > > On Wed, Mar 7, 2012 at 4:10 PM, Maciej Stachowiak wrote: > >> >> I'd prefer we not modify imported test suites. That will just make it >> more confusing to update. Perhaps future CSS test suites will be changed to >> a reftest model. >> >> Regards, >> Maciej >> >> On Mar 7, 2012, at 1:41 PM, Ojan Vafai wrote: >> >> I just did a first pass a greening the Chromium Lion bot: >> http://trac.webkit.org/changeset/110096. Of these hundreds of tests, >> ~99% of them are perfect candidates for being reftests (e.g. they contain >> one line of text and a solid box or two under the text), but most of them >> are in the CSS imported test suites. >> >> Is it kosher to convert them to reftests or should we leave pixel tests >> from imported test suites alone? >> ___ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >> >> >> >> ___ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >> >> > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
Hrm, if the test expectations are customized already for different ports of WebKit, then why not support replacing a PNG file with a HTML file that is intended to generate exactly the same result? How does this impair our ability to update the tests? (I realize that our current reftest system may not work like this. I'm not familiar with the details of how it works in fact, but it seems like it could be as simple as having an expected result that is a HTML file instead of a PNG file.) -Darin On Wed, Mar 7, 2012 at 4:10 PM, Maciej Stachowiak wrote: > > I'd prefer we not modify imported test suites. That will just make it more > confusing to update. Perhaps future CSS test suites will be changed to a > reftest model. > > Regards, > Maciej > > On Mar 7, 2012, at 1:41 PM, Ojan Vafai wrote: > > I just did a first pass a greening the Chromium Lion bot: > http://trac.webkit.org/changeset/110096. Of these hundreds of tests, > ~99% of them are perfect candidates for being reftests (e.g. they contain > one line of text and a solid box or two under the text), but most of them > are in the CSS imported test suites. > > Is it kosher to convert them to reftests or should we leave pixel tests > from imported test suites alone? > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > > > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
I'd prefer we not modify imported test suites. That will just make it more confusing to update. Perhaps future CSS test suites will be changed to a reftest model. Regards, Maciej On Mar 7, 2012, at 1:41 PM, Ojan Vafai wrote: > I just did a first pass a greening the Chromium Lion bot: > http://trac.webkit.org/changeset/110096. Of these hundreds of tests, ~99% of > them are perfect candidates for being reftests (e.g. they contain one line of > text and a solid box or two under the text), but most of them are in the CSS > imported test suites. > > Is it kosher to convert them to reftests or should we leave pixel tests from > imported test suites alone? > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
I just did a first pass a greening the Chromium Lion bot: http://trac.webkit.org/changeset/110096. Of these hundreds of tests, ~99% of them are perfect candidates for being reftests (e.g. they contain one line of text and a solid box or two under the text), but most of them are in the CSS imported test suites. Is it kosher to convert them to reftests or should we leave pixel tests from imported test suites alone? ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
