[webkit-dev] Importing W3C tests
Hi, I improved a bit W3C test importer lately. First, I'd like to remind people importing tests (great) without the importer (copy/paste a directory e.g.) to update LayoutTests/imported/w3c/resources/ImportExpectations. That allows me to refresh those tests when doing a full resync more quickly. Otherwise, I need to know why some tests were needlessly removed when doing a full resync, Note that, as part of the full resync, I am removing tests that are removed in WPT repository. I also made two improvements to W3C test importer. Resource files in WPT repository are not always in 'resources' folder. Test importer is generating a list of resource files in LayoutTests/imported/w3c/resources/resource-files.json All listed files there will be skipped. If you import tests without the importer, it might be better to update that file than adding Skip TestExpectations. Slow tests in WPT repository are identified with a meta element. Test importer is generating a list of slow tests in LayoutTests/tests-options.json If you import tests without the importer, it might be better to update that file than adding Slow TestExpectations. If you encounter any issue there, please let me know. Thanks y ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] Importing W3C tests for editing
Hi all, Some time ago there was a conversation about importing W3C tests into LayoutTests for WebKit (https://lists.webkit.org/pipermail/webkit-dev/2013-November/025876.html). This conversation was paused in November with conclusion, that it may be a good idea. A brief research showed, that number of editing tests available in https://github.com/w3c/web-platform-tests is very small. However, there are a lot of editing tests placed directly in W3C HTML Editing APIs specification (https://dvcs.w3.org/hg/editing/raw-file/tip/editing.html#conformance-tests) . It may be a good idea to import those tests instead. There are 90412 tests covering all aspects of W3C specification. Those tests can be divided into test cases containing several tests with structure similar to this: PASS [[stylewithcss,true]] foo[bar]baz: execCommand(stylewithcss, false, true) return value PASS [[stylewithcss,true]] foo[bar]baz checks for modifications to non-editable content PASS [[stylewithcss,true]] foo[bar]baz compare innerHTML PASS [[stylewithcss,true]] foo[bar]baz queryCommandIndeterm(stylewithcss) before PASS [[stylewithcss,true]] foo[bar]baz queryCommandState(stylewithcss) before FAIL [[stylewithcss,true]] foo[bar]baz queryCommandValue(stylewithcss) before assert_equals: Wrong result returned expected but got false PASS [[stylewithcss,true]] foo[bar]baz queryCommandIndeterm(stylewithcss) after PASS [[stylewithcss,true]] foo[bar]baz queryCommandState(stylewithcss) after FAIL [[stylewithcss,true]] foo[bar]baz queryCommandValue(stylewithcss) after assert_equals: Wrong result returned expected but got true As a result, we would have 5536 new test files divided into 36 categories: - backcolor: 70 - bold: 210 - createlink: 48 - defaultparagraphseparator: 11 - delete: 494 - fontname: 141 - fontsize: 156 - forecolor: 147 - formatblock: 362 - forwarddelete: 471 - hilitecolor: 82 - indent: 143 - inserthorizontalrule: 117 - inserthtml: 104 - insertimage: 70 - insertlinebreak: 166 - insertorderedlist: 135 - insertparagraph: 320 - inserttext: 238 - insertunorderedlist: 148 - italic: 143 - justifycenter: 294 - justifyfull: 233 - justifyleft: 149 - justifyright: 233 - outdent: 175 - quasit: 1 - removeformat: 143 - selectall: 1 - strikethrough: 145 - stylewithcss: 17 - subscript: 85 - superscript: 87 - underline: 145 - unlink: 35 - usecss: 17 Furthermore, newly created tests has been run and results showed, that 1120/5536 tests run as expected (all sub-tests returned PASS) and 4416/5536 did not. There are 4 WebProcess crushes; 12 tests have some stderr output. Rest of failures are because of one or more sub-tests failing. Running all 5536 tests takes 32 min 51.978 sec. What do you think about it? Is it worth paying some time for importing those tests and then aligning WebKit with W3C specification? Best regards, Lukasz Bialek ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for editing
On 05/02/2014 13:16 , Lukasz Bialek wrote: What do you think about it? Is it worth paying some time for importing those tests and then aligning WebKit with W3C specification? Depending on the amount of energy you're willing to dedicate to this, the ideal option could well be to import those tests into the web-platform-tests repo and then into WebKit. Note that I've heard reports in the past that some browsers tried to align with the spec but couldn't. The problem is that many of the discrepancies are difficult to feature-detect so that an awful lot of rich text editing code out there branches off UA identification. As a result, fixing bugs that bring browsers more in line with one another breaks the Web. I'd love to hear indication that this is less of a SNAFU than previously reported. If not we might be stuck and in need of a more radical solution. -- Robin Berjon - http://berjon.com/ - @robinberjon ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for editing
We are still far from importing into WebKit the whole test set from web-platform-tests repo. Improving the situation here would already be a great step forward. It would also be good to regularly run the whole web-platform-tests repo test set so as to get statistics in terms of feature alignment and test import completeness/synchronization. See https://bugs.webkit.org/show_bug.cgi?id=127095#c9 for rough notes on that. I hope to work on that once the web platform test server will be integrated into WebKit. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
I created a bug to track this (serve imported w3c tests using wptserve: https://bugs.webkit.org/show_bug.cgi?id=127094). I also plan to work on merging Blink patches to allow checking testharness-based tests without the use of any -expected file: https://bugs.webkit.org/show_bug.cgi?id=127095. 2014/1/6 Dirk Pranke dpra...@chromium.org Ryosuke and I discussed this a bit over IRC. Ryosuke's main concern was that supporting multiple document roots adds a fair amount of complexity to NRWT. Conceptually, it's probably easier to add support to run the W3C's new server (known as wptserve) and then maybe use it for *all* imported tests from the W3C. If it turns out that wptserve is too slow, and you would prefer to run only some of the directories over http (i.e., the 10k+ CSS tests don't need http), you'll probably need to modify how wptserve is run (and NRWT) as well, anyway, so the patch would be different. Ryosuke, please let me know if I've misstated your thinking at all. -- Dirk On Mon, Jan 6, 2014 at 11:23 AM, Ryosuke Niwa rn...@webkit.org wrote: I don't think we should do this given that the python server has been added to W3C testing harness, and they're gonna convert all existing tests to use that instead: http://lists.w3.org/Archives/Public/public-test-infra/2014JanMar/.html We should simply wait for that effort to take place and add a support for the python server instead. - R. Niwa On Fri, Dec 6, 2013 at 12:25 AM, youenn fablet youe...@gmail.com wrote: As long as the newly imported tests use relative URLs, alias may be used as a workaround. I will give it a try. Bug entry is at https://bugs.webkit.org/show_bug.cgi?id=125339 Any further help appreciated, Youenn 2013/12/6 Darin Adler da...@apple.com If that's really ends up being super hard we can always put yet another third-party or imported directory inside the http directory as previously suggested. it's annoying to have three different places for imported tests and code, but not something I want to hold us up for a long time. -- Darin Sent from my iPhone On Dec 5, 2013, at 5:51 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Dec 4, 2013 at 11:19 AM, Darin Adler da...@apple.com wrote: On Dec 4, 2013, at 6:48 AM, youenn fablet youe...@gmail.com wrote: I am planning to add some XHR tests from https://github.com/w3c/web-platform-tests. My initial plan was to add them in a subdirectory of LayoutTests/http/tests/w3c. If adding them into LayoutTests/imported/w3c, that would probably require updating the test scripts to start/stop the HTTP test server for that particular sub-folder. Any preference? I’d prefer LayoutTests/imported/w3c. Although I’m not so happy about the different terminology we are using for “imported” vs. “ThirdParty”, which seems like the same concept at the top level of the directory structure. One trickiness to it is that we don't currently run any HTTP test in parallel and the document root of the HTTP server is set to LayoutTests/http/tests so we might need to modify that or restart the HTTP server whenever we're running HTTP tests outside of LayoutTests/http. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
I don't think we should do this given that the python server has been added to W3C testing harness, and they're gonna convert all existing tests to use that instead: http://lists.w3.org/Archives/Public/public-test-infra/2014JanMar/.html We should simply wait for that effort to take place and add a support for the python server instead. - R. Niwa On Fri, Dec 6, 2013 at 12:25 AM, youenn fablet youe...@gmail.com wrote: As long as the newly imported tests use relative URLs, alias may be used as a workaround. I will give it a try. Bug entry is at https://bugs.webkit.org/show_bug.cgi?id=125339 Any further help appreciated, Youenn 2013/12/6 Darin Adler da...@apple.com If that's really ends up being super hard we can always put yet another third-party or imported directory inside the http directory as previously suggested. it's annoying to have three different places for imported tests and code, but not something I want to hold us up for a long time. -- Darin Sent from my iPhone On Dec 5, 2013, at 5:51 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Dec 4, 2013 at 11:19 AM, Darin Adler da...@apple.com wrote: On Dec 4, 2013, at 6:48 AM, youenn fablet youe...@gmail.com wrote: I am planning to add some XHR tests from https://github.com/w3c/web-platform-tests. My initial plan was to add them in a subdirectory of LayoutTests/http/tests/w3c. If adding them into LayoutTests/imported/w3c, that would probably require updating the test scripts to start/stop the HTTP test server for that particular sub-folder. Any preference? I’d prefer LayoutTests/imported/w3c. Although I’m not so happy about the different terminology we are using for “imported” vs. “ThirdParty”, which seems like the same concept at the top level of the directory structure. One trickiness to it is that we don't currently run any HTTP test in parallel and the document root of the HTTP server is set to LayoutTests/http/tests so we might need to modify that or restart the HTTP server whenever we're running HTTP tests outside of LayoutTests/http. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
On 06/01/2014 20:23 , Ryosuke Niwa wrote: I don't think we should do this given that the python server has been added to W3C testing harness, and they're gonna convert all existing tests to use that instead: http://lists.w3.org/Archives/Public/public-test-infra/2014JanMar/.html We should simply wait for that effort to take place and add a support for the python server instead. I'm happy to report that the conversion to a python server as well as of all the server-side dynamic content that it needs to run is now complete and has been merged to master as of today. No need to wait! -- Robin Berjon - http://berjon.com/ - @robinberjon ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
Ryosuke and I discussed this a bit over IRC. Ryosuke's main concern was that supporting multiple document roots adds a fair amount of complexity to NRWT. Conceptually, it's probably easier to add support to run the W3C's new server (known as wptserve) and then maybe use it for *all* imported tests from the W3C. If it turns out that wptserve is too slow, and you would prefer to run only some of the directories over http (i.e., the 10k+ CSS tests don't need http), you'll probably need to modify how wptserve is run (and NRWT) as well, anyway, so the patch would be different. Ryosuke, please let me know if I've misstated your thinking at all. -- Dirk On Mon, Jan 6, 2014 at 11:23 AM, Ryosuke Niwa rn...@webkit.org wrote: I don't think we should do this given that the python server has been added to W3C testing harness, and they're gonna convert all existing tests to use that instead: http://lists.w3.org/Archives/Public/public-test-infra/2014JanMar/.html We should simply wait for that effort to take place and add a support for the python server instead. - R. Niwa On Fri, Dec 6, 2013 at 12:25 AM, youenn fablet youe...@gmail.com wrote: As long as the newly imported tests use relative URLs, alias may be used as a workaround. I will give it a try. Bug entry is at https://bugs.webkit.org/show_bug.cgi?id=125339 Any further help appreciated, Youenn 2013/12/6 Darin Adler da...@apple.com If that's really ends up being super hard we can always put yet another third-party or imported directory inside the http directory as previously suggested. it's annoying to have three different places for imported tests and code, but not something I want to hold us up for a long time. -- Darin Sent from my iPhone On Dec 5, 2013, at 5:51 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Dec 4, 2013 at 11:19 AM, Darin Adler da...@apple.com wrote: On Dec 4, 2013, at 6:48 AM, youenn fablet youe...@gmail.com wrote: I am planning to add some XHR tests from https://github.com/w3c/web-platform-tests. My initial plan was to add them in a subdirectory of LayoutTests/http/tests/w3c. If adding them into LayoutTests/imported/w3c, that would probably require updating the test scripts to start/stop the HTTP test server for that particular sub-folder. Any preference? I’d prefer LayoutTests/imported/w3c. Although I’m not so happy about the different terminology we are using for “imported” vs. “ThirdParty”, which seems like the same concept at the top level of the directory structure. One trickiness to it is that we don't currently run any HTTP test in parallel and the document root of the HTTP server is set to LayoutTests/http/tests so we might need to modify that or restart the HTTP server whenever we're running HTTP tests outside of LayoutTests/http. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
As long as the newly imported tests use relative URLs, alias may be used as a workaround. I will give it a try. Bug entry is at https://bugs.webkit.org/show_bug.cgi?id=125339 Any further help appreciated, Youenn 2013/12/6 Darin Adler da...@apple.com If that's really ends up being super hard we can always put yet another third-party or imported directory inside the http directory as previously suggested. it's annoying to have three different places for imported tests and code, but not something I want to hold us up for a long time. -- Darin Sent from my iPhone On Dec 5, 2013, at 5:51 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Dec 4, 2013 at 11:19 AM, Darin Adler da...@apple.com wrote: On Dec 4, 2013, at 6:48 AM, youenn fablet youe...@gmail.com wrote: I am planning to add some XHR tests from https://github.com/w3c/web-platform-tests. My initial plan was to add them in a subdirectory of LayoutTests/http/tests/w3c. If adding them into LayoutTests/imported/w3c, that would probably require updating the test scripts to start/stop the HTTP test server for that particular sub-folder. Any preference? I’d prefer LayoutTests/imported/w3c. Although I’m not so happy about the different terminology we are using for “imported” vs. “ThirdParty”, which seems like the same concept at the top level of the directory structure. One trickiness to it is that we don't currently run any HTTP test in parallel and the document root of the HTTP server is set to LayoutTests/http/tests so we might need to modify that or restart the HTTP server whenever we're running HTTP tests outside of LayoutTests/http. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
The way I got around this when I was first working on it was to simply map imported/w3c onto a subdirectory of the document root in apache; it's a two line change. For some time I've toyed with the idea of changing the DocumentRoot to just be LayoutTests/, so that any test could be run over http directly. A lot of tests and test results would need to be updated for this, but I think it could simplify the test organization a fair amount and open up some interesting possibilities. -- Dirk On Thu, Dec 5, 2013 at 6:25 PM, Darin Adler da...@apple.com wrote: If that's really ends up being super hard we can always put yet another third-party or imported directory inside the http directory as previously suggested. it's annoying to have three different places for imported tests and code, but not something I want to hold us up for a long time. -- Darin Sent from my iPhone On Dec 5, 2013, at 5:51 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Dec 4, 2013 at 11:19 AM, Darin Adler da...@apple.com wrote: On Dec 4, 2013, at 6:48 AM, youenn fablet youe...@gmail.com wrote: I am planning to add some XHR tests from https://github.com/w3c/web-platform-tests. My initial plan was to add them in a subdirectory of LayoutTests/http/tests/w3c. If adding them into LayoutTests/imported/w3c, that would probably require updating the test scripts to start/stop the HTTP test server for that particular sub-folder. Any preference? I’d prefer LayoutTests/imported/w3c. Although I’m not so happy about the different terminology we are using for “imported” vs. “ThirdParty”, which seems like the same concept at the top level of the directory structure. One trickiness to it is that we don't currently run any HTTP test in parallel and the document root of the HTTP server is set to LayoutTests/http/tests so we might need to modify that or restart the HTTP server whenever we're running HTTP tests outside of LayoutTests/http. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
On Wed, Dec 4, 2013 at 11:19 AM, Darin Adler da...@apple.com wrote: On Dec 4, 2013, at 6:48 AM, youenn fablet youe...@gmail.com wrote: I am planning to add some XHR tests from https://github.com/w3c/web-platform-tests. My initial plan was to add them in a subdirectory of LayoutTests/http/tests/w3c. If adding them into LayoutTests/imported/w3c, that would probably require updating the test scripts to start/stop the HTTP test server for that particular sub-folder. Any preference? I’d prefer LayoutTests/imported/w3c. Although I’m not so happy about the different terminology we are using for “imported” vs. “ThirdParty”, which seems like the same concept at the top level of the directory structure. Might be nice to think about a more elegant way to trigger the HTTP server when needed. Even requiring some kind of trigger file in each directory would be better than the hard-coded path rule we use now. I'd prefer/recommend LayoutTests/imported/w3c as well (and just modify the test scripts to deal w/ the additional http servers). As you start to pull in more of the w3c's tests, there will probably need to be some overall cleanup and refactoring of how stuff like this works and is shared (e.g., duplication of files under resources/ , paths mapped in the http server, etc.). Blink has the same issues. I'd be happy to help out with this where possible to try and keep things relatively consistent. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
On Wed, Dec 4, 2013 at 11:19 AM, Darin Adler da...@apple.com wrote: On Dec 4, 2013, at 6:48 AM, youenn fablet youe...@gmail.com wrote: I am planning to add some XHR tests from https://github.com/w3c/web-platform-tests. My initial plan was to add them in a subdirectory of LayoutTests/http/tests/w3c. If adding them into LayoutTests/imported/w3c, that would probably require updating the test scripts to start/stop the HTTP test server for that particular sub-folder. Any preference? I’d prefer LayoutTests/imported/w3c. Although I’m not so happy about the different terminology we are using for “imported” vs. “ThirdParty”, which seems like the same concept at the top level of the directory structure. One trickiness to it is that we don't currently run any HTTP test in parallel and the document root of the HTTP server is set to LayoutTests/http/tests so we might need to modify that or restart the HTTP server whenever we're running HTTP tests outside of LayoutTests/http. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
If that's really ends up being super hard we can always put yet another third-party or imported directory inside the http directory as previously suggested. it's annoying to have three different places for imported tests and code, but not something I want to hold us up for a long time. -- Darin Sent from my iPhone On Dec 5, 2013, at 5:51 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Dec 4, 2013 at 11:19 AM, Darin Adler da...@apple.com wrote: On Dec 4, 2013, at 6:48 AM, youenn fablet youe...@gmail.com wrote: I am planning to add some XHR tests from https://github.com/w3c/web-platform-tests. My initial plan was to add them in a subdirectory of LayoutTests/http/tests/w3c. If adding them into LayoutTests/imported/w3c, that would probably require updating the test scripts to start/stop the HTTP test server for that particular sub-folder. Any preference? I’d prefer LayoutTests/imported/w3c. Although I’m not so happy about the different terminology we are using for “imported” vs. “ThirdParty”, which seems like the same concept at the top level of the directory structure. One trickiness to it is that we don't currently run any HTTP test in parallel and the document root of the HTTP server is set to LayoutTests/http/tests so we might need to modify that or restart the HTTP server whenever we're running HTTP tests outside of LayoutTests/http. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
I am planning to add some XHR tests from https://github.com/w3c/web-platform-tests. My initial plan was to add them in a subdirectory of LayoutTests/http/tests/w3c. If adding them into LayoutTests/imported/w3c, that would probably require updating the test scripts to start/stop the HTTP test server for that particular sub-folder. Any preference? Youenn 2013/11/22 Ryosuke Niwa rn...@webkit.org On Fri, Nov 22, 2013 at 12:15 AM, Sergio Villar Senin svil...@igalia.comwrote: On 21/11/13 09:16, - R. Niwa wrote: Maciej says he'd rather see w3c directory under imported directory so I'm doing that. That was not done for other set of imported tests. Are you planning to move them also to that same place? The idea is that. I don't intend on doing that work myself. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
On Dec 4, 2013, at 6:48 AM, youenn fablet youe...@gmail.com wrote: I am planning to add some XHR tests from https://github.com/w3c/web-platform-tests. My initial plan was to add them in a subdirectory of LayoutTests/http/tests/w3c. If adding them into LayoutTests/imported/w3c, that would probably require updating the test scripts to start/stop the HTTP test server for that particular sub-folder. Any preference? I’d prefer LayoutTests/imported/w3c. Although I’m not so happy about the different terminology we are using for “imported” vs. “ThirdParty”, which seems like the same concept at the top level of the directory structure. Might be nice to think about a more elegant way to trigger the HTTP server when needed. Even requiring some kind of trigger file in each directory would be better than the hard-coded path rule we use now. — Darin___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] Importing W3C tests for HTML template elements
Hi, There has been a lot of discussions about importing W3C tests. Since I'm already trying to enable HTML template elements, I've decided to take lead on this and add LayoutTests/w3c directory as we've previously come to consensus. I've posted a patch to import some of HTML template elements tests into LayoutTests/w3c/html-templates at https://bugs.webkit.org/show_bug.cgi?id=124699 Since W3C is planning to move all tests into web-platform-tests repository: https://github.com/w3c/web-platform-tests, I didn't feel the need to put it inside LayoutTests/w3c/web-platform-tests/html-templates but I can do that if someone feels strongly about it. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
Maciej says he'd rather see w3c directory under imported directory so I'm doing that. - R. Niwa On Thu, Nov 21, 2013 at 5:01 PM, Ryosuke Niwa rn...@webkit.org wrote: Hi, There has been a lot of discussions about importing W3C tests. Since I'm already trying to enable HTML template elements, I've decided to take lead on this and add LayoutTests/w3c directory as we've previously come to consensus. I've posted a patch to import some of HTML template elements tests into LayoutTests/w3c/html-templates at https://bugs.webkit.org/show_bug.cgi?id=124699 Since W3C is planning to move all tests into web-platform-tests repository: https://github.com/w3c/web-platform-tests, I didn't feel the need to put it inside LayoutTests/w3c/web-platform-tests/html-templates but I can do that if someone feels strongly about it. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
On 21/11/13 09:16, Ryosuke Niwa wrote: Maciej says he'd rather see w3c directory under imported directory so I'm doing that. That was not done for other set of imported tests. Are you planning to move them also to that same place? BR ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
On Nov 21, 2013, at 00:01 , Ryosuke Niwa rn...@webkit.orgmailto:rn...@webkit.org wrote: Hi, There has been a lot of discussions about importing W3C tests. Since I'm already trying to enable HTML template elements, I've decided to take lead on this and add LayoutTests/w3c directory as we've previously come to consensus. I've posted a patch to import some of HTML template elements tests into LayoutTests/w3c/html-templates at https://bugs.webkit.org/show_bug.cgi?id=124699 Since W3C is planning to move all tests into web-platform-tests repository: https://github.com/w3c/web-platform-tests, I didn't feel the need to put it inside LayoutTests/w3c/web-platform-tests/html-templates but I can do that if someone feels strongly about it. I'm all for this. Are you aware of the script in Tools/Scripts/import-w3c-tests ? By default, it imports into LayoutTests/w3c. - Bem ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
On Fri, Nov 22, 2013 at 12:15 AM, Sergio Villar Senin svil...@igalia.comwrote: On 21/11/13 09:16, - R. Niwa wrote: Maciej says he'd rather see w3c directory under imported directory so I'm doing that. That was not done for other set of imported tests. Are you planning to move them also to that same place? The idea is that. I don't intend on doing that work myself. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] Importing W3C tests to webkit
At the WebKit contributor's meeting in April, we discussed a process for importing third party tests into the WebKit repository (specifically, from the W3C test repository). I documented the process that we came up with at the meeting on the WebKit wiki, here: http://trac.webkit.org/wiki/ImportingThirdPartyTests It would be great if we could get some feedback on this process. Once we can finalize the details for how this should function, the effort can proceed - scripts can be written to help automate the effort and an actual import of select W3C test suites can be performed. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Importing W3C tests to webkit
As I have said in the past, we should just import all tests, and treat non-text, non-ref tests as pixel tests. If we wanted to reduce the number of pixel tests we import, then we should submit those patches to W3C instead of directly submitting them to WebKit. In general, I don't buy the argument that adding more pixel tests incurs more cost than the benefit we get from importing the tests. If that were the case, we can just get rid of the existing pixel tests we have. The only sane argument I've heard so far to gate pixel tests is that the correctness of such tests need to be manually inspected, which requires a lot of manual labor and is very error prone. For script-based tests that print PASS/FAIL, it seems better to just check in *-expected-failure.txt or even -expected.txt files because we'll see PASS/FAIL in the expected tests themselves. - Ryosuke On Wed, May 23, 2012 at 10:26 AM, Jacob Goldstein jac...@adobe.com wrote: At the WebKit contributor's meeting in April, we discussed a process for importing third party tests into the WebKit repository (specifically, from the W3C test repository). I documented the process that we came up with at the meeting on the WebKit wiki, here: http://trac.webkit.org/wiki/ImportingThirdPartyTests It would be great if we could get some feedback on this process. Once we can finalize the details for how this should function, the effort can proceed - scripts can be written to help automate the effort and an actual import of select W3C test suites can be performed. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Importing W3C tests to webkit
I agree. If nothing else, getting W3C tests into the WebKit repository will help catch regressions. From: Ryosuke Niwa rn...@webkit.orgmailto:rn...@webkit.org To: Jacob Goldstein jac...@adobe.commailto:jac...@adobe.com Cc: WebKit Development webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org Subject: Re: [webkit-dev] Importing W3C tests to webkit As I have said in the past, we should just import all tests, and treat non-text, non-ref tests as pixel tests. If we wanted to reduce the number of pixel tests we import, then we should submit those patches to W3C instead of directly submitting them to WebKit. In general, I don't buy the argument that adding more pixel tests incurs more cost than the benefit we get from importing the tests. If that were the case, we can just get rid of the existing pixel tests we have. The only sane argument I've heard so far to gate pixel tests is that the correctness of such tests need to be manually inspected, which requires a lot of manual labor and is very error prone. For script-based tests that print PASS/FAIL, it seems better to just check in *-expected-failure.txt or even -expected.txt files because we'll see PASS/FAIL in the expected tests themselves. - Ryosuke On Wed, May 23, 2012 at 10:26 AM, Jacob Goldstein jac...@adobe.commailto:jac...@adobe.com wrote: At the WebKit contributor's meeting in April, we discussed a process for importing third party tests into the WebKit repository (specifically, from the W3C test repository). I documented the process that we came up with at the meeting on the WebKit wiki, here: http://trac.webkit.org/wiki/ImportingThirdPartyTests It would be great if we could get some feedback on this process. Once we can finalize the details for how this should function, the effort can proceed - scripts can be written to help automate the effort and an actual import of select W3C test suites can be performed. ___ webkit-dev mailing list webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Importing W3C tests to webkit
On Wed, May 23, 2012 at 11:56 AM, Ryosuke Niwa rn...@webkit.org wrote: As I have said in the past, we should just import all tests, and treat non-text, non-ref tests as pixel tests. If we wanted to reduce the number of pixel tests we import, then we should submit those patches to W3C instead of directly submitting them to WebKit. In general, I don't buy the argument that adding more pixel tests incurs more cost than the benefit we get from importing the tests. If that were the case, we can just get rid of the existing pixel tests we have. Not so: if the tests we had had 100% coverage, then importing more tests would buy us nothing, but getting rid of the existing tests would be quite unfortunate. Clearly adding tests incurs some costs and probably provides some benefit; the question is when does the cost exceed the benefit? As I am not against importing more tests per se, I think this only makes sense to evaluate on a case-by-case basis. The only sane argument I've heard so far to gate pixel tests is that the correctness of such tests need to be manually inspected, which requires a lot of manual labor and is very error prone. I'm assuming the above includes the ongoing maintenance cost of keeping pixel tests up to date, as well as the cost at the initial checkin. There is also the fact that the more tests we have, the more tests we have to run, and increasing cycle time by itself is a cost to developer productivity. Of course, it's also potentially the case that we have to update tests from time to time, although this doesn't happen often. Jacob, I gave you a bunch of feedback not long after you published the initial writeup, but it doesn't look like any of that has been incorporated? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Importing W3C tests to webkit
Dirk, my apologies, I was on travel the week you replied and missed your message. I found it and will review / update now. On 5/23/12 1:25 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, May 23, 2012 at 11:56 AM, Ryosuke Niwa rn...@webkit.org wrote: As I have said in the past, we should just import all tests, and treat non-text, non-ref tests as pixel tests. If we wanted to reduce the number of pixel tests we import, then we should submit those patches to W3C instead of directly submitting them to WebKit. In general, I don't buy the argument that adding more pixel tests incurs more cost than the benefit we get from importing the tests. If that were the case, we can just get rid of the existing pixel tests we have. Not so: if the tests we had had 100% coverage, then importing more tests would buy us nothing, but getting rid of the existing tests would be quite unfortunate. Clearly adding tests incurs some costs and probably provides some benefit; the question is when does the cost exceed the benefit? As I am not against importing more tests per se, I think this only makes sense to evaluate on a case-by-case basis. The only sane argument I've heard so far to gate pixel tests is that the correctness of such tests need to be manually inspected, which requires a lot of manual labor and is very error prone. I'm assuming the above includes the ongoing maintenance cost of keeping pixel tests up to date, as well as the cost at the initial checkin. There is also the fact that the more tests we have, the more tests we have to run, and increasing cycle time by itself is a cost to developer productivity. Of course, it's also potentially the case that we have to update tests from time to time, although this doesn't happen often. Jacob, I gave you a bunch of feedback not long after you published the initial writeup, but it doesn't look like any of that has been incorporated? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Importing W3C tests to webkit
On Wed, May 23, 2012 at 1:25 PM, Dirk Pranke dpra...@chromium.org wrote: Not so: if the tests we had had 100% coverage, then importing more tests would buy us nothing, but getting rid of the existing tests would be quite unfortunate. We certainly don't have 100% test coverage. Clearly adding tests incurs some costs and probably provides some benefit; the question is when does the cost exceed the benefit? Doing anything incurs some cost. In fact, not adding a test itself incurs a risk cost of potentially taking regressions in the future that could have been caught by the test. As I am not against importing more tests per se, I think this only makes sense to evaluate on a case-by-case basis. Sure. We shouldn't be importing tests that are obviously duplicates of our existing tests. The only sane argument I've heard so far to gate pixel tests is that the correctness of such tests need to be manually inspected, which requires a lot of manual labor and is very error prone. I'm assuming the above includes the ongoing maintenance cost of keeping pixel tests up to date, as well as the cost at the initial checkin. I'm not concerned of those. Once the correct expected result is checked in, it's pretty easy to rebaseline tests per rendering engine changes assuming people who are rebaselining tests know what they're doing. There is also the fact that the more tests we have, the more tests we have to run, and increasing cycle time by itself is a cost to developer productivity. Sure, but I don't think that's a valid argument for not adding tests especially since there is no way for us to mechanically test whether two tests test the same set of features or not (this is an intractable problem even in its limited form and an undecidable one in its most general form). Also, using ref test or pixel test, etc... doesn't change the cycle time significantly so I don't understand what your argument is. Or are you suggesting that non-ref tests are somehow more redundant than ref tests? (please give us why). Of course, it's also potentially the case that we have to update tests from time to time, although this doesn't happen often. Because the current process is broken :) In the ideal world, we would be updating our copy of W3C tests every so often (e.g. every month or so). - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Importing W3C tests to webkit
On Wed, May 23, 2012 at 1:41 PM, Ryosuke Niwa rn...@webkit.org wrote: The only sane argument I've heard so far to gate pixel tests is that the correctness of such tests need to be manually inspected, which requires a lot of manual labor and is very error prone. I'm assuming the above includes the ongoing maintenance cost of keeping pixel tests up to date, as well as the cost at the initial checkin. I'm not concerned of those. Once the correct expected result is checked in, it's pretty easy to rebaseline tests per rendering engine changes assuming people who are rebaselining tests know what they're doing. You should be concerned; keeping pixel tests up-to-date is clearly a non-zero cost that only the chromium port thus far has been willing to bear, and I suspect that the cost of updating baselines is substantially higher than the cost of the initial review over time (since it's a recurring cost). We only have to ask Emil and Levi what the cost of updating all of the pixel tests were for the subpixel layout test change, or ask the skia guys how many bug fixes they're reluctant to make because of the cost of reviewing literally thousands of images that change inconsequentially for empirical evidence for this. There is also the fact that the more tests we have, the more tests we have to run, and increasing cycle time by itself is a cost to developer productivity. Sure, but I don't think that's a valid argument for not adding tests especially since there is no way for us to mechanically test whether two tests test the same set of features or not (this is an intractable problem even in its limited form and an undecidable one in its most general form). At some point adding more tests will introduce a declining marginal rate of return in finding more bugs; this is a truism of software development, and is why *all* software testing efforts stop at some point (ignoring formal proofs of completeness in model checkers). Either you don't think this is true, or you think this is true and we're just not at that point yet. If you think the latter then we agree, but I don't understand why you are arguing as if you believe the former. To repeat myself, I have never said that we shouldn't ever add more tests, just that we should have a rational process for doing so that includes looking at what overlap we have with the existing tests and making sure that adding more tests delivers value. Since you agree at least that we shouldn't be adding duplicate tests, you clearly agree with this to some degree, so I'm not sure if you and I have any real disagreements or if we're just talking past each other. Also, using ref test or pixel test, etc... doesn't change the cycle time significantly so I don't understand what your argument is. Or are you suggesting that non-ref tests are somehow more redundant than ref tests? (please give us why). I am saying that I believe that adding pixel tests incur more cost on the project than adding ref tests, and since all testing is about cost vs. benefit, you need to be more careful when adding pixel tests. Since we actively discourage people from writing pixel tests in favor of text-only or ref tests, I hardly think this is a controversial stance. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Importing W3C tests to webkit
On May 23, 2012, at 2:16 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, May 23, 2012 at 1:41 PM, Ryosuke Niwa rn...@webkit.org wrote: The only sane argument I've heard so far to gate pixel tests is that the correctness of such tests need to be manually inspected, which requires a lot of manual labor and is very error prone. I'm assuming the above includes the ongoing maintenance cost of keeping pixel tests up to date, as well as the cost at the initial checkin. I'm not concerned of those. Once the correct expected result is checked in, it's pretty easy to rebaseline tests per rendering engine changes assuming people who are rebaselining tests know what they're doing. You should be concerned; keeping pixel tests up-to-date is clearly a non-zero cost that only the chromium port thus far has been willing to bear, and I suspect that the cost of updating baselines is substantially higher than the cost of the initial review over time (since it's a recurring cost). Are you concerned just about the actual pixel results or also about keeping render tree dumps up to date? We can address the pixel result issue by introducing a new test that dumps its render tree but does not do pixel testing. I think there is a high value to importing standards test suites wholesale, even if they overlap with our existing coverage. Picking and choosing subsets makes things more complicated. If there are significant externalities to adding particular kinds of tests, I would prefer we mitigate those externalities rather than run fewer tests. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Importing W3C tests to webkit
On 5/23/12 2:30 PM, Maciej Stachowiak m...@apple.com wrote: On May 23, 2012, at 2:16 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, May 23, 2012 at 1:41 PM, Ryosuke Niwa rn...@webkit.org wrote: The only sane argument I've heard so far to gate pixel tests is that the correctness of such tests need to be manually inspected, which requires a lot of manual labor and is very error prone. I'm assuming the above includes the ongoing maintenance cost of keeping pixel tests up to date, as well as the cost at the initial checkin. I'm not concerned of those. Once the correct expected result is checked in, it's pretty easy to rebaseline tests per rendering engine changes assuming people who are rebaselining tests know what they're doing. You should be concerned; keeping pixel tests up-to-date is clearly a non-zero cost that only the chromium port thus far has been willing to bear, and I suspect that the cost of updating baselines is substantially higher than the cost of the initial review over time (since it's a recurring cost). Are you concerned just about the actual pixel results or also about keeping render tree dumps up to date? We can address the pixel result issue by introducing a new test that dumps its render tree but does not do pixel testing. I think there is a high value to importing standards test suites wholesale, even if they overlap with our existing coverage. Picking and choosing subsets makes things more complicated. If there are significant externalities to adding particular kinds of tests, I would prefer we mitigate those externalities rather than run fewer tests. As a side note to this discussion, there is talk in the W3C community regarding their test approval process. At the recent working group meetings in Germany the idea was floated to simply approve all tests that are currently waiting for review (and doing this going forward, e.g. no longer requiring approval upon submission). Apparently, not enough people are reviewing tests, and as a result, tests can linger for months (or longer) before ever being looked at. Once browser vendors start implementing features, associated tests will be revisited for that area. This has by no means been decided, but something we should consider if it ultimately does come to fruition. If W3C tests are no longer reviewed, this would mean importing tests without any knowledge of their accuracy - though that will still allow us to catch regressions. Dirk, I've updated the process on the Wiki page with the feedback you provided. I hope I captured it all - I included everything that appeared to have agreement between you and Ryosuke. Feel free to modify it directly if I missed anything, or let me know and I can refine it further. Jacob Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Importing W3C tests to webkit
On May 23, 2012, at 3:13 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, May 23, 2012 at 2:30 PM, Maciej Stachowiak m...@apple.com wrote: Are you concerned just about the actual pixel results or also about keeping render tree dumps up to date? Both are more maintenance than a text-only test. In my experience, maintaining pixel tests is more expensive, but I also don't have any experience maintaining a non-pixel-test-running port. There are occasional changes that require rebaselining a very large number of render tree dumps. In my experience, this happens much less often than the frequency at which Chromium ports update pixel results, by several orders of magnitude. We can address the pixel result issue by introducing a new test that dumps its render tree but does not do pixel testing. Yes, it might make sense to do this (although it's not obvious to me how we would do this without modifying the test sources). Maybe we would need to maintain a separate manifest or some other list somewhere to indicate which dirs or tests should include pixel results. The quick and dirty way would be to add a SkipPixel directive (or whatever) to TestExpectations. Cheers, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Importing W3C tests to webkit
On Wed, May 23, 2012 at 2:59 PM, Jacob Goldstein jac...@adobe.com wrote: As a side note to this discussion, there is talk in the W3C community regarding their test approval process. At the recent working group meetings in Germany the idea was floated to simply approve all tests that are currently waiting for review (and doing this going forward, e.g. no longer requiring approval upon submission). Apparently, not enough people are reviewing tests, and as a result, tests can linger for months (or longer) before ever being looked at. Once browser vendors start implementing features, associated tests will be revisited for that area. This has by no means been decided, but something we should consider if it ultimately does come to fruition. If W3C tests are no longer reviewed, this would mean importing tests without any knowledge of their accuracy - though that will still allow us to catch regressions. Ick :(. I suppose if the test suites are mostly coming from established browser vendors or other trusted sources (who have been running them on their own browsers for a while) this isn't so bad, but it would be nice to see the suites get vetted somehow prior to them getting blessed by the w3c. Levels of approval, or something? Dirk, I've updated the process on the Wiki page with the feedback you provided. I hope I captured it all - I included everything that appeared to have agreement between you and Ryosuke. Feel free to modify it directly if I missed anything, or let me know and I can refine it further. It looks much closer to what I'd like, thanks! I'll probably modify it a bit and/or send you some suggestions offlist for some minor things. The one major issue I still have that I wouldn't want to just add/sneak into the wiki, is that I still don't see much of a discussion for how we identify duplicate tests. The mindset I would prefer would be that, for a given test suite, the person or persons importing the test suite should have to start by identifying which existing tests we have for similar functionality, and produce lists of which existing tests look like they are duplicates and should be remvoed, which tests we have that should be submitted back to the w3c for future inclusion, and which tests are clearly webkit-specific and should be kept but identified as clearly webkit-specific. For example, if we import a flexbox test suite from css3, we should default to removing fast/flexbox (and possibly also ietestcenter/css3/flexbox). [ This is a purely hypothetical example (I know nothing about these test suites). I wish to re-emphasize that we will have to look at these things on a case-by-case basis. ] I have no idea how much work we'd be asking the importer to undertake, but that's kinda the point. If we don't ask the importer to de-dup things, who is realistically going to de-dup them later? What do others thinks, is this too much to ask? If it is, how do we avoid incurring further test debt over time? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
