Re: [webkit-dev] Supporting w3c ref tests and changing our convention
On Nov 11, 2011, at 4:24 AM, Tor Arne Vestbø wrote: On 07.11.11 15:44, Alan Stearns wrote: What if we defer some of the W3C metadata work until tests were actually submitted to the W3C? 1. Tests we pull from W3C can run from manifests, since they are provided. 2. Tests we develop ourselves just use a naming convention (refs are named *-ref.html, and there's one ref per test even if that's duplicative) 3. When we choose to share a set of tests with the W3C, we do the extra work of adding metadata to the tests and possibly refactoring to reduce the number of -ref files. Once the W3C approves the tests we pull their copies and delete ours. I think this is the best of both worlds approach Yes, I support this proposal too. We may find we can automate (3) with a script. It sounds pretty mechanical. For (2), I would name the reference files *-expected.html to fit in with other formats of expected results such as *-expected.txt and *-expected.png. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Supporting w3c ref tests and changing our convention
On Mon, Nov 14, 2011 at 11:41 AM, Darin Adler da...@apple.com wrote: We may find we can automate (3) with a script. It sounds pretty mechanical. It appears that W3C mandates author name, etc... be included in the meta data as well but I guess we can add something like WebKit Community or WebKit Authors? W3C folks told us we can create a WebKit directory under submissions so we can put all tests we export from our layout tests suite there with some canonical author name. For (2), I would name the reference files *-expected.html to fit in with other formats of expected results such as *-expected.txt and *-expected.png. I'm sure this was discussed in length when we first implemented reftests, but the problem with that name convention is that some ref-tests expect the actual result to differ from the reference fie. That's why we currently use -match.html / -mismatch.html Given W3C test suites use -ref.html and -noref.html, it seems more natural for us to follow their naming convention. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Supporting w3c ref tests and changing our convention
On 11/14/11 12:07 PM, Ryosuke Niwa rn...@webkit.org wrote: On Mon, Nov 14, 2011 at 11:41 AM, Darin Adler da...@apple.com wrote: We may find we can automate (3) with a script. It sounds pretty mechanical. It appears that W3C mandates author name, etc... be included in the meta data as well but I guess we can add something like WebKit Community or WebKit Authors? W3C folks told us we can create a WebKit directory under submissions so we can put all tests we export from our layout tests suite there with some canonical author name. AFAIK the mandate is only that there be an author link with a title and an appropriate href. There are many examples of tests with only this author data: link rel=author title=Microsoft href=http://microsoft.com/; / So I assume something like this would be fine for WebKit submissions: link rel=author title=WebKit href=http://webkit.org/; / Tests can have more than one author link if anyone wants to add additional contact info. Alan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Supporting w3c ref tests and changing our convention
On 11/4/11 7:20 PM, Ojan Vafai o...@chromium.org wrote: On Fri, Nov 4, 2011 at 7:16 PM, Ryosuke Niwa rn...@webkit.org wrote: On Fri, Nov 4, 2011 at 7:07 PM, Ojan Vafai o...@chromium.org wrote: On Fri, Nov 4, 2011 at 7:03 PM, Ryosuke Niwa rn...@webkit.org wrote: I am, but I'm particularly concerned about W3C tests. It'll be nice if we had exactly one way of running / writing ref tests. I think we can easily automate the process of generating manifest files. The work flow will be as follows: 1. Write new ref tests using link element 2. Run some tool (maybe we can teach run-webkit-tests to do it automatically) 3. Upload patch with auto-regenerated manifest file It's mainly step 2 that I have a problem with, although I also don't like that the test is not self-contained. Generating manifest file when we add a test is much more efficient than generating it every time we run tests because we tend to do the latter much more often than the former. I think we're at an empass here. I don't see that further technical arguments will sway either of us. I do, however, expect that the vast majority of webkit developers would prefer to avoid a manifest file given the way the project has been structured up to now. What if we defer some of the W3C metadata work until tests were actually submitted to the W3C? 1. Tests we pull from W3C can run from manifests, since they are provided. 2. Tests we develop ourselves just use a naming convention (refs are named *-ref.html, and there's one ref per test even if that's duplicative) 3. When we choose to share a set of tests with the W3C, we do the extra work of adding metadata to the tests and possibly refactoring to reduce the number of -ref files. Once the W3C approves the tests we pull their copies and delete ours. Alan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Supporting w3c ref tests and changing our convention
It appears that we're not reaching any consensus here, and I don't feel like continuing any more bikeshedding. If anyone's interested, my patch for https://bugs.webkit.org/show_bug.cgi?id=66837 and Hayato's patch for https://bugs.webkit.org/show_bug.cgi?id=71567 add a basic support for W3C style ref tests based on reftest.list parsing. - Ryouske ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] Supporting w3c ref tests and changing our convention
Hi, There was a discussion about supporting W3C ref tests at TPAC yesterday. However, there appears to be some disagreement in how we support them. *Overview* CSS WG ref tests contain link elements that specify reference files. e.g. !DOCTYPE html PUBLIC -//W3C//DTD XHTML 1.0 Strict//EN http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd;html http://december.com/html/4/element/html.html xmlns=http://www.w3.org/1999/xhtml; head http://december.com/html/4/element/head.html title http://december.com/html/4/element/title.htmlCSS Reftest Reference/title http://december.com/html/4/element/title.html link http://december.com/html/4/element/link.html rel=author title=NAME_OF_AUTHOR href=mailto:EMAIL OR http://CONTACT_PAGE/ link http://december.com/html/4/element/link.html rel=match href=green-box-ref.xht / style http://december.com/html/4/element/style.html type=text/css![CDATA[ CSS FOR REFERENCE ]]/style http://december.com/html/4/element/style.html /head http://december.com/html/4/element/head.html body http://december.com/html/4/element/body.html CONTENT OF REFERENCE /body http://december.com/html/4/element/body.html/html http://december.com/html/4/element/html.html The highlighted line specifies that the rendered result of this page should be compared with that of green-box-ref.xht. This is very different from our current method to use filenames such as -match.html, -mismatch.html, etc.. to identify reference files. In addition, W3C test suites have a build step, which generates Mozilla-like test manifest files. This file will contain entries that identifies tests and corresponding reference files. e.g. == floats-wrap-bfc-outside-001.xht floats-wrap-bfc-outside-001-ref.xht == floats-wrap-top-below-bfc-001l.xht floats-wrap-top-below-001l-ref.xht != floats-wrap-top-below-bfc-001l.xht floats-wrap-top-below-001l-notref.xht == floats-wrap-top-below-bfc-001r.xht floats-wrap-top-below-001r-ref.xht != floats-wrap-top-below-bfc-001r.xht floats-wrap-top-below-001r-notref.xht All browser vendors who attended the meeting preferred this format. Microsoft representatives didn't attend the meeting. *Link element approach* Pros: - Can reuse same ref. file for multiple tests - Can have multiple ref. files for single test - Information is self-contained in the test file - We may get away with test suite build step (It turns out that we can't convert W3C ref tests to use WebKit conventions due to the first two points.) Cons: - Requires us modifying each port's DRT to support this format - Adding link elements itself may affect tests (all W3C tests are required to have link elements at the moment) - Hard to understand relationship between files. e.g. if we want to figure out which tests use ref.html, we must look at all test files - Other browser vendors (Firefox and Opera) prefer manifest file approach *Manifest file approach* Pros: - Easy to parse, and do not require us modifiying each port's DRT to support - Easy to tell relationships between files at glance - Do not affect tests since manifest files are external to tests - Preferred approach of Firefox and Opera Cons: - Have to maintain information in two places, tests and manifests (W3C test suites auto-generates manifest files in their build step) - Require build step (new W3C tests only have link elements but not manifests) I strongly prefer WebKit use the manifest file approach and deprecate our current file-name convention because it won't burden each port, and it's the preferred method of other browser vendors. Any opinions? * * Best, Ryosuke Niwa Software Engineer Google Inc. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Supporting w3c ref tests and changing our convention
Could you clarify why we have to need to modify DRT if we have Link Element approach? I guess one of the reasons is the performance. It might be better that we use DRT to parse and extract reference links from HTML since parsing HTML using Python might take unacceptable time and might be inaccurate. Is there any other reasons we should modify DRT to support Link Element approach? On Fri, Nov 4, 2011 at 12:34 PM, Ryosuke Niwa rn...@webkit.org wrote: Hi, There was a discussion about supporting W3C ref tests at TPAC yesterday. However, there appears to be some disagreement in how we support them. *Overview* CSS WG ref tests contain link elements that specify reference files. e.g. !DOCTYPE html PUBLIC -//W3C//DTD XHTML 1.0 Strict//EN http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd;html http://december.com/html/4/element/html.html xmlns=http://www.w3.org/1999/xhtml; head http://december.com/html/4/element/head.html title http://december.com/html/4/element/title.htmlCSS Reftest Reference/title http://december.com/html/4/element/title.html link http://december.com/html/4/element/link.html rel=author title=NAME_OF_AUTHOR href=mailto:EMAIL OR http://CONTACT_PAGE/ link http://december.com/html/4/element/link.html rel=match href=green-box-ref.xht / style http://december.com/html/4/element/style.html type=text/css![CDATA[ CSS FOR REFERENCE ]]/style http://december.com/html/4/element/style.html /head http://december.com/html/4/element/head.html body http://december.com/html/4/element/body.html CONTENT OF REFERENCE /body http://december.com/html/4/element/body.html/html http://december.com/html/4/element/html.html The highlighted line specifies that the rendered result of this page should be compared with that of green-box-ref.xht. This is very different from our current method to use filenames such as -match.html, -mismatch.html, etc.. to identify reference files. In addition, W3C test suites have a build step, which generates Mozilla-like test manifest files. This file will contain entries that identifies tests and corresponding reference files. e.g. == floats-wrap-bfc-outside-001.xht floats-wrap-bfc-outside-001-ref.xht == floats-wrap-top-below-bfc-001l.xht floats-wrap-top-below-001l-ref.xht != floats-wrap-top-below-bfc-001l.xht floats-wrap-top-below-001l-notref.xht == floats-wrap-top-below-bfc-001r.xht floats-wrap-top-below-001r-ref.xht != floats-wrap-top-below-bfc-001r.xht floats-wrap-top-below-001r-notref.xht All browser vendors who attended the meeting preferred this format. Microsoft representatives didn't attend the meeting. *Link element approach* Pros: - Can reuse same ref. file for multiple tests - Can have multiple ref. files for single test - Information is self-contained in the test file - We may get away with test suite build step (It turns out that we can't convert W3C ref tests to use WebKit conventions due to the first two points.) Cons: - Requires us modifying each port's DRT to support this format - Adding link elements itself may affect tests (all W3C tests are required to have link elements at the moment) - Hard to understand relationship between files. e.g. if we want to figure out which tests use ref.html, we must look at all test files - Other browser vendors (Firefox and Opera) prefer manifest file approach *Manifest file approach* Pros: - Easy to parse, and do not require us modifiying each port's DRT to support - Easy to tell relationships between files at glance - Do not affect tests since manifest files are external to tests - Preferred approach of Firefox and Opera Cons: - Have to maintain information in two places, tests and manifests (W3C test suites auto-generates manifest files in their build step) - Require build step (new W3C tests only have link elements but not manifests) I strongly prefer WebKit use the manifest file approach and deprecate our current file-name convention because it won't burden each port, and it's the preferred method of other browser vendors. Any opinions? * * Best, Ryosuke Niwa Software Engineer Google Inc. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev -- Hayato ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Supporting w3c ref tests and changing our convention
On Fri, Nov 4, 2011 at 1:32 PM, Hayato Ito hay...@chromium.org wrote: Could you clarify why we have to need to modify DRT if we have Link Element approach? I guess one of the reasons is the performance. It might be better that we use DRT to parse and extract reference links from HTML since parsing HTML using Python might take unacceptable time and might be inaccurate. Is there any other reasons we should modify DRT to support Link Element approach? That's the primary reason. I can see that there are more than 100,000 .xht files just in css2.1 test suite, and the number is growing. If we include css3, and other w3c test suites, we'll end up having tens of thousands of tests, if not millions. In addition, link element approach doesn't scale well in that there's no mandate as to how reference files are named, so we'll end up needlessly parsing all reference files as well depending on the order they appear in the directory. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Supporting w3c ref tests and changing our convention
On Fri, Nov 4, 2011 at 1:32 PM, Hayato Ito hay...@chromium.org wrote: Could you clarify why we have to need to modify DRT if we have Link Element approach? I guess one of the reasons is the performance. It might be better that we use DRT to parse and extract reference links from HTML since parsing HTML using Python might take unacceptable time and might be inaccurate. Is there any other reasons we should modify DRT to support Link Element approach? I agree with Ito-san that I don't think you have to modify each DRT. I see no reason to think that we couldn't parse the files reliably in NRWT. It's unclear how much of a perf impact there would be but that's easy enough to determine - I would expect it to be minimal compared to the time of actually rendering a page. That said, supporting a manifest file is clearly fairly easy to do in NRWT, and presumably easy to add as a build step (e.g., make DRT depend on it) and may have the added bonus of allowing us to run various mozilla reference test suites that wouldn't be using the links, so I'm fine with that. I think we should voice a concern w/ the W3C that their tests must follow consistent naming styles (for maintainability); we shouldn't view the links and the manifest step as a carte blanche to name tests and results whatever they want. Separately, if we are throwing around numbers in the range of 100K for tests to run, we should consider when we actually want to run them - i.e., what will the cycle time be if we run them on every change, etc.? But that can be dealt with when we get there. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Supporting w3c ref tests and changing our convention
If we import a bunch of w3c reftests in the future, it sounds reasonable and unavoidable to use manifest file, assuming the manifest file is auto-generated by w3c's build process. But I'd like to leave an option to developers to write reftests more casually without worrying about maintaing the manifest file. At the same time, I don't want to increase the number of ways to write/run tests anymore. So that would be great that we have the best of both worlds somehow. On Fri, Nov 4, 2011 at 1:39 PM, Ryosuke Niwa rn...@webkit.org wrote: On Fri, Nov 4, 2011 at 1:32 PM, Hayato Ito hay...@chromium.org wrote: Could you clarify why we have to need to modify DRT if we have Link Element approach? I guess one of the reasons is the performance. It might be better that we use DRT to parse and extract reference links from HTML since parsing HTML using Python might take unacceptable time and might be inaccurate. Is there any other reasons we should modify DRT to support Link Element approach? That's the primary reason. I can see that there are more than 100,000 .xht files just in css2.1 test suite, and the number is growing. If we include css3, and other w3c test suites, we'll end up having tens of thousands of tests, if not millions. In addition, link element approach doesn't scale well in that there's no mandate as to how reference files are named, so we'll end up needlessly parsing all reference files as well depending on the order they appear in the directory. - Ryosuke -- Hayato ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Supporting w3c ref tests and changing our convention
On Fri, Nov 4, 2011 at 1:47 PM, Dirk Pranke dpra...@chromium.org wrote: It's unclear how much of a perf impact there would be but that's easy enough to determine - I would expect it to be minimal compared to the time of actually rendering a page. Since I expect w3c to end up having hundreds of thousands of tests, I see any performance implication to be a serious threat. That said, supporting a manifest file is clearly fairly easy to do in NRWT, and presumably easy to add as a build step (e.g., make DRT depend on it) and may have the added bonus of allowing us to run various mozilla reference test suites that wouldn't be using the links, so I'm fine with that. In fact, we already have a patch to support it on : https://bugs.webkit.org/show_bug.cgi?id=66837 and https://bugs.webkit.org/show_bug.cgi?id=71567 I think we should voice a concern w/ the W3C that their tests must follow consistent naming styles (for maintainability); we shouldn't view the links and the manifest step as a carte blanche to name tests and results whatever they want. Yes, I have. More comments on http://www.w3.org/html/wg/wiki/Testing or http://wiki.csswg.org/test will be helpful. Separately, if we are throwing around numbers in the range of 100K for tests to run, we should consider when we actually want to run them - i.e., what will the cycle time be if we run them on every change, etc.? But that can be dealt with when we get there. We need separate bots in the long term for sure. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Supporting w3c ref tests and changing our convention
On Fri, Nov 4, 2011 at 4:47 PM, Dirk Pranke dpra...@chromium.org wrote: Separately, if we are throwing around numbers in the range of 100K for tests to run, we should consider when we actually want to run them - i.e., what will the cycle time be if we run them on every change, etc.? But that can be dealt with when we get there. I would also like to throw out the idea of pulling the layout tests out into their own repo, maybe even per platform. Currently the huge number of layout tests in WebKit make many git operations unbearably slow, such as git status, which basically does a stat() on every file in a repo when the plain git status is used. Even on Linux where stat() is mega-fast it takes many seconds to show git status even on a clean checkout (at least on my laptop, which admittedly isn't the fastest.) It is worse on other platforms with a slower stat(). But I expect there might be a lot of pushback on such an idea, especially just to make one tool run faster. -- Regards, Ryan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Supporting w3c ref tests and changing our convention
I don't see any need for manifest files. Needing to maintain information about a test somewhere other than in the test itself or in the expected result file is a significant maintenance burden. We should avoid it if we can. On Fri, Nov 4, 2011 at 2:00 PM, Ryosuke Niwa rn...@webkit.org wrote: On Fri, Nov 4, 2011 at 1:47 PM, Dirk Pranke dpra...@chromium.org wrote: It's unclear how much of a perf impact there would be but that's easy enough to determine - I would expect it to be minimal compared to the time of actually rendering a page. Since I expect w3c to end up having hundreds of thousands of tests, I see any performance implication to be a serious threat. There is no inherent performance problem that I can see. We just need to structure things in a way that avoids performance problems. This only requires that we ensure that all reference files are either themselves tests or have ref in the name and/or path. Even if the W3C and/or Mozilla test suites don't end up enforcing this requirement we can check in a script to Tools/Scripts that restructures the tests appropriately before we commit them to our tree. The performance benefits of the manifest files could be addressed by not walking the tree before running the tests. We only do that now because that was the expedient way to write the code. If walking the tree turns out to be a performance problem down the road, we can run the tests as we walk the tree. If that's still too slow, we can generate a transient manifest file that goes in your output directory. Separately, if we are throwing around numbers in the range of 100K for tests to run, we should consider when we actually want to run them - i.e., what will the cycle time be if we run them on every change, etc.? But that can be dealt with when we get there. We need separate bots in the long term for sure. I don't see what's special about reftests that we'd run them on a separate bot. We might decide to shard test running across different bots in some way, but sharding by test type seems unhelpful. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Supporting w3c ref tests and changing our convention
On Fri, Nov 4, 2011 at 4:01 PM, Ojan Vafai o...@chromium.org wrote: I don't see any need for manifest files. W3C's build step auto-generates them. On Fri, Nov 4, 2011 at 2:00 PM, Ryosuke Niwa rn...@webkit.org wrote: On Fri, Nov 4, 2011 at 1:47 PM, Dirk Pranke dpra...@chromium.org wrote: It's unclear how much of a perf impact there would be but that's easy enough to determine - I would expect it to be minimal compared to the time of actually rendering a page. Since I expect w3c to end up having hundreds of thousands of tests, I see any performance implication to be a serious threat. There is no inherent performance problem that I can see. We just need to structure things in a way that avoids performance problems. But we don't need to structure them. They're organized in W3C's repo, and we're just going to import them in some directory; e.g. LayoutTests/w3c/. If we're adding new tests that we intend to contribute back to W3C, then those tests should live outside of this directory. The preferred approach, however, is to add tests to W3C repo first, then import them back to WebKit. This only requires that we ensure that all reference files are either themselves tests or have ref in the name and/or path. Even if the W3C and/or Mozilla test suites don't end up enforcing this requirement we can check in a script to Tools/Scripts that restructures the tests appropriately before we commit them to our tree. That, to me, is a problem. We try not to modify external tests or restructure them when we import them. Renaming files will also make it hard to figure out the correspondence between tests in WebKit's repo and W3C's repo. It's a significant cognitive stress if one is to contribute tests back to W3C's test suite. The performance benefits of the manifest files could be addressed by not walking the tree before running the tests. We only do that now because that was the expedient way to write the code. If walking the tree turns out to be a performance problem down the road, we can run the tests as we walk the tree. If that's still too slow, we can generate a transient manifest file that goes in your output directory. Generating a transient manifest file may make sense for WebKit tests but not for W3C tests although I don't see why generating a transient manifest file should improve the performance. Also, I don't see why we should be generating manifest files on demand when W3C test suite's build step automatically does that for us. We can build check tests into the repo once and we'll have manifest files. Since we shouldn't be modifying those tests anyway, I don't see why we need to generate manifests on demand. I don't see what's special about reftests that we'd run them on a separate bot. We might decide to shard test running across different bots in some way, but sharding by test type seems unhelpful. I'm not talking about ref-tests. I'm talking about tests imported from the W3C test suite. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Supporting w3c ref tests and changing our convention
On Fri, Nov 4, 2011 at 6:31 PM, Ojan Vafai o...@chromium.org wrote: On Fri, Nov 4, 2011 at 5:52 PM, Ryosuke Niwa rn...@webkit.org wrote: W3C's build step auto-generates them. I thought you were arguing that we should use manifest files for all reftests because having multiple ways to do things is confusing. I am, but I'm particularly concerned about W3C tests. It'll be nice if we had exactly one way of running / writing ref tests. I think we can easily automate the process of generating manifest files. The work flow will be as follows: 1. Write new ref tests using link element 2. Run some tool (maybe we can teach run-webkit-tests to do it automatically) 3. Upload patch with auto-regenerated manifest file But we don't need to structure them. They're organized in W3C's repo, and we're just going to import them in some directory; e.g. LayoutTests/w3c/. I'm reluctantly OK with doing this exclusively for test suites we import that come with a manifest file, since the maintenance cost is lower. Although, I'd prefer if we limited the number of ways you could mark a test as a reftest. Yeah, I won't be particularly happy about having two ways to write ref tests. If we're adding new tests that we intend to contribute back to W3C, then those tests should live outside of this directory. The preferred approach, however, is to add tests to W3C repo first, then import them back to WebKit. I don't think this is realistic. For example, for the new flexbox tests, we're writing them against WebKit's code and they're changing as we go. In addition, all the properties are webkit prefixed. We're not writing test suites in isolation. We're writing them as we implement the feature or fix the appropriate bugs. Right, but then you'll need to unprefix those tests and reformat them in order to upstream the tests anyway. I don't see what's special about reftests that we'd run them on a separate bot. We might decide to shard test running across different bots in some way, but sharding by test type seems unhelpful. I'm not talking about ref-tests. I'm talking about tests imported from the W3C test suite. Even then, I don't see any benefit to not running them as part of our main test suite. It all depends on the number of tests. CSS 2.1 test suite appear to contain 10,000+ tests (there are 10,000+ .xht files). If we're adding CSS3, and other test suites from W3C, it can easily be in the order of 100,000s given that W3C is trying to increase the number of tests significantly in the coming years. Given we spend 10+ minutes running 26,000+ layout tests on our bots, adding 100,000+ tests will result in quadrupling our bot cycle time (to 40-50+ minutes; or to 3-4 hours for any bots that run pixel tests). It's definitely nice if we could run them all on the main waterfall with our regression tests, but slowed cycle time may significantly hinder our ability to catch new test failures on time. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Supporting w3c ref tests and changing our convention
On Fri, Nov 4, 2011 at 7:07 PM, Ojan Vafai o...@chromium.org wrote: On Fri, Nov 4, 2011 at 7:03 PM, Ryosuke Niwa rn...@webkit.org wrote: I am, but I'm particularly concerned about W3C tests. It'll be nice if we had exactly one way of running / writing ref tests. I think we can easily automate the process of generating manifest files. The work flow will be as follows: 1. Write new ref tests using link element 2. Run some tool (maybe we can teach run-webkit-tests to do it automatically) 3. Upload patch with auto-regenerated manifest file It's mainly step 2 that I have a problem with, although I also don't like that the test is not self-contained. Generating manifest file when we add a test is much more efficient than generating it every time we run tests because we tend to do the latter much more often than the former. If we're adding new tests that we intend to contribute back to W3C, then those tests should live outside of this directory. The preferred approach, however, is to add tests to W3C repo first, then import them back to WebKit. I don't think this is realistic. For example, for the new flexbox tests, we're writing them against WebKit's code and they're changing as we go. In addition, all the properties are webkit prefixed. We're not writing test suites in isolation. We're writing them as we implement the feature or fix the appropriate bugs. Right, but then you'll need to unprefix those tests and reformat them in order to upstream the tests anyway. Yes, but the point is that it doesn't work to write the tests in the W3C repo first because what goes in the W3C repo doesn't work for us until we unprefix our implementation. Yeah, so those tests should be kept separate from W3C test suites. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Supporting w3c ref tests and changing our convention
On Fri, Nov 4, 2011 at 7:16 PM, Ryosuke Niwa rn...@webkit.org wrote: On Fri, Nov 4, 2011 at 7:07 PM, Ojan Vafai o...@chromium.org wrote: On Fri, Nov 4, 2011 at 7:03 PM, Ryosuke Niwa rn...@webkit.org wrote: I am, but I'm particularly concerned about W3C tests. It'll be nice if we had exactly one way of running / writing ref tests. I think we can easily automate the process of generating manifest files. The work flow will be as follows: 1. Write new ref tests using link element 2. Run some tool (maybe we can teach run-webkit-tests to do it automatically) 3. Upload patch with auto-regenerated manifest file It's mainly step 2 that I have a problem with, although I also don't like that the test is not self-contained. Generating manifest file when we add a test is much more efficient than generating it every time we run tests because we tend to do the latter much more often than the former. I think we're at an empass here. I don't see that further technical arguments will sway either of us. I do, however, expect that the vast majority of webkit developers would prefer to avoid a manifest file given the way the project has been structured up to now. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
