[webkit-dev] The WebKit2 bot is failing 100% of the tests
The WebKit2 bot seems to be failing 100% of the tests: http://build.webkit.org/waterfall?show=SnowLeopard%20Intel%20Release%20(WebKit2%20Tests) The regression range appears to be the following: http://trac.webkit.org/log/trunk?rev=90166stop_rev=90160verbose=on That regression range has four patches that appear related to WebKit2, mostly related to API versioning. I don't know enough about WebKit2 to dig us out of this one, but hopefully someone on this list does. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] [WebKit][Windows] compilation errors on Windows (using Cygwin)
Dear All, I am trying to compile WebKit on Windows using WebKit. I got below compiler errors : JavaScriptCore/jit/JITStubs.cpp:133: error: size of array 'dummyJITStackFrame_maintains_16byte_stack_alignment' is negative JavaScriptCore/jit/JITStubs.cpp:134: error: size of array 'dummyJITStackFrame_stub_argument_space_matches_ctiTrampoline' is negative JavaScriptCore/jit/JITStubs.cpp:135: error: size of array 'dummyJITStackFrame_callFrame_offset_matches_ctiTrampoline' is negative JavaScriptCore/jit/JITStubs.cpp:136: error: size of array 'dummyJITStackFrame_code_offset_matches_ctiTrampoline' is negative make[3]: *** [JavaScriptCore/jit/libJavaScriptCore_la-JITStubs.lo] Error 1 I commented out those four line which were compiler assertion and then I got below linking errors : ./.libs/libJavaScriptCore.a(libJavaScriptCore_la-Interpreter.o):Interpreter.cpp:(.text+0x96f): undefined reference to `_ctiTrampoline' ./.libs/libJavaScriptCore.a(libJavaScriptCore_la-Interpreter.o):Interpreter.cpp:(.text+0xb74): undefined reference to `_ctiTrampoline' ./.libs/libJavaScriptCore.a(libJavaScriptCore_la-Interpreter.o):Interpreter.cpp:(.text+0xf1a): undefined reference to `_ctiTrampoline' ./.libs/libJavaScriptCore.a(libJavaScriptCore_la-Interpreter.o):Interpreter.cpp:(.text+0x1d58): undefined reference to `_ctiTrampoline' ./.libs/libJavaScriptCore.a(libJavaScriptCore_la-JITStubs.o):JITStubs.cpp:(.text+0x1f): undefined reference to `cti_vm_throw' ./.libs/libJavaScriptCore.a(libJavaScriptCore_la-JITStubs.o):JITStubs.cpp:(.text+0x4c): undefined reference to `_ctiVMThrowTrampoline' ./.libs/libJavaScriptCore.a(libJavaScriptCore_la-JITStubs.o):JITStubs.cpp:(.text+0x3862): undefined reference to `_ctiOpThrowNotCaught' ./.libs/libJavaScriptCore.a(libJavaScriptCore_la-JITOpcodes.o):JITOpcodes.cpp:(.text+0x3a42): undefined reference to `_ctiVMThrowTrampoline' I have tried using __attribute__(used). But I again got the same error. I am not able to proceed. Please anybody let me know how to remove these errors. Awaiting reply. Many Thanks, Dipak ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Switching to new-run-webkit-tests
Sorry, no switch over tonight. The WebKit2 build is too destroyed to have any confidence that we're doing things correctly and the Snow Leopard buildbot appears to have an errant HTTP server bound to port 8080. :( Adam On Thu, Jun 30, 2011 at 11:47 AM, Adam Barth aba...@webkit.org wrote: According to https://bugs.webkit.org/show_bug.cgi?id=34984, there are only two remaining blocking issues: 1) Teaching new-run-webkit-tests to understand the difference between WebProcess crashes and other sorts of crashes in WebKit2. 2) Fixing some issues with the Apple Windows port. Once we get (1) fixed (hopefully today), we're going to start moving (non-Windows) ports over to new-run-webkit-tests (hopefully tonight). I suspect more issues will crop up once we actually flip the switch. Please do not hesitate to contact Eric or myself (either via #webkit or via bugs.webkit.org). We'll try to respond to any issues that arise as quickly as possible. Thanks for your patience. Adam On Mon, Jun 27, 2011 at 4:26 PM, Xan Lopez x...@gnome.org wrote: On Tue, Jun 28, 2011 at 1:17 AM, Eric Seidel e...@webkit.org wrote: There appear to be 6 remaining blocking issues: https://bugs.webkit.org/showdependencytree.cgi?id=34984hide_resolved=1 We would like to hear from others who have tried new-run-webkit-tests, if they have issues which they believe should block migrating to NRWT. (If so, please file and block the master bug!) I can see the GTK+ port thing with Xvfb is there already, so not a lot to add. NWRT is more sensitive to slow tests than the old infrastructure, so we had to add a bunch of them to the expectations file; I don't think this is particularly bad. In any case with the Xvfb patch and my local expectations file things run beautifully and way faster than before, so looking great from our side! Xan Thanks. -eric ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] Fwd: mipsel-uclibc-linux
Hi All,
I am porting webkit on MIPS for different devices, when I an using
MIPS-LINUX it is working fine.
But when I am using MIPSEL-UCLIBC-LINUX tool chain I am getting crash on
signal 11 (SIG_SEGV) and launcher stopped at
GtkWidget* webkit_web_view_new(void)
{
WebKitWebView* webView
=WEBKIT_WEB_VIEW(g_object_new(WEBKIT_TYPE_WEB_VIEW, NULL));
return GTK_WIDGET(webView);
}
Please let me know if any way out it...
Thanx Regards,
K L Yadav
Samsung India Software Center,
Noida.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] The WebKit2 bot is failing 100% of the tests
On 2011-06-30, at 23:51, Adam Barth wrote: The WebKit2 bot seems to be failing 100% of the tests: http://build.webkit.org/waterfall?show=SnowLeopard%20Intel%20Release%20(WebKit2%20Tests) The regression range appears to be the following: http://trac.webkit.org/log/trunk?rev=90166stop_rev=90160verbose=on That regression range has four patches that appear related to WebKit2, mostly related to API versioning. I don't know enough about WebKit2 to dig us out of this one, but hopefully someone on this list does. It was one of my changes that broke this. I went ahead and landed a fix in r90224. Is there some reason that you didn't file a bug report about this issue or even email me directly about it? If you'd done either of these things I would have seen it last night and had this fixed much sooner. - Mark ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Fwd: mipsel-uclibc-linux
On 07/01/2011 11:10 AM, kanhaiya yadav wrote: Hi All, I am porting webkit on MIPS for different devices, when I an using MIPS-LINUX it is working fine. But when I am using MIPSEL-UCLIBC-LINUX tool chain I am getting crash on signal 11 (SIG_SEGV) and launcher stopped at please use the webkit-help or the webkit-gtk mailinglist, this one is for the development of webkit. holger ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Proposal: Commit messages should start with a one-line summary of the change
On Jun 30, 2011, at 7:52 PM, Darin Adler wrote: I don’t think it’s a good idea to add yet another thing to every change log entry. I wouldn't consider this something being added. Most ChangeLog entries (the good ones, anyway) already contain a description of the fix in addition to the bug title, URL and reviewer. http://trac.webkit.org/changeset/90086 is a good example. All I'm proposing is that part of the explanation of the fix be extracted out into a one-line summary at the top of the ChangeLog. So it's not so much adding something as rearranging the information that is (or should) already be present. I do think that given the tools behavior you described, we should move the reviewer text after the bug title and patch description paragraph, even though you explicitly said that’s not what you are proposing. I think that would be a good first step. It would definitely be better than what we have now from a tools perspective. And in a lot of cases these days, the bug title really is a short description of the change. [1] I think these two bugs will get us started: http://webkit.org/b/26755 webkit-patch's commit messages are less readable than commit-log-editor's http://webkit.org/b/63804 commit-log-editor reorders ChangeLog entries in unexpected ways -Adam 1. I don't think this is necessarily a good thing; it would be better in my opinion if bug titles described user-visible symptoms or missing features. The typical webkit-patch-based workflow of writing a patch first and then uploading it to a new bug encourages these kinds of solution-based rather than issue-based bug titles. But webkit-patch makes so many things so much easier that it's hard to fault it! ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] [WebKit][Windows] compilation errors on Windows (using Cygwin)
For help building WebKit, please use the webkit-help mailing list. This list is for discussion of development of WebKit. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Proposal: Commit messages should start with a one-line summary of the change
I don’t think it’s a good idea to add yet another thing to every change log entry. I wouldn't consider this something being added. Most ChangeLog entries (the good ones, anyway) already contain a description of the fix in addition to the bug title, URL and reviewer. While I sympathize with the idea of the one line explanations, I think this adds unneeded complexity to our workflow and the added value you mentioned does not seem to overcome it (namely making a couple of tools work better). Also you are implicitly constraining people to explain their change with one-liners which IMHO is not a good constraint. I would rather see good explanations than explanations that need to match a format. Thanks, Julien ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Proposal: Commit messages should start with a one-line summary of the change
On Jul 1, 2011, at 11:57 AM, Julien Chaffraix wrote: I don’t think it’s a good idea to add yet another thing to every change log entry. I wouldn't consider this something being added. Most ChangeLog entries (the good ones, anyway) already contain a description of the fix in addition to the bug title, URL and reviewer. While I sympathize with the idea of the one line explanations, I think this adds unneeded complexity to our workflow and the added value you mentioned does not seem to overcome it (namely making a couple of tools work better). I wasn't trying to add any complexity. My hope is that people are already adding descriptions of their changes to their ChangeLog entries, like you do. I was just trying to suggest a better way to organize that description. In my original email I noted two benefits of making this change: 1) It would make it much easier to understand at a glance what the change actually does. 2) It would make many of our tools work better I see (1) as the primary benefit, and (2) as a nice side-benefit. Also you are implicitly constraining people to explain their change with one-liners which IMHO is not a good constraint. I would rather see good explanations than explanations that need to match a format. I'm not trying to constrain descriptions to one line. I think that would be horrible in a lot of cases! I'm just suggesting that there be a one-line summary at the top followed by a more detailed description below. (Note that our current ChangeLog template doesn't say that you should include *any* description of the change.) -Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Proposal: Commit messages should start with a one-line summary of the change
I wasn't trying to add any complexity. My hope is that people are already adding descriptions of their changes to their ChangeLog entries, like you do. I was just trying to suggest a better way to organize that description. In my original email I noted two benefits of making this change: 1) It would make it much easier to understand at a glance what the change actually does. 2) It would make many of our tools work better Thanks for clarifying here. I would have equated 1) to the very existence of ChangeLog which is supposed to give this background. I see (1) as the primary benefit, and (2) as a nice side-benefit. Also you are implicitly constraining people to explain their change with one-liners which IMHO is not a good constraint. I would rather see good explanations than explanations that need to match a format. I'm not trying to constrain descriptions to one line. I think that would be horrible in a lot of cases! I'm just suggesting that there be a one-line summary at the top followed by a more detailed description below. I said that it _implicitly_ added a constraint. Let me take an example to make this point clear: if you have a 1.5 line long description, it doesn't seem to make sense to split it in 2. I, maybe other people too, would side with removing maybe important information for shortness's sake. This sounds like the kind of case where the strict format is harmful. It may be theoretical but it looks like a slippery slope. (Note that our current ChangeLog template doesn't say that you should include *any* description of the change.) Indeed and I agree with you that this is wrong. However my feeling is that reviewers usually asks for that information to be included as part of the ChangeLog if it is not present. My argument is against the format I would say, not about making this information available / mandatory. Thanks, Julien ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] No getters for iconURL/title/body in Notification.idl
Hi, we are adding all the Notification related objects to the GTK+ DOM bindings, and also adding the necessary APIs to WebKitGTK+ to interact with the UA. One thing we have noticed is that although the constructors for the Notification objects have the three data parameters mentioned in the subject, the spec we follow (http://dev.chromium.org/developers/design-documents/desktop-notifications/api-specification, as I understand it) does not provide attributes for those in the Notification interface. Since those are needed by the UA to actually show the notification in a meaningful way, we wonder if that's an oversight or a design decision. It seems the way Chromium solves this is by doing a custom class with those getters, but since we already have pretty complete DOM bindings we'd like our low-level signals and APIs to use those instead of reimplementing them (or, at best, subclassing to add the needed functionality). Any comments? Xan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] The WebKit2 bot is failing 100% of the tests
On Fri, Jul 1, 2011 at 6:31 AM, Mark Rowe mr...@apple.com wrote: On 2011-06-30, at 23:51, Adam Barth wrote: The WebKit2 bot seems to be failing 100% of the tests: http://build.webkit.org/waterfall?show=SnowLeopard%20Intel%20Release%20(WebKit2%20Tests) The regression range appears to be the following: http://trac.webkit.org/log/trunk?rev=90166stop_rev=90160verbose=on That regression range has four patches that appear related to WebKit2, mostly related to API versioning. I don't know enough about WebKit2 to dig us out of this one, but hopefully someone on this list does. It was one of my changes that broke this. I went ahead and landed a fix in r90224. Is there some reason that you didn't file a bug report about this issue or even email me directly about it? If you'd done either of these things I would have seen it last night and had this fixed much sooner. I assumed you were asleep. :( Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?
When a test starts failing on a bot that uses old-run-webkit-tests, we typically check in expected failure results for that test (e.g., http://trac.webkit.org/changeset/90235). That way we can find out if the test's behavior changes in the future even though the test is still failing. This is particularly useful for tests that are actually testing multiple things at once. (Maybe they should be broken up into multiple tests, but that's a different discussion.) Is there a way to achieve this with new-run-webkit-tests? I know that you can mark a test as an expected failure (either a text diff, or an image diff, or both). Does it let you specify that the test should fail in a particular way? -Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?
When a test starts failing on a bot that uses old-run-webkit-tests, we typically check in expected failure results for that test (e.g., http://trac.webkit.org/changeset/90235). That way we can find out if the test's behavior changes in the future even though the test is still failing. This is particularly useful for tests that are actually testing multiple things at once. (Maybe they should be broken up into multiple tests, but that's a different discussion.) Is there a way to achieve this with new-run-webkit-tests? I know that you can mark a test as an expected failure (either a text diff, or an image diff, or both). Does it let you specify that the test should fail in a particular way? -Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?
You can do the same thing with NRWT that you can do with ORWT in this regard, but nothing new. The test_expectation.txt file does give you more fine-grained control than Skipped in the sense that you can distinguish between TEXT, IMAGE, CRASH, and TIMEOUT failures, but it doesn't let you distinguish between different sorts of TEXT failures, for example. My sense is that the test_expectation.txt file is already somewhat over complicated for the problem it solves. In this case, the workflow of changing the expected results and filing a bug to track the failure seems like a reasonable solution, especially if there's a keyword or master bug that lets you find all these bugs easily. Adam On Fri, Jul 1, 2011 at 10:58 AM, Adam Roben aro...@apple.com wrote: When a test starts failing on a bot that uses old-run-webkit-tests, we typically check in expected failure results for that test (e.g., http://trac.webkit.org/changeset/90235). That way we can find out if the test's behavior changes in the future even though the test is still failing. This is particularly useful for tests that are actually testing multiple things at once. (Maybe they should be broken up into multiple tests, but that's a different discussion.) Is there a way to achieve this with new-run-webkit-tests? I know that you can mark a test as an expected failure (either a text diff, or an image diff, or both). Does it let you specify that the test should fail in a particular way? ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] No getters for iconURL/title/body in Notification.idl
On Fri, Jul 1, 2011 at 10:31 AM, Xan Lopez x...@gnome.org wrote: Hi, we are adding all the Notification related objects to the GTK+ DOM bindings, and also adding the necessary APIs to WebKitGTK+ to interact with the UA. One thing we have noticed is that although the constructors for the Notification objects have the three data parameters mentioned in the subject, the spec we follow ( http://dev.chromium.org/developers/design-documents/desktop-notifications/api-specification , as I understand it) does not provide attributes for those in the Notification interface. Since those are needed by the UA to actually show the notification in a meaningful way, we wonder if that's an oversight or a design decision. It seems the way Chromium solves this is by doing a custom class with those getters, but since we already have pretty complete DOM bindings we'd like our low-level signals and APIs to use those instead of reimplementing them (or, at best, subclassing to add the needed functionality). Any comments? Xan The data parameters are certainly needed by the UA to show the notification, but it seems like you're suggesting they need to be exposed as part of the Notification API (i.e., to web pages). I don't see why that would be necessary. WebCore has a Notification object in C++ which does provide access to the fields, and which is passed up through the NotificationPresenter interface implemented in the ChromeClient. That's how Chromium does it -- would it not work for GTK? -John ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] No getters for iconURL/title/body in Notification.idl
On Fri, Jul 1, 2011 at 8:17 PM, John Gregg john...@google.com wrote: The data parameters are certainly needed by the UA to show the notification, but it seems like you're suggesting they need to be exposed as part of the Notification API (i.e., to web pages). I don't see why that would be necessary. WebCore has a Notification object in C++ which does provide access to the fields, and which is passed up through the NotificationPresenter interface implemented in the ChromeClient. That's how Chromium does it -- would it not work for GTK? There's several things there. - Indeed we'd use this WebCore Notification object, same than Chromium. The problem is that this object doubles as the representation of whatever is in the spec we follow (usually the DOM spec, not in this case but close enough) and it also has extra things useful for the UA, as we are discussing. - In the normal case we just let our bindings generate automatically a GObject wrapper for the core object, but since this is generated from the IDL it would lack the needed accessors. - You say that you don't see any need to let binding users access that information, in that case our version of the DOM object probably shouldn't allow that either. - In that case we can either subclass it and add what we need, or just not use the DOM objects in this case. The problem with the latter is that we'd need to reimplement portions of the expected DOM interfaces in the new class we'd do, so it seems a bit pointless to do so, and that we'd expose two different versions of the object in different scenarios (for instance, createNotification would create something different than what you'd receive from the WebKit API in some cases). In any case, I guess the obvious answer is: why is it not interesting for the web page to access the parameters passed to the object when it's constructed? Since it will generally be constructed from within the page itself (unless I'm totally wrong), it seems reasonable. Cheers, Xan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?
I proposed a while back to chromium folk that we minimize the use of TEXT and IMAGE and instead check in the failing results the way we do with the non-chromium ports*. I don't like that we rely on bugs to track that the result is incorrect though, so my suggestion was that we change the filename to indicate it. So, foo/bar-expected.txt becomes foo/bar-failing.txt and we just add the -failing version to the fallback order. The main thing I like about this approach is that it allows you to still have a clear list of failing tests that need fixing. I believe that with the current model of checking in a failing result and filing a bug, failing tests are forgotten about. Ojan * My original proposal to Chromium folk wanted to get rid of TEXT and IMAGE entirely from the expectations format. It was generally well received except it it makes handling certain temporary failures considerably more difficult (e.g. pulling in a new version of Skia). On Fri, Jul 1, 2011 at 11:09 AM, Adam Barth aba...@webkit.org wrote: You can do the same thing with NRWT that you can do with ORWT in this regard, but nothing new. The test_expectation.txt file does give you more fine-grained control than Skipped in the sense that you can distinguish between TEXT, IMAGE, CRASH, and TIMEOUT failures, but it doesn't let you distinguish between different sorts of TEXT failures, for example. My sense is that the test_expectation.txt file is already somewhat over complicated for the problem it solves. In this case, the workflow of changing the expected results and filing a bug to track the failure seems like a reasonable solution, especially if there's a keyword or master bug that lets you find all these bugs easily. Adam On Fri, Jul 1, 2011 at 10:58 AM, Adam Roben aro...@apple.com wrote: When a test starts failing on a bot that uses old-run-webkit-tests, we typically check in expected failure results for that test (e.g., http://trac.webkit.org/changeset/90235). That way we can find out if the test's behavior changes in the future even though the test is still failing. This is particularly useful for tests that are actually testing multiple things at once. (Maybe they should be broken up into multiple tests, but that's a different discussion.) Is there a way to achieve this with new-run-webkit-tests? I know that you can mark a test as an expected failure (either a text diff, or an image diff, or both). Does it let you specify that the test should fail in a particular way? ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?
I like the idea of -failing. But what happens when you have both -failing and -expected in the same directory? Are either accepted? (in which case it's like a file-system version of test-expetations flaky lists) On Fri, Jul 1, 2011 at 12:49 PM, Ojan Vafai o...@chromium.org wrote: I proposed a while back to chromium folk that we minimize the use of TEXT and IMAGE and instead check in the failing results the way we do with the non-chromium ports*. I don't like that we rely on bugs to track that the result is incorrect though, so my suggestion was that we change the filename to indicate it. So, foo/bar-expected.txt becomes foo/bar-failing.txt and we just add the -failing version to the fallback order. The main thing I like about this approach is that it allows you to still have a clear list of failing tests that need fixing. I believe that with the current model of checking in a failing result and filing a bug, failing tests are forgotten about. Ojan * My original proposal to Chromium folk wanted to get rid of TEXT and IMAGE entirely from the expectations format. It was generally well received except it it makes handling certain temporary failures considerably more difficult (e.g. pulling in a new version of Skia). On Fri, Jul 1, 2011 at 11:09 AM, Adam Barth aba...@webkit.org wrote: You can do the same thing with NRWT that you can do with ORWT in this regard, but nothing new. The test_expectation.txt file does give you more fine-grained control than Skipped in the sense that you can distinguish between TEXT, IMAGE, CRASH, and TIMEOUT failures, but it doesn't let you distinguish between different sorts of TEXT failures, for example. My sense is that the test_expectation.txt file is already somewhat over complicated for the problem it solves. In this case, the workflow of changing the expected results and filing a bug to track the failure seems like a reasonable solution, especially if there's a keyword or master bug that lets you find all these bugs easily. Adam On Fri, Jul 1, 2011 at 10:58 AM, Adam Roben aro...@apple.com wrote: When a test starts failing on a bot that uses old-run-webkit-tests, we typically check in expected failure results for that test (e.g., http://trac.webkit.org/changeset/90235). That way we can find out if the test's behavior changes in the future even though the test is still failing. This is particularly useful for tests that are actually testing multiple things at once. (Maybe they should be broken up into multiple tests, but that's a different discussion.) Is there a way to achieve this with new-run-webkit-tests? I know that you can mark a test as an expected failure (either a text diff, or an image diff, or both). Does it let you specify that the test should fail in a particular way? ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?
On Fri, Jul 1, 2011 at 12:49 PM, Ojan Vafai o...@chromium.org wrote: I proposed a while back to chromium folk that we minimize the use of TEXT and IMAGE and instead check in the failing results the way we do with the non-chromium ports*. I don't like that we rely on bugs to track that the result is incorrect though, so my suggestion was that we change the filename to indicate it. So, foo/bar-expected.txt becomes foo/bar-failing.txt and we just add the -failing version to the fallback order. One concern I have is that this might increase the size of the already-massive LayoutTest directory considerably at least for Chromium port. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?
-failing should trump -expected. I also like Ojan's idea. I do not believe that -expected should be used to track incorrect results, because that makes understanding how tests are supposed to run dependent on the knowledge of the bug database as well. I think Ryosuke's concern is legitimate, both out of concern for Chromium's long list of failures and for what would happen if other ports started also running pixel tests, but I don't know if it's a big enough concern to kill the idea. It would be interesting to see how big of an impact there is, and, obviously, a given port could chose not to use -failure files if it didn't want to. -- Dirk On Fri, Jul 1, 2011 at 12:56 PM, Eric Seidel e...@webkit.org wrote: I like the idea of -failing. But what happens when you have both -failing and -expected in the same directory? Are either accepted? (in which case it's like a file-system version of test-expetations flaky lists) On Fri, Jul 1, 2011 at 12:49 PM, Ojan Vafai o...@chromium.org wrote: I proposed a while back to chromium folk that we minimize the use of TEXT and IMAGE and instead check in the failing results the way we do with the non-chromium ports*. I don't like that we rely on bugs to track that the result is incorrect though, so my suggestion was that we change the filename to indicate it. So, foo/bar-expected.txt becomes foo/bar-failing.txt and we just add the -failing version to the fallback order. The main thing I like about this approach is that it allows you to still have a clear list of failing tests that need fixing. I believe that with the current model of checking in a failing result and filing a bug, failing tests are forgotten about. Ojan * My original proposal to Chromium folk wanted to get rid of TEXT and IMAGE entirely from the expectations format. It was generally well received except it it makes handling certain temporary failures considerably more difficult (e.g. pulling in a new version of Skia). On Fri, Jul 1, 2011 at 11:09 AM, Adam Barth aba...@webkit.org wrote: You can do the same thing with NRWT that you can do with ORWT in this regard, but nothing new. The test_expectation.txt file does give you more fine-grained control than Skipped in the sense that you can distinguish between TEXT, IMAGE, CRASH, and TIMEOUT failures, but it doesn't let you distinguish between different sorts of TEXT failures, for example. My sense is that the test_expectation.txt file is already somewhat over complicated for the problem it solves. In this case, the workflow of changing the expected results and filing a bug to track the failure seems like a reasonable solution, especially if there's a keyword or master bug that lets you find all these bugs easily. Adam On Fri, Jul 1, 2011 at 10:58 AM, Adam Roben aro...@apple.com wrote: When a test starts failing on a bot that uses old-run-webkit-tests, we typically check in expected failure results for that test (e.g., http://trac.webkit.org/changeset/90235). That way we can find out if the test's behavior changes in the future even though the test is still failing. This is particularly useful for tests that are actually testing multiple things at once. (Maybe they should be broken up into multiple tests, but that's a different discussion.) Is there a way to achieve this with new-run-webkit-tests? I know that you can mark a test as an expected failure (either a text diff, or an image diff, or both). Does it let you specify that the test should fail in a particular way? ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?
So then you'd never want both in the same directory, doing so shoudl be an error? -eric On Fri, Jul 1, 2011 at 2:18 PM, Ojan Vafai o...@chromium.org wrote: What Dirk said. It's just adding another layer into the fallback order. On Fri, Jul 1, 2011 at 1:54 PM, Dirk Pranke dpra...@chromium.org wrote: -failing should trump -expected. I also like Ojan's idea. I do not believe that -expected should be used to track incorrect results, because that makes understanding how tests are supposed to run dependent on the knowledge of the bug database as well. I think Ryosuke's concern is legitimate, both out of concern for Chromium's long list of failures and for what would happen if other ports started also running pixel tests, but I don't know if it's a big enough concern to kill the idea. It would be interesting to see how big of an impact there is, and, obviously, a given port could chose not to use -failure files if it didn't want to. -- Dirk On Fri, Jul 1, 2011 at 12:56 PM, Eric Seidel e...@webkit.org wrote: I like the idea of -failing. But what happens when you have both -failing and -expected in the same directory? Are either accepted? (in which case it's like a file-system version of test-expetations flaky lists) On Fri, Jul 1, 2011 at 12:49 PM, Ojan Vafai o...@chromium.org wrote: I proposed a while back to chromium folk that we minimize the use of TEXT and IMAGE and instead check in the failing results the way we do with the non-chromium ports*. I don't like that we rely on bugs to track that the result is incorrect though, so my suggestion was that we change the filename to indicate it. So, foo/bar-expected.txt becomes foo/bar-failing.txt and we just add the -failing version to the fallback order. The main thing I like about this approach is that it allows you to still have a clear list of failing tests that need fixing. I believe that with the current model of checking in a failing result and filing a bug, failing tests are forgotten about. Ojan * My original proposal to Chromium folk wanted to get rid of TEXT and IMAGE entirely from the expectations format. It was generally well received except it it makes handling certain temporary failures considerably more difficult (e.g. pulling in a new version of Skia). On Fri, Jul 1, 2011 at 11:09 AM, Adam Barth aba...@webkit.org wrote: You can do the same thing with NRWT that you can do with ORWT in this regard, but nothing new. The test_expectation.txt file does give you more fine-grained control than Skipped in the sense that you can distinguish between TEXT, IMAGE, CRASH, and TIMEOUT failures, but it doesn't let you distinguish between different sorts of TEXT failures, for example. My sense is that the test_expectation.txt file is already somewhat over complicated for the problem it solves. In this case, the workflow of changing the expected results and filing a bug to track the failure seems like a reasonable solution, especially if there's a keyword or master bug that lets you find all these bugs easily. Adam On Fri, Jul 1, 2011 at 10:58 AM, Adam Roben aro...@apple.com wrote: When a test starts failing on a bot that uses old-run-webkit-tests, we typically check in expected failure results for that test (e.g., http://trac.webkit.org/changeset/90235). That way we can find out if the test's behavior changes in the future even though the test is still failing. This is particularly useful for tests that are actually testing multiple things at once. (Maybe they should be broken up into multiple tests, but that's a different discussion.) Is there a way to achieve this with new-run-webkit-tests? I know that you can mark a test as an expected failure (either a text diff, or an image diff, or both). Does it let you specify that the test should fail in a particular way? ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?
What Dirk said. It's just adding another layer into the fallback order. On Fri, Jul 1, 2011 at 1:54 PM, Dirk Pranke dpra...@chromium.org wrote: -failing should trump -expected. I also like Ojan's idea. I do not believe that -expected should be used to track incorrect results, because that makes understanding how tests are supposed to run dependent on the knowledge of the bug database as well. I think Ryosuke's concern is legitimate, both out of concern for Chromium's long list of failures and for what would happen if other ports started also running pixel tests, but I don't know if it's a big enough concern to kill the idea. It would be interesting to see how big of an impact there is, and, obviously, a given port could chose not to use -failure files if it didn't want to. -- Dirk On Fri, Jul 1, 2011 at 12:56 PM, Eric Seidel e...@webkit.org wrote: I like the idea of -failing. But what happens when you have both -failing and -expected in the same directory? Are either accepted? (in which case it's like a file-system version of test-expetations flaky lists) On Fri, Jul 1, 2011 at 12:49 PM, Ojan Vafai o...@chromium.org wrote: I proposed a while back to chromium folk that we minimize the use of TEXT and IMAGE and instead check in the failing results the way we do with the non-chromium ports*. I don't like that we rely on bugs to track that the result is incorrect though, so my suggestion was that we change the filename to indicate it. So, foo/bar-expected.txt becomes foo/bar-failing.txt and we just add the -failing version to the fallback order. The main thing I like about this approach is that it allows you to still have a clear list of failing tests that need fixing. I believe that with the current model of checking in a failing result and filing a bug, failing tests are forgotten about. Ojan * My original proposal to Chromium folk wanted to get rid of TEXT and IMAGE entirely from the expectations format. It was generally well received except it it makes handling certain temporary failures considerably more difficult (e.g. pulling in a new version of Skia). On Fri, Jul 1, 2011 at 11:09 AM, Adam Barth aba...@webkit.org wrote: You can do the same thing with NRWT that you can do with ORWT in this regard, but nothing new. The test_expectation.txt file does give you more fine-grained control than Skipped in the sense that you can distinguish between TEXT, IMAGE, CRASH, and TIMEOUT failures, but it doesn't let you distinguish between different sorts of TEXT failures, for example. My sense is that the test_expectation.txt file is already somewhat over complicated for the problem it solves. In this case, the workflow of changing the expected results and filing a bug to track the failure seems like a reasonable solution, especially if there's a keyword or master bug that lets you find all these bugs easily. Adam On Fri, Jul 1, 2011 at 10:58 AM, Adam Roben aro...@apple.com wrote: When a test starts failing on a bot that uses old-run-webkit-tests, we typically check in expected failure results for that test (e.g., http://trac.webkit.org/changeset/90235). That way we can find out if the test's behavior changes in the future even though the test is still failing. This is particularly useful for tests that are actually testing multiple things at once. (Maybe they should be broken up into multiple tests, but that's a different discussion.) Is there a way to achieve this with new-run-webkit-tests? I know that you can mark a test as an expected failure (either a text diff, or an image diff, or both). Does it let you specify that the test should fail in a particular way? ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?
No. Leave the -expected test alone with the correct but different on this platform expectation (of course, if the platform matches the generic version or some upstream version, you don't need an -expected.txt file at all). -- Dirk On Fri, Jul 1, 2011 at 2:21 PM, Eric Seidel e...@webkit.org wrote: So then you'd never want both in the same directory, doing so shoudl be an error? -eric On Fri, Jul 1, 2011 at 2:18 PM, Ojan Vafai o...@chromium.org wrote: What Dirk said. It's just adding another layer into the fallback order. On Fri, Jul 1, 2011 at 1:54 PM, Dirk Pranke dpra...@chromium.org wrote: -failing should trump -expected. I also like Ojan's idea. I do not believe that -expected should be used to track incorrect results, because that makes understanding how tests are supposed to run dependent on the knowledge of the bug database as well. I think Ryosuke's concern is legitimate, both out of concern for Chromium's long list of failures and for what would happen if other ports started also running pixel tests, but I don't know if it's a big enough concern to kill the idea. It would be interesting to see how big of an impact there is, and, obviously, a given port could chose not to use -failure files if it didn't want to. -- Dirk On Fri, Jul 1, 2011 at 12:56 PM, Eric Seidel e...@webkit.org wrote: I like the idea of -failing. But what happens when you have both -failing and -expected in the same directory? Are either accepted? (in which case it's like a file-system version of test-expetations flaky lists) On Fri, Jul 1, 2011 at 12:49 PM, Ojan Vafai o...@chromium.org wrote: I proposed a while back to chromium folk that we minimize the use of TEXT and IMAGE and instead check in the failing results the way we do with the non-chromium ports*. I don't like that we rely on bugs to track that the result is incorrect though, so my suggestion was that we change the filename to indicate it. So, foo/bar-expected.txt becomes foo/bar-failing.txt and we just add the -failing version to the fallback order. The main thing I like about this approach is that it allows you to still have a clear list of failing tests that need fixing. I believe that with the current model of checking in a failing result and filing a bug, failing tests are forgotten about. Ojan * My original proposal to Chromium folk wanted to get rid of TEXT and IMAGE entirely from the expectations format. It was generally well received except it it makes handling certain temporary failures considerably more difficult (e.g. pulling in a new version of Skia). On Fri, Jul 1, 2011 at 11:09 AM, Adam Barth aba...@webkit.org wrote: You can do the same thing with NRWT that you can do with ORWT in this regard, but nothing new. The test_expectation.txt file does give you more fine-grained control than Skipped in the sense that you can distinguish between TEXT, IMAGE, CRASH, and TIMEOUT failures, but it doesn't let you distinguish between different sorts of TEXT failures, for example. My sense is that the test_expectation.txt file is already somewhat over complicated for the problem it solves. In this case, the workflow of changing the expected results and filing a bug to track the failure seems like a reasonable solution, especially if there's a keyword or master bug that lets you find all these bugs easily. Adam On Fri, Jul 1, 2011 at 10:58 AM, Adam Roben aro...@apple.com wrote: When a test starts failing on a bot that uses old-run-webkit-tests, we typically check in expected failure results for that test (e.g., http://trac.webkit.org/changeset/90235). That way we can find out if the test's behavior changes in the future even though the test is still failing. This is particularly useful for tests that are actually testing multiple things at once. (Maybe they should be broken up into multiple tests, but that's a different discussion.) Is there a way to achieve this with new-run-webkit-tests? I know that you can mark a test as an expected failure (either a text diff, or an image diff, or both). Does it let you specify that the test should fail in a particular way? ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev
Re: [webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?
On Jul 1, 2011, at 1:54 PM, Dirk Pranke wrote: I do not believe that -expected should be used to track incorrect results A huge percentage of our expected results are incorrect in one sense or another. It’s a massive project to rename them all to reflect the fact that they reflect some amount of failure rather than total success. The project may be worthwhile. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?
On Fri, Jul 1, 2011 at 2:38 PM, Darin Adler da...@apple.com wrote: On Jul 1, 2011, at 1:54 PM, Dirk Pranke wrote: I do not believe that -expected should be used to track incorrect results A huge percentage of our expected results are incorrect in one sense or another. It’s a massive project to rename them all to reflect the fact that they reflect some amount of failure rather than total success. The project may be worthwhile. Really? Fascinating. Does that apply to -expected.txt files in the base directories, or just platform-specific exceptions? I wonder how it is that I've been working (admittedly, mostly on tooling) in WebKit for more that two years and this is the first I'm hearing about this. It seems like we're making our jobs much harder by doing this. Are there reasons we doing things this way apart from not having a solution along the lines of something like we've been talking about in this thread? E.g., a test's results on given port may differ from what might be correct, but the port may not want to fix this (chromium often uses WONTFIX in the test_expectations.txt for this) seems like a sensible reason, but, assuming you run the test at all, I would've prefer that to be tracked by a -failure.txt sort of solution. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?
On Jul 1, 2011, at 2:54 PM, Dirk Pranke wrote: Does that apply to -expected.txt files in the base directories, or just platform-specific exceptions? Base directories. Expected files contain output reflecting the behavior of WebKit at the time the test was checked in. The expected result when we re-run a test. Many expected files contain text that says “FAIL” in them. The fact that these expected results are not successes, but rather expected failures does not seem to me to be a subtle point, but one of the basic things about how these tests are set up. I wonder how it is that I've been working (admittedly, mostly on tooling) in WebKit for more that two years and this is the first I'm hearing about this. I’m guessing it’s because you have been working on Chrome. The Chrome project came up with a different system for testing layered on top of the original layout test machinery based on different concepts. I don’t think anyone ever discussed that system with me; I was the one who created the original layout test system, to help Dave Hyatt originally, and then later the rest of the team started using it. Are there reasons we [are] doing things this way[?] Sure. The idea of the layout test framework is to check if the code is still behaving as it did when the test was created and last run; we want to detect any changes in behavior that are not expected. When there are expected changes in behavior, we change the contents of the expected results files. It seems possibly helpful to augment the test system with editorial comments about which tests show bugs that we’d want to fix. But I wouldn’t want to stop running all regression tests where the output reflects the effects of a bug or missing feature. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?
On Fri, Jul 1, 2011 at 3:04 PM, Darin Adler da...@apple.com wrote: On Jul 1, 2011, at 2:54 PM, Dirk Pranke wrote: Does that apply to -expected.txt files in the base directories, or just platform-specific exceptions? Base directories. Expected files contain output reflecting the behavior of WebKit at the time the test was checked in. The expected result when we re-run a test. Many expected files contain text that says “FAIL” in them. The fact that these expected results are not successes, but rather expected failures does not seem to me to be a subtle point, but one of the basic things about how these tests are set up. Right, it helps us keep track of where we are, so that we don't regress, and only make forward progress. I wonder how it is that I've been working (admittedly, mostly on tooling) in WebKit for more that two years and this is the first I'm hearing about this. I’m guessing it’s because you have been working on Chrome. The Chrome project came up with a different system for testing layered on top of the original layout test machinery based on different concepts. I don’t think anyone ever discussed that system with me; I was the one who created the original layout test system, to help Dave Hyatt originally, and then later the rest of the team started using it. The granular annotations (more than just SKIP) in test_expectations.txt was something we introduced back when Chrome was failing a large percentage of layout tests, and we needed a system to help us triage the failures. It was useful to distinguish tests that crash from tests that generate bad results, for example. We then focused on the crashing tests first. In addition, we wanted to understand how divergent we were from the standard WebKit port, and we wanted to know if we were failing to match text results or just image results. This allowed us to measure our degree of incompatibility with standard WebKit. We basically used this mechanism to classify differences that mattered and differences that didn't matter. I think that if we had just checked in a bunch of port-specific failure expectations as -expected files, then we would have had a hard time distinguishing failures we needed to fix for compat reasons from failures that were expected (e.g., because we have different looking form controls). I'm not sure if we are at a point now where this mechanism isn't useful, but I kind of suspect that it will always be useful. Afterall, it is not uncommon for a code change to result in different rendering behavior between the ports. I think it is valuable to have a measure of divergence between the various WebKit ports. We want to minimize such divergence from a web compat point of view, of course. Maybe the count of SKIPPED tests is enough? But, then we suffer from not running the tests at all. At least by annotating expected IMAGE failures, we get to know that the TEXT output is the same and that we don't expect a CRASH. I suspect this isn't the best solution to the problem though. -Darin Are there reasons we [are] doing things this way[?] Sure. The idea of the layout test framework is to check if the code is still behaving as it did when the test was created and last run; we want to detect any changes in behavior that are not expected. When there are expected changes in behavior, we change the contents of the expected results files. It seems possibly helpful to augment the test system with editorial comments about which tests show bugs that we’d want to fix. But I wouldn’t want to stop running all regression tests where the output reflects the effects of a bug or missing feature. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?
On Fri, Jul 1, 2011 at 3:24 PM, Darin Fisher da...@chromium.org wrote: On Fri, Jul 1, 2011 at 3:04 PM, Darin Adler da...@apple.com wrote: On Jul 1, 2011, at 2:54 PM, Dirk Pranke wrote: Does that apply to -expected.txt files in the base directories, or just platform-specific exceptions? Base directories. Expected files contain output reflecting the behavior of WebKit at the time the test was checked in. The expected result when we re-run a test. Many expected files contain text that says “FAIL” in them. The fact that these expected results are not successes, but rather expected failures does not seem to me to be a subtle point, but one of the basic things about how these tests are set up. Right, it helps us keep track of where we are, so that we don't regress, and only make forward progress. I wonder how it is that I've been working (admittedly, mostly on tooling) in WebKit for more that two years and this is the first I'm hearing about this. I’m guessing it’s because you have been working on Chrome. The Chrome project came up with a different system for testing layered on top of the original layout test machinery based on different concepts. I don’t think anyone ever discussed that system with me; I was the one who created the original layout test system, to help Dave Hyatt originally, and then later the rest of the team started using it. The granular annotations (more than just SKIP) in test_expectations.txt was something we introduced back when Chrome was failing a large percentage of layout tests, and we needed a system to help us triage the failures. It was useful to distinguish tests that crash from tests that generate bad results, for example. We then focused on the crashing tests first. In addition, we wanted to understand how divergent we were from the standard WebKit port, and we wanted to know if we were failing to match text results or just image results. This allowed us to measure our degree of incompatibility with standard WebKit. We basically used this mechanism to classify differences that mattered and differences that didn't matter. I think that if we had just checked in a bunch of port-specific failure expectations as -expected files, then we would have had a hard time distinguishing failures we needed to fix for compat reasons from failures that were expected (e.g., because we have different looking form controls). I'm not sure if we are at a point now where this mechanism isn't useful, but I kind of suspect that it will always be useful. Afterall, it is not uncommon for a code change to result in different rendering behavior between the ports. I think it is valuable to have a measure of divergence between the various WebKit ports. We want to minimize such divergence from a web compat point of view, of course. Maybe the count of SKIPPED tests is enough? But, then we suffer from not running the tests at all. At least by annotating expected IMAGE failures, we get to know that the TEXT output is the same and that we don't expect a CRASH. There's at least two reasons for divergence .. one is that the port is actually doing the wrong thing, and the other is that the port is doing the right thing but the output is different anyway (e.g., a control is rendered differently). We cannot easily separate the two if we have only a single convention (platform-specific -expected files), but SKIPPING tests seems wrong for either category. It seems like -failing gives you the control you would want, no? Obviously, it wouldn't help the thousands of -expected files that are wrong but at least it could keep things from getting worse. I will note that reftests might solve some issues but not all of them (since obviously code could render both pages wrong). -- Dirk I suspect this isn't the best solution to the problem though. -Darin Are there reasons we [are] doing things this way[?] Sure. The idea of the layout test framework is to check if the code is still behaving as it did when the test was created and last run; we want to detect any changes in behavior that are not expected. When there are expected changes in behavior, we change the contents of the expected results files. It seems possibly helpful to augment the test system with editorial comments about which tests show bugs that we’d want to fix. But I wouldn’t want to stop running all regression tests where the output reflects the effects of a bug or missing feature. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?
On Fri, Jul 1, 2011 at 3:37 PM, Dirk Pranke dpra...@chromium.org wrote: On Fri, Jul 1, 2011 at 3:24 PM, Darin Fisher da...@chromium.org wrote: On Fri, Jul 1, 2011 at 3:04 PM, Darin Adler da...@apple.com wrote: On Jul 1, 2011, at 2:54 PM, Dirk Pranke wrote: Does that apply to -expected.txt files in the base directories, or just platform-specific exceptions? Base directories. Expected files contain output reflecting the behavior of WebKit at the time the test was checked in. The expected result when we re-run a test. Many expected files contain text that says “FAIL” in them. The fact that these expected results are not successes, but rather expected failures does not seem to me to be a subtle point, but one of the basic things about how these tests are set up. Right, it helps us keep track of where we are, so that we don't regress, and only make forward progress. I wonder how it is that I've been working (admittedly, mostly on tooling) in WebKit for more that two years and this is the first I'm hearing about this. I’m guessing it’s because you have been working on Chrome. The Chrome project came up with a different system for testing layered on top of the original layout test machinery based on different concepts. I don’t think anyone ever discussed that system with me; I was the one who created the original layout test system, to help Dave Hyatt originally, and then later the rest of the team started using it. The granular annotations (more than just SKIP) in test_expectations.txt was something we introduced back when Chrome was failing a large percentage of layout tests, and we needed a system to help us triage the failures. It was useful to distinguish tests that crash from tests that generate bad results, for example. We then focused on the crashing tests first. In addition, we wanted to understand how divergent we were from the standard WebKit port, and we wanted to know if we were failing to match text results or just image results. This allowed us to measure our degree of incompatibility with standard WebKit. We basically used this mechanism to classify differences that mattered and differences that didn't matter. I think that if we had just checked in a bunch of port-specific failure expectations as -expected files, then we would have had a hard time distinguishing failures we needed to fix for compat reasons from failures that were expected (e.g., because we have different looking form controls). I'm not sure if we are at a point now where this mechanism isn't useful, but I kind of suspect that it will always be useful. Afterall, it is not uncommon for a code change to result in different rendering behavior between the ports. I think it is valuable to have a measure of divergence between the various WebKit ports. We want to minimize such divergence from a web compat point of view, of course. Maybe the count of SKIPPED tests is enough? But, then we suffer from not running the tests at all. At least by annotating expected IMAGE failures, we get to know that the TEXT output is the same and that we don't expect a CRASH. There's at least two reasons for divergence .. one is that the port is actually doing the wrong thing, and the other is that the port is doing the right thing but the output is different anyway (e.g., a control is rendered differently). We cannot easily separate the two if we have only a single convention (platform-specific -expected files), but SKIPPING tests seems wrong for either category. It seems like -failing gives you the control you would want, no? Obviously, it wouldn't help the thousands of -expected files that are wrong but at least it could keep things from getting worse. I will note that reftests might solve some issues but not all of them (since obviously code could render both pages wrong). -- Dirk I'm not sure. It makes me a bit uneasy adding even more heft to the LayoutTests directory. -Darin I suspect this isn't the best solution to the problem though. -Darin Are there reasons we [are] doing things this way[?] Sure. The idea of the layout test framework is to check if the code is still behaving as it did when the test was created and last run; we want to detect any changes in behavior that are not expected. When there are expected changes in behavior, we change the contents of the expected results files. It seems possibly helpful to augment the test system with editorial comments about which tests show bugs that we’d want to fix. But I wouldn’t want to stop running all regression tests where the output reflects the effects of a bug or missing feature. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?
On Jul 1, 2011, at 3:24 PM, Darin Fisher wrote: The granular annotations (more than just SKIP) in test_expectations.txt was something we introduced back when Chrome was failing a large percentage of layout tests, and we needed a system to help us triage the failures. I do think that a system more sophisticated than our original one is a good idea. Generally speaking, I can only think of one reason to consider not running a test at all: 1) The test causes a crash or runs excessively slow as part of how it fails, so it makes running tests unwieldy. And only two reasons to not compare the results of the test with the expected output: 2) The test fails in unpredictably different ways so has no value as a regression test. 3) There are no expected results for the test; no one has generated them yet, or there is some reason to not check them in. None of this has anything to with success or failure, in my opinion. I think it’s great to be organized about the tests that are run. It would be wonderful to associate bugs with each set of known failures in tests, whether the failures are cross-platform or platform-specific. But I don’t think they way the test is run and compared with expected output should have anything to do with whether it’s an expected success or an expected failure. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Switching to new-run-webkit-tests
Thanks for your patience with the disruptions on the tree today. The bots are having some trouble with the HTTP lock that don't manifest locally or on our test slave. I've turned NRWT back off while we try to sort that out. Thanks, Adam On Fri, Jul 1, 2011 at 12:49 AM, Adam Barth aba...@webkit.org wrote: Sorry, no switch over tonight. The WebKit2 build is too destroyed to have any confidence that we're doing things correctly and the Snow Leopard buildbot appears to have an errant HTTP server bound to port 8080. :( Adam On Thu, Jun 30, 2011 at 11:47 AM, Adam Barth aba...@webkit.org wrote: According to https://bugs.webkit.org/show_bug.cgi?id=34984, there are only two remaining blocking issues: 1) Teaching new-run-webkit-tests to understand the difference between WebProcess crashes and other sorts of crashes in WebKit2. 2) Fixing some issues with the Apple Windows port. Once we get (1) fixed (hopefully today), we're going to start moving (non-Windows) ports over to new-run-webkit-tests (hopefully tonight). I suspect more issues will crop up once we actually flip the switch. Please do not hesitate to contact Eric or myself (either via #webkit or via bugs.webkit.org). We'll try to respond to any issues that arise as quickly as possible. Thanks for your patience. Adam On Mon, Jun 27, 2011 at 4:26 PM, Xan Lopez x...@gnome.org wrote: On Tue, Jun 28, 2011 at 1:17 AM, Eric Seidel e...@webkit.org wrote: There appear to be 6 remaining blocking issues: https://bugs.webkit.org/showdependencytree.cgi?id=34984hide_resolved=1 We would like to hear from others who have tried new-run-webkit-tests, if they have issues which they believe should block migrating to NRWT. (If so, please file and block the master bug!) I can see the GTK+ port thing with Xvfb is there already, so not a lot to add. NWRT is more sensitive to slow tests than the old infrastructure, so we had to add a bunch of them to the expectations file; I don't think this is particularly bad. In any case with the Xvfb patch and my local expectations file things run beautifully and way faster than before, so looking great from our side! Xan Thanks. -eric ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
