Re: [webkit-dev] builds pending on cr-linux bot
That builder just had a stuck build. I canceled the build, and things seem better. I believe I also fixed the bug which was hanging the commit-queue and chromium-linux EWS bots. Those should return to normal by morning (just in time for the rush!). -eric On Sun, Apr 8, 2012 at 10:56 PM, Eric Seidel e...@webkit.org wrote: Unclear what's going on with that builder. The CommitQueue also seems unhappy. Will poke around (I don't have access to the build bots, but I can look at the EWS/CQ.) On Sun, Apr 8, 2012 at 8:02 PM, Sravan sra1sand...@gmail.com wrote: Hi, I've submitted a patch almost 2 days ago, and i've seen that its still in queue. From http://build.webkit.org/builders/Chromium%20Linux%20Release i am seeing that builds are pending almost from 79hours. Can some one re-start/kick the bot to get the builds going. -Sravan. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] builds pending on cr-linux bot
Yes Thanks, its running now. -Sravan. On Mon, Apr 9, 2012 at 1:49 PM, Eric Seidel e...@webkit.org wrote: That builder just had a stuck build. I canceled the build, and things seem better. I believe I also fixed the bug which was hanging the commit-queue and chromium-linux EWS bots. Those should return to normal by morning (just in time for the rush!). -eric On Sun, Apr 8, 2012 at 10:56 PM, Eric Seidel e...@webkit.org wrote: Unclear what's going on with that builder. The CommitQueue also seems unhappy. Will poke around (I don't have access to the build bots, but I can look at the EWS/CQ.) On Sun, Apr 8, 2012 at 8:02 PM, Sravan sra1sand...@gmail.com wrote: Hi, I've submitted a patch almost 2 days ago, and i've seen that its still in queue. From http://build.webkit.org/builders/Chromium%20Linux%20Release i am seeing that builds are pending almost from 79hours. Can some one re-start/kick the bot to get the builds going. -Sravan. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] check-webkit-style should remind folks to update the results for run-bindings-tests
Hi All, It is observed that if changes are made in Codegenerator*.pm we need to rebase results of run-bindings-tests. Many times authors forgot to update these binding results. IMO we should add check in ./check-webkit-style to warn/(give a chance) author to run run-bindings-tests if Codegenerator is modified. I have filed bug for this https://bugs.webkit.org/show_bug.cgi?id=83354 . Please let me know if any suggestions to make this change. Thanks, Vineet ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] check-webkit-style should remind folks to update the results for run-bindings-tests
IMO we should add check in ./check-webkit-style to warn/(give a chance) author to run run-bindings-tests if Codegenerator is modified. +1 to the idea. I think that it would make sense to warn if any file under WebCore/bindings/ is changed. Another idea might be to run ./run-bindings-tests in ./check-webkit-style (instead of warning), but it is too heavy (it takes 5~ seconds). run-bindings-tests are just for showing reviewers what changes the patch is going to make. Even if run-bindings-tests fail, it is not a serious problem. Thus, the build bots ignore the run-bindings-tests results. Consequently, people sometimes forget to update the results. I've been manually rebaselining run-bindings-tests results more than once a week:) On Mon, Apr 9, 2012 at 3:39 PM, Vineet Chaudhary rgf...@motorola.com wrote: Hi All, It is observed that if changes are made in Codegenerator*.pm we need to rebase results of run-bindings-tests. Many times authors forgot to update these binding results. IMO we should add check in ./check-webkit-style to warn/(give a chance) author to run run-bindings-tests if Codegenerator is modified. I have filed bug for this https://bugs.webkit.org/show_bug.cgi?id=83354 . Please let me know if any suggestions to make this change. Thanks, Vineet ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev -- Kentaro Hara, Tokyo, Japan (http://haraken.info) ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] check-webkit-style should remind folks to update the results for run-bindings-tests
[Sending from a correct mail address...] IMO we should add check in ./check-webkit-style to warn/(give a chance) author to run run-bindings-tests if Codegenerator is modified. +1 to the idea. I think that it would make sense to warn if any file under WebCore/bindings/ is changed. Another idea might be to run ./run-bindings-tests in ./check-webkit-style (instead of warning), but it is too heavy (it takes 5~ seconds). run-bindings-tests are just for showing reviewers what changes the patch is going to make. Even if run-bindings-tests fail, it is not a serious problem. Thus, the build bots ignore the run-bindings-tests results. Consequently, people sometimes forget to update the results. I've been manually rebaselining run-bindings-tests results more than once a week:) On Mon, Apr 9, 2012 at 3:39 PM, Vineet Chaudhary rgf...@motorola.com wrote: Hi All, It is observed that if changes are made in Codegenerator*.pm we need to rebase results of run-bindings-tests. Many times authors forgot to update these binding results. IMO we should add check in ./check-webkit-style to warn/(give a chance) author to run run-bindings-tests if Codegenerator is modified. I have filed bug for this https://bugs.webkit.org/show_bug.cgi?id=83354 . Please let me know if any suggestions to make this change. Thanks, Vineet ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev -- Kentaro Hara, Tokyo, Japan (http://haraken.info) -- Kentaro Hara, Tokyo, Japan (http://haraken.info) ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] check-webkit-style should remind folks to update the results for run-bindings-tests
We have discussed this before: https://lists.webkit.org/pipermail/webkit-dev/2011-September/017868.html A better change would be for us to generate the diff on EWS, and get rid of binding tests step from build bots since they aren't really testing anything. The primary use case of run-bindings-tests is to see the diff before/after a code generator change and it has no business running on the bot or in check-webkit-style. - Ryosuke On Mon, Apr 9, 2012 at 6:39 AM, Vineet Chaudhary rgf...@motorola.comwrote: Hi All, It is observed that if changes are made in Codegenerator*.pm we need to rebase results of run-bindings-tests. Many times authors forgot to update these binding results. IMO we should add check in ./check-webkit-style to warn/(give a chance) author to run run-bindings-tests if Codegenerator is modified. I have filed bug for this https://bugs.webkit.org/show_bug.cgi?id=83354 . Please let me know if any suggestions to make this change. Thanks, Vineet ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] check-webkit-style should remind folks to update the results for run-bindings-tests
Also note Maciej's proposal about moving it to a build step. Either way, it shouldn't be ran as a test step on build bots. - Ryosuke On Mon, Apr 9, 2012 at 9:18 AM, Ryosuke Niwa rn...@webkit.org wrote: We have discussed this before: https://lists.webkit.org/pipermail/webkit-dev/2011-September/017868.html A better change would be for us to generate the diff on EWS, and get rid of binding tests step from build bots since they aren't really testing anything. The primary use case of run-bindings-tests is to see the diff before/after a code generator change and it has no business running on the bot or in check-webkit-style. - Ryosuke On Mon, Apr 9, 2012 at 6:39 AM, Vineet Chaudhary rgf...@motorola.comwrote: Hi All, It is observed that if changes are made in Codegenerator*.pm we need to rebase results of run-bindings-tests. Many times authors forgot to update these binding results. IMO we should add check in ./check-webkit-style to warn/(give a chance) author to run run-bindings-tests if Codegenerator is modified. I have filed bug for this https://bugs.webkit.org/show_bug.cgi?id=83354 . Please let me know if any suggestions to make this change. Thanks, Vineet ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] check-webkit-style should remind folks to update the results for run-bindings-tests
[From the correct email] On Mon, Apr 9, 2012 at 12:26 PM, Jarred Nicholls jar...@sencha.com wrote: On Mon, Apr 9, 2012 at 12:18 PM, Ryosuke Niwa rn...@webkit.org wrote: We have discussed this before: https://lists.webkit.org/pipermail/webkit-dev/2011-September/017868.html A better change would be for us to generate the diff on EWS, and get rid of binding tests step from build bots since they aren't really testing anything. The primary use case of run-bindings-tests is to see the diff before/after a code generator change and it has no business running on the bot or in check-webkit-style. Either way, we concluded that running the binding tests is very quick and cheap, so wherever it is still effective to prevent regressions in code generation is fine by me; on an as-needed basis (EWS) is clearly ideal, rather than on all builds. - Ryosuke On Mon, Apr 9, 2012 at 6:39 AM, Vineet Chaudhary rgf...@motorola.comwrote: Hi All, It is observed that if changes are made in Codegenerator*.pm we need to rebase results of run-bindings-tests. Many times authors forgot to update these binding results. IMO we should add check in ./check-webkit-style to warn/(give a chance) author to run run-bindings-tests if Codegenerator is modified. I have filed bug for this https://bugs.webkit.org/show_bug.cgi?id=83354. Please let me know if any suggestions to make this change. Thanks, Vineet ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev -- *Sencha* Jarred Nicholls, Senior Software Architect @jarrednicholls http://twitter.com/jarrednicholls ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is it OK to remove Frame::setIsDisconnected() and isDisconnected() ?
On Wed, Apr 15, 2009 at 2:21 PM, Maciej Stachowiak m...@apple.com wrote: On Apr 15, 2009, at 1:29 PM, Sverrir Á. Berg wrote: Hi Adam, Thanks for the links. These are simply exposing the functions as a formal a API's. I understand that you typically don't want to change externally exposed API's but these can easily be stubbed out (or removed). I should have pointed out in my original email that I have tried to remove these API's and I can still run all the WebKit/Mac tests fine. So at least two things are missing (IMHO) - tests that verify that this functionality is working as intended and documentation to tell what that intent is. But this is only required if somebody is actually using these functions... The API (system-private API actually, or SPI as we call it) in question is used by Safari. I don't believe it would be safe to remove this flag. The intent of the WebFrame API is that you can make a frame act (from the point of view of content inside it) as if it is a top-level frame, even though it is actually a subframe. One case where this might be useful is if you wanted to build a custom browser-like UI partially using HTML, which could then load pages as subframes without exposing that fact. It is true and unfortunate that we don't have tests - the ObjC API does not have as much test coverage as features exposed to Web content. Pardon my thread necromancy, but is this SPI still used by Safari? A brief read through the code turns up tons of bugs related to this feature. Are these bugs important to fix? Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] run-webkit-tests as an uber-test-harness?
Hi all, The latest thread on run-bindings-tests has reminded me that a few months ago we discussed having one top-level script that would run *all* of the tests in the repository (perl tests, python tests, new-run-webkit-tests, unit tests, run-bindings-tests, etc.), but as far as I know, no one has actually ever stepped up to do such a thing. (discussion in https://lists.webkit.org/pipermail/webkit-dev/2011-September/017868.html ) I don't think writing this would be particularly time consuming, so if there is general agreement that this would be a good thing, I'd be happy to take a crack at it. Any thoughts? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] handling failing tests (test_expectations, Skipped files, etc.)
Hi all, Recently I've noticed more people making changes and adding test failure suppressions to various ports' test_expectations.txt files. This is great! However, I don't think we have an agreement over what the best practices are here, so I thought I'd list out what I thought they were, and others can comment / correct me as necessary: 1) Don't mix test_expectations.txt files and Skipped files. This is really confusing to everyone involved ... your port should use one or the other where possible. (*) 2) Entries in the test_expectations files should be short-lived. We should be checking in what we believe the current output is, *regardless of whether it's right or wrong*. A different way of putting this is that we are more interested in changes in behavior (regressions) rather than correctness. If you expect a test to be failing for a long time in the same consistent way, check in the failure and file a bug. Don't use test_expectations to suppress this. (**) 3) Don't use test_expectations.txt to suppress failures across a single cycle of the bot, just so you can gather updated baselines without the tree going red. While it might seem that you're doing tree maintainers a favor, in my experience this just makes things confusing and it's better to let the tree go red until you can actually rebaseline things. It's too easy to add a suppression for now and then forget about it. These rules are not set in stone, they're just what I try to do. If people think there are better ways of doing this, please speak up! If there is consensus, I'll update the wikis accordingly. Thanks! -- Dirk (*) I have an outstanding to-do to modify new-run-webkit-tests to a better way of tracking expectations to merge the inheritance/cascade aspects of Skipped files with the flexibility in types of failures that you get from expectations. Eventually we should have a mechanism that replaces both, but for now, we don't. See https://bugs.webkit.org/show_bug.cgi?id=83508 . (**) Note that Chromium has not historically worked this way -- we suppress failures rather than check in failing output -- but I believe many / most of the chromium gardeners have come to believe that this is not the way things should work and we'd be better off checking in the failing output instead. Note that it may make sense to do different things based on your ports' maturity, so you suppress things while bringing a port up, and stop suppressing once your port is stable. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] handling failing tests (test_expectations, Skipped files, etc.)
On Mon, Apr 9, 2012 at 3:02 PM, James Robinson jam...@google.com wrote: 3) Don't use test_expectations.txt to suppress failures across a single cycle of the bot, just so you can gather updated baselines without the tree going red. While it might seem that you're doing tree maintainers a favor, in my experience this just makes things confusing and it's better to let the tree go red until you can actually rebaseline things. It's too easy to add a suppression for now and then forget about it. How would you suggest someone contribute a patch that changes layout test results, if not by marking the test as failing? Both this and Dana's response are good questions which I don't have great answers for. Let me explain my thinking in a little more detail. In my ideal world, you would be able to get updated baselines *prior* to trying to land a patch. This is of course not really possible today for any test that fails on multiple ports with different results, as it's practically impossible to run more than a couple of ports by hand, and we don't have a bot infrastructure to help you here. In my closer-to-the-real-world ideal, the test_expectations.txt file would be *empty*, all of the ports would be using them, and then adding in suppressions for files you expect to fail in your patch would be a good signal. In practice, however, marking a test as failing has a few problems: 1) if you don't get the list of suppressions right, than what shows up at the bots can be confusing - you don't realize that more (or different) tests are failing than the ones that showed up. Also, you can see tests fail on some bots that you didn't account for that don't use the expectations at all (e.g., you suppressed Chromium correctly, but the Apple bots go red). 2) adding suppressions to test_expectations.txt increases contention on the test_expectations.txt file, confusing the gardeners / bot-watchers and making merges harder. It is practically difficult to tell the difference between I just added these lines as a temporary suppression and I added these lines as a suppression two weeks ago but then forgot about them, and as Ojan will tell you, you end up with a bunch of lines in the file that just have a comment saying just needs rebaselining, but you have no idea if those statements are still true or not. 3) the current tool-of-the-day for managing rebaselining, garden-o-matic, is much better suited to handling the unexpected failures on the bots rather than the expected failures you've introduced. That said, as Dana points out, my proposal makes it hard if not impossible for the EWS to work at all; I'm biased because I hardly ever post changes that are expected to introduce failures on the EWS bots that should still land. I don't know what to do about this, although perhaps moving to my near-ideal model might work. If there's consensus in the mean time that it is better on balance to check in suppressions, perhaps we can figure out a better way to do that. Maybe (shudder) a second test_expectations file? Or maybe it would be better to actually check in suppressions marked as REBASELINE (or something like that)? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is it OK to remove Frame::setIsDisconnected() and isDisconnected() ?
On Apr 9, 2012, at 12:27 PM, Adam Barth wrote: On Wed, Apr 15, 2009 at 2:21 PM, Maciej Stachowiak m...@apple.com wrote: On Apr 15, 2009, at 1:29 PM, Sverrir Á. Berg wrote: Hi Adam, Thanks for the links. These are simply exposing the functions as a formal a API's. I understand that you typically don't want to change externally exposed API's but these can easily be stubbed out (or removed). I should have pointed out in my original email that I have tried to remove these API's and I can still run all the WebKit/Mac tests fine. So at least two things are missing (IMHO) - tests that verify that this functionality is working as intended and documentation to tell what that intent is. But this is only required if somebody is actually using these functions... The API (system-private API actually, or SPI as we call it) in question is used by Safari. I don't believe it would be safe to remove this flag. The intent of the WebFrame API is that you can make a frame act (from the point of view of content inside it) as if it is a top-level frame, even though it is actually a subframe. One case where this might be useful is if you wanted to build a custom browser-like UI partially using HTML, which could then load pages as subframes without exposing that fact. It is true and unfortunate that we don't have tests - the ObjC API does not have as much test coverage as features exposed to Web content. Pardon my thread necromancy, but is this SPI still used by Safari? Yes. It is used by the Reader feature of Safari. A brief read through the code turns up tons of bugs related to this feature. Are these bugs important to fix? Hard to say, without knowing what the specific bugs are. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is it OK to remove Frame::setIsDisconnected() and isDisconnected() ?
On Mon, Apr 9, 2012 at 3:24 PM, Maciej Stachowiak m...@apple.com wrote: On Apr 9, 2012, at 12:27 PM, Adam Barth wrote: On Wed, Apr 15, 2009 at 2:21 PM, Maciej Stachowiak m...@apple.com wrote: On Apr 15, 2009, at 1:29 PM, Sverrir Á. Berg wrote: Hi Adam, Thanks for the links. These are simply exposing the functions as a formal a API's. I understand that you typically don't want to change externally exposed API's but these can easily be stubbed out (or removed). I should have pointed out in my original email that I have tried to remove these API's and I can still run all the WebKit/Mac tests fine. So at least two things are missing (IMHO) - tests that verify that this functionality is working as intended and documentation to tell what that intent is. But this is only required if somebody is actually using these functions... The API (system-private API actually, or SPI as we call it) in question is used by Safari. I don't believe it would be safe to remove this flag. The intent of the WebFrame API is that you can make a frame act (from the point of view of content inside it) as if it is a top-level frame, even though it is actually a subframe. One case where this might be useful is if you wanted to build a custom browser-like UI partially using HTML, which could then load pages as subframes without exposing that fact. It is true and unfortunate that we don't have tests - the ObjC API does not have as much test coverage as features exposed to Web content. Pardon my thread necromancy, but is this SPI still used by Safari? Yes. It is used by the Reader feature of Safari. A brief read through the code turns up tons of bugs related to this feature. Are these bugs important to fix? Hard to say, without knowing what the specific bugs are. Ok. I'll file the ones I noticed. Thanks! Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] handling failing tests (test_expectations, Skipped files, etc.)
On Monday 09 April 2012 22:42:33 Dirk Pranke wrote: Hi all, Recently I've noticed more people making changes and adding test failure suppressions to various ports' test_expectations.txt files. This is great! Hi Dirk, It would be good to have a page describing the right thing to do for each of the ports for each of the following situations: - Adding a new test with rendtree/pixel results - Changing the rendtree/pixel results for existing tests I think the correct practice will differ between ports as only the Chromium bots (AFAIK) check pixel results. So this has the unfortunate consquence that when you change the expected pixel result for a test that has pixel results on all platforms you end up with stale png results in the tree for those that you are not in a position either to generate yourself or persuade someone on #webkit to generate for you. Thanks, Robert ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] handling failing tests (test_expectations, Skipped files, etc.)
In my ideal world, you would be able to get updated baselines *prior* to trying to land a patch. This is of course not really possible today for any test that fails on multiple ports with different results, as it's practically impossible to run more than a couple of ports by hand, and we don't have a bot infrastructure to help you here. That would be the best outcome indeed, however that would more or less require all the EWS to be able to run the tests (including pixel tests for the platforms that enable them). 3) the current tool-of-the-day for managing rebaselining, garden-o-matic, is much better suited to handling the unexpected failures on the bots rather than the expected failures you've introduced. You could see it the other way. How could we make garden-o-matic handle newly added suppression better? Maybe some new sub-tool listing the newly added suppressions would help? Ignore the suppressions added XX hours ago? Your issue seems to be suppressions sticking into the tree not strictly suppressing in itself. Our current tool (garden-o-matic) handles failures a lot better than it handles suppressions. If there's consensus in the mean time that it is better on balance to check in suppressions, perhaps we can figure out a better way to do that. Maybe (shudder) a second test_expectations file? Or maybe it would be better to actually check in suppressions marked as REBASELINE (or something like that)? That sounds quirky as it involves maintaining 2 sets of files. From my perspective, saying that we should discard the EWS result and allow changes to get in WebKit trunk, knowing they will turn the bots red, is a bad proposal regardless of how you justify it. In the small delta where the bots are red, you can bet people will miss something else that breaks. Thanks, Julien ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] handling failing tests (test_expectations, Skipped files, etc.)
On Mon, Apr 9, 2012 at 3:50 PM, Julien Chaffraix jchaffr...@webkit.orgwrote: If there's consensus in the mean time that it is better on balance to check in suppressions, perhaps we can figure out a better way to do that. Maybe (shudder) a second test_expectations file? Or maybe it would be better to actually check in suppressions marked as REBASELINE (or something like that)? That sounds quirky as it involves maintaining 2 sets of files. From my perspective, saying that we should discard the EWS result and allow changes to get in WebKit trunk, knowing they will turn the bots red, is a bad proposal regardless of how you justify it. In the small delta where the bots are red, you can bet people will miss something else that breaks. But that's the status quo. Until we get to the ideal world where every port's EWS at least runs tests, there's no way around it since we can't possibly require all contributors to run tests on all ports and platforms. Also, adding new lines to test_expectations.txt will only work on Chromium port since other ports use Skipped file instead, and skipping the test won't give you new baseline on the bot. And this discrepancy between Chromium and non-Chromium ports imposes a significant cognitive stress on contributors and is not justifiable in my opinion. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] handling failing tests (test_expectations, Skipped files, etc.)
On Mon, Apr 9, 2012 at 3:50 PM, Julien Chaffraix jchaffr...@webkit.org wrote: In my ideal world, you would be able to get updated baselines *prior* to trying to land a patch. This is of course not really possible today for any test that fails on multiple ports with different results, as it's practically impossible to run more than a couple of ports by hand, and we don't have a bot infrastructure to help you here. That would be the best outcome indeed, however that would more or less require all the EWS to be able to run the tests (including pixel tests for the platforms that enable them). Yes. Probably even harder, we'd have to build tooling to extract the results off of this bot and merge them into your patch. 3) the current tool-of-the-day for managing rebaselining, garden-o-matic, is much better suited to handling the unexpected failures on the bots rather than the expected failures you've introduced. You could see it the other way. How could we make garden-o-matic handle newly added suppression better? Maybe some new sub-tool listing the newly added suppressions would help? Ignore the suppressions added XX hours ago? Yes, that is similar. Your issue seems to be suppressions sticking into the tree not strictly suppressing in itself. Our current tool (garden-o-matic) handles failures a lot better than it handles suppressions. Yes. If there's consensus in the mean time that it is better on balance to check in suppressions, perhaps we can figure out a better way to do that. Maybe (shudder) a second test_expectations file? Or maybe it would be better to actually check in suppressions marked as REBASELINE (or something like that)? That sounds quirky as it involves maintaining 2 sets of files. From my perspective, saying that we should discard the EWS result and allow changes to get in WebKit trunk, knowing they will turn the bots red, is a bad proposal regardless of how you justify it. In the small delta where the bots are red, you can bet people will miss something else that breaks. As Ryosuke points out, practically we're already in that situation - from what I can tell, the tree is red virtually all of the time, at least during US/Pacific working hours. It's not clear to me if the EWS has made this better or worse, but perhaps others have noticed a difference. That said, I doubt I like red trees any more than you do :) I had not considered the EWS implications when I wrote the initial note, so I haven't decided if (or how) to revise my opinions yet. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] handling failing tests (test_expectations, Skipped files, etc.)
If there's consensus in the mean time that it is better on balance to check in suppressions, perhaps we can figure out a better way to do that. Maybe (shudder) a second test_expectations file? Or maybe it would be better to actually check in suppressions marked as REBASELINE (or something like that)? That sounds quirky as it involves maintaining 2 sets of files. From my perspective, saying that we should discard the EWS result and allow changes to get in WebKit trunk, knowing they will turn the bots red, is a bad proposal regardless of how you justify it. In the small delta where the bots are red, you can bet people will miss something else that breaks. As Ryosuke points out, practically we're already in that situation - from what I can tell, the tree is red virtually all of the time, at least during US/Pacific working hours. It's not clear to me if the EWS has made this better or worse, but perhaps others have noticed a difference. That said, I doubt I like red trees any more than you do :) I wasn't talking about the tree's status quo here as it shouldn't impact the discussion. Just because the tree is red, doesn't mean it's the right thing (tm) to just drop the towel and make it more red (even if we seem to agree on the badness of that :)). To add some thoughts here, saying that the tree is red covers several states (failing tests, not building...) and IMHO the EWS has at least helped on the building side. As far as the tests goes, a lot of platform differences are unfortunately uncovered on the bots. Thanks, Julien ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
