Re: [webkit-dev] A bot-filled future?
On Mon, 2009-11-16 at 18:48 -0800, Geoffrey Garen wrote: I understand why you want to start with a Mac bot, and I applaud starting small. However, bear in mind that lots of WebKit contributors work primarily on the Mac, so they probably won't get any use out of a Mac try bot, and you probably won't get any feedback from them until the try bot expands to cover other platforms. I believe the idea is in fact to start with a GTK+/Qt on Linux bot (since it should be simpler to setup). See you, -- Gustavo Noronha Silva g...@gnome.org GNOME Project ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
I believe the idea is in fact to start with a GTK+/Qt on Linux bot (since it should be simpler to setup). Neat. Geoff ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
A try bot would help me a lot. As others have mentioned, being able to see output from build+test is a big deal. I understand why you want to start with a Mac bot, and I applaud starting small. However, bear in mind that lots of WebKit contributors work primarily on the Mac, so they probably won't get any use out of a Mac try bot, and you probably won't get any feedback from them until the try bot expands to cover other platforms. Geoff On Nov 12, 2009, at 11:37 AM, Adam Barth wrote: As the project grows, we need to scale our processes to match. In large part, that means automating as much work as possible. Commit-queue has done a good job of solving the land patches from non-committers efficiently problem, effectively removing that as a pain point. I'd like to ask you to open your hearts and your minds to the idea of automating more of our processes. Currently, I see the biggest pain-point in our process as the always-burgeoning pending-review list. It's difficult to automate the process of accepting good patches because that requires attention from experts. Instead, I think we should make it easier to reject bad patches. As a first step, I've started extending bugzilla-tool to be a try server in https://bugs.webkit.org/show_bug.cgi?id=31422. Here's how this might work: 1) Contributor posts patch for review. 2) Committer marks patch with the try? flag. 3) The try-queue downloads, applies, builds, and tests the patch. 4) If all systems are go, the try-queue marks the patch as try+. Otherwise, it marks the patch as try- with an explanation of what went wrong. The try-queue will be purely optional and advisory. Hopefully a try- notation will encourage the contributor to post a new version of the patch that passes the try-queue. Further down the road, one can also imagine another bot that automates step (2) by scanning the pending-review list for untried patches and marking them as try? when the try-queue has unused bandwidth. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
On Nov 12, 2009, at 8:20 PM, Adam Roben wrote: On Nov 12, 2009, at 6:59 PM, Ojan Vafai wrote: It's also easy to spot tests with FAIL or PASS in the output. :) I'd hope that those tests would have cross-platform results, since clearly the condition is testable without having access to a full render tree dump. Creating and maintaing cross-platform results is a real challenge in some cases. With HTMLMediaElement tests, for example, we have to deal with the fact that different ports uses different backend media engines and not all ports support support the same file formats and codecs. This means it can be impossible to predict results without actually running a tests in each port. Having a try server would be quite helpful for these types of tests. eric ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
I think that sounds like a really good idea, and I can see my self using that when touching cross platform code. Kenneth On Thu, Nov 12, 2009 at 4:37 PM, Adam Barth aba...@webkit.org wrote: As the project grows, we need to scale our processes to match. In large part, that means automating as much work as possible. Commit-queue has done a good job of solving the land patches from non-committers efficiently problem, effectively removing that as a pain point. I'd like to ask you to open your hearts and your minds to the idea of automating more of our processes. Currently, I see the biggest pain-point in our process as the always-burgeoning pending-review list. It's difficult to automate the process of accepting good patches because that requires attention from experts. Instead, I think we should make it easier to reject bad patches. As a first step, I've started extending bugzilla-tool to be a try server in https://bugs.webkit.org/show_bug.cgi?id=31422. Here's how this might work: 1) Contributor posts patch for review. 2) Committer marks patch with the try? flag. 3) The try-queue downloads, applies, builds, and tests the patch. 4) If all systems are go, the try-queue marks the patch as try+. Otherwise, it marks the patch as try- with an explanation of what went wrong. The try-queue will be purely optional and advisory. Hopefully a try- notation will encourage the contributor to post a new version of the patch that passes the try-queue. Further down the road, one can also imagine another bot that automates step (2) by scanning the pending-review list for untried patches and marking them as try? when the try-queue has unused bandwidth. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev -- Kenneth Rohde Christiansen Technical Lead / Software Engineer Qt Labs Americas, Nokia Technology Institute, INdT Phone +55 81 8895 6002 / E-mail kenneth.christiansen at openbossa.org ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
It's so easy to have code that builds on one platform but not another. Even if the try servers were only builders to begin with, I think they'd provide a lot of value to the project. On Thu, Nov 12, 2009 at 11:43 AM, Kenneth Christiansen kenneth.christian...@openbossa.org wrote: I think that sounds like a really good idea, and I can see my self using that when touching cross platform code. Kenneth On Thu, Nov 12, 2009 at 4:37 PM, Adam Barth aba...@webkit.org wrote: As the project grows, we need to scale our processes to match. In large part, that means automating as much work as possible. Commit-queue has done a good job of solving the land patches from non-committers efficiently problem, effectively removing that as a pain point. I'd like to ask you to open your hearts and your minds to the idea of automating more of our processes. Currently, I see the biggest pain-point in our process as the always-burgeoning pending-review list. It's difficult to automate the process of accepting good patches because that requires attention from experts. Instead, I think we should make it easier to reject bad patches. As a first step, I've started extending bugzilla-tool to be a try server in https://bugs.webkit.org/show_bug.cgi?id=31422. Here's how this might work: 1) Contributor posts patch for review. 2) Committer marks patch with the try? flag. 3) The try-queue downloads, applies, builds, and tests the patch. 4) If all systems are go, the try-queue marks the patch as try+. Otherwise, it marks the patch as try- with an explanation of what went wrong. The try-queue will be purely optional and advisory. Hopefully a try- notation will encourage the contributor to post a new version of the patch that passes the try-queue. Further down the road, one can also imagine another bot that automates step (2) by scanning the pending-review list for untried patches and marking them as try? when the try-queue has unused bandwidth. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev -- Kenneth Rohde Christiansen Technical Lead / Software Engineer Qt Labs Americas, Nokia Technology Institute, INdT Phone +55 81 8895 6002 / E-mail kenneth.christiansen at openbossa.org ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
Seconded (or Thirded). I'd been working on a try-server using Chromium's try-change.py, but this seems like a much cleaner way to handle it, and ties into the Bugzilla workflow much better than my solution, and would be much easier to limit who can set the try bit, based on what we decide the policy to be. On Nov 12, 2009, at 12:41 PM, Jeremy Orlow wrote: It's so easy to have code that builds on one platform but not another. Even if the try servers were only builders to begin with, I think they'd provide a lot of value to the project. On Thu, Nov 12, 2009 at 11:43 AM, Kenneth Christiansen kenneth.christian...@openbossa.org wrote: I think that sounds like a really good idea, and I can see my self using that when touching cross platform code. Kenneth On Thu, Nov 12, 2009 at 4:37 PM, Adam Barth aba...@webkit.org wrote: As the project grows, we need to scale our processes to match. In large part, that means automating as much work as possible. Commit-queue has done a good job of solving the land patches from non-committers efficiently problem, effectively removing that as a pain point. I'd like to ask you to open your hearts and your minds to the idea of automating more of our processes. Currently, I see the biggest pain-point in our process as the always-burgeoning pending-review list. It's difficult to automate the process of accepting good patches because that requires attention from experts. Instead, I think we should make it easier to reject bad patches. As a first step, I've started extending bugzilla-tool to be a try server in https://bugs.webkit.org/show_bug.cgi?id=31422. Here's how this might work: 1) Contributor posts patch for review. 2) Committer marks patch with the try? flag. 3) The try-queue downloads, applies, builds, and tests the patch. 4) If all systems are go, the try-queue marks the patch as try+. Otherwise, it marks the patch as try- with an explanation of what went wrong. The try-queue will be purely optional and advisory. Hopefully a try- notation will encourage the contributor to post a new version of the patch that passes the try-queue. Further down the road, one can also imagine another bot that automates step (2) by scanning the pending-review list for untried patches and marking them as try? when the try-queue has unused bandwidth. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev -- Kenneth Rohde Christiansen Technical Lead / Software Engineer Qt Labs Americas, Nokia Technology Institute, INdT Phone +55 81 8895 6002 / E-mail kenneth.christiansen at openbossa.org ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
On 2009-11-12, at 11:37, Adam Barth wrote: 1) Contributor posts patch for review. 2) Committer marks patch with the try? flag. 3) The try-queue downloads, applies, builds, and tests the patch. 4) If all systems are go, the try-queue marks the patch as try+. Otherwise, it marks the patch as try- with an explanation of what went wrong. I have a few comments / questions about this: 1) People are already confused about how to handle the recently-added commit-queue flag. Adding an extra flag is going to increase the confusion. 2) What machines are going to be doing these tests, and on which platforms? 3) Which patches would this test? Running tests on an arbitrary patch uploaded in Bugzilla opens up the testing machine to executing arbitrary code unless there are limitations in place. - Mark smime.p7s Description: S/MIME cryptographic signature ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
On Thu, Nov 12, 2009 at 1:04 PM, Mark Rowe mr...@apple.com wrote: On 2009-11-12, at 11:37, Adam Barth wrote: 1) Contributor posts patch for review. 2) Committer marks patch with the try? flag. 3) The try-queue downloads, applies, builds, and tests the patch. 4) If all systems are go, the try-queue marks the patch as try+. Otherwise, it marks the patch as try- with an explanation of what went wrong. I have a few comments / questions about this: I suspected you might, hence the post to this list. :) 1) People are already confused about how to handle the recently-added commit-queue flag. Adding an extra flag is going to increase the confusion. Do you have other ideas about how to present the information? We want to make the information easily available to reviewers when they're reviewing patches. 2) What machines are going to be doing these tests, and on which platforms? I was going to start by running the try-queue on a Mac laptop I don't use very often. If/when we want to expand coverage to other platforms, we can worry about where to get more machines. Ultimately, machines are much cheaper than people. I'm not worried about finding hardware. 3) Which patches would this test? Running tests on an arbitrary patch uploaded in Bugzilla opens up the testing machine to executing arbitrary code unless there are limitations in place. My plan what to require a committer to sign-off on the patch (e.g., by setting the flag), the same way we require a committer to sign off for running the commit-queue or the build-bot. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
On Thu, Nov 12, 2009 at 12:41 PM, Jeremy Orlow jor...@chromium.org wrote: It's so easy to have code that builds on one platform but not another. Even if the try servers were only builders to begin with, I think they'd provide a lot of value to the project. That's a good idea, especially for ports that have perennially red tests. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
On Thu, Nov 12, 2009 at 1:04 PM, Mark Rowe mr...@apple.com wrote: 1) People are already confused about how to handle the recently-added commit-queue flag. Adding an extra flag is going to increase the confusion. I chatted with Eric about how to solve this problem. One option is to just try every change that has review? and add a comment to the bug about success / failure. That minimizes the UI surface and avoids adding yet-another-flag. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
On 2009-11-12, at 14:43, Adam Barth wrote: On Thu, Nov 12, 2009 at 1:04 PM, Mark Rowe mr...@apple.com wrote: 1) People are already confused about how to handle the recently-added commit-queue flag. Adding an extra flag is going to increase the confusion. I chatted with Eric about how to solve this problem. One option is to just try every change that has review? and add a comment to the bug about success / failure. That minimizes the UI surface and avoids adding yet-another-flag. That doesn't seem to fit with your suggestion for how we deal with the risk of running arbitrary code on the test machines. - Mark smime.p7s Description: S/MIME cryptographic signature ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
That sounds good to me. As for the security issues: It seems like we could build code from anyone but only run the tests from committers. On Thu, Nov 12, 2009 at 2:43 PM, Adam Barth aba...@webkit.org wrote: On Thu, Nov 12, 2009 at 1:04 PM, Mark Rowe mr...@apple.com wrote: 1) People are already confused about how to handle the recently-added commit-queue flag. Adding an extra flag is going to increase the confusion. I chatted with Eric about how to solve this problem. One option is to just try every change that has review? and add a comment to the bug about success / failure. That minimizes the UI surface and avoids adding yet-another-flag. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
What if someone changed build-webkit or the build procedure in one of the vcproj's? On Nov 12, 2009, at 2:50 PM, Jeremy Orlow wrote: That sounds good to me. As for the security issues: It seems like we could build code from anyone but only run the tests from committers. On Thu, Nov 12, 2009 at 2:43 PM, Adam Barth aba...@webkit.org wrote: On Thu, Nov 12, 2009 at 1:04 PM, Mark Rowe mr...@apple.com wrote: 1) People are already confused about how to handle the recently-added commit-queue flag. Adding an extra flag is going to increase the confusion. I chatted with Eric about how to solve this problem. One option is to just try every change that has review? and add a comment to the bug about success / failure. That minimizes the UI surface and avoids adding yet-another-flag. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
On 2009-11-12, at 14:50, Jeremy Orlow wrote: That sounds good to me. As for the security issues: It seems like we could build code from anyone but only run the tests from committers. Building involves running code too. build-webkit, makefiles for dependencies, scripts in Xcode projects, etc. - Mark On Thu, Nov 12, 2009 at 2:43 PM, Adam Barth aba...@webkit.org wrote: On Thu, Nov 12, 2009 at 1:04 PM, Mark Rowe mr...@apple.com wrote: 1) People are already confused about how to handle the recently-added commit-queue flag. Adding an extra flag is going to increase the confusion. I chatted with Eric about how to solve this problem. One option is to just try every change that has review? and add a comment to the bug about success / failure. That minimizes the UI surface and avoids adding yet-another-flag. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev smime.p7s Description: S/MIME cryptographic signature ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
Ok. The only run stuff uploaded by committers automatically? Maybe have a web page committers can visit to submit (i.e. vouch for) patches to be run through the bots? J On Thu, Nov 12, 2009 at 2:52 PM, Brian Weinstein bweinst...@apple.comwrote: What if someone changed build-webkit or the build procedure in one of the vcproj's? On Nov 12, 2009, at 2:50 PM, Jeremy Orlow wrote: That sounds good to me. As for the security issues: It seems like we could build code from anyone but only run the tests from committers. On Thu, Nov 12, 2009 at 2:43 PM, Adam Barth aba...@webkit.org wrote: On Thu, Nov 12, 2009 at 1:04 PM, Mark Rowe mr...@apple.com wrote: 1) People are already confused about how to handle the recently-added commit-queue flag. Adding an extra flag is going to increase the confusion. I chatted with Eric about how to solve this problem. One option is to just try every change that has review? and add a comment to the bug about success / failure. That minimizes the UI surface and avoids adding yet-another-flag. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
Agreed. Running every r=? patch through such a build-service is insecure. However if Adam wishes to run it on his own hardware, I certainly have no objections to such. :) -eric On Thu, Nov 12, 2009 at 2:49 PM, Mark Rowe mr...@apple.com wrote: On 2009-11-12, at 14:43, Adam Barth wrote: On Thu, Nov 12, 2009 at 1:04 PM, Mark Rowe mr...@apple.com wrote: 1) People are already confused about how to handle the recently-added commit-queue flag. Adding an extra flag is going to increase the confusion. I chatted with Eric about how to solve this problem. One option is to just try every change that has review? and add a comment to the bug about success / failure. That minimizes the UI surface and avoids adding yet-another-flag. That doesn't seem to fit with your suggestion for how we deal with the risk of running arbitrary code on the test machines. - Mark ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
This approach doesn't lend itself as well to trying patches before putting them up for review. Specifically, I want to be able to try patches without spamming everyone with bugzilla mail. This is solvable in this bugzilla-based approach, but it doesn't lend itself to this as naturally, e.g. presumably there's a way to tell bugzilla not to send mail for a given comment. Also, it would be great if the commit-queue, try-server, whatever, had a UI like the buildbot waterfall. There's a couple advantages: 1. Can see the stdio as the tests run and get better information about why it failed. 2. Can grab layout test results from the try servers. This would reduce the need/occurence of committing Mac expectations and then cleaning up other platforms post commit. Ojan On Thu, Nov 12, 2009 at 12:58 PM, Brian Weinstein bweinst...@apple.comwrote: Seconded (or Thirded). I'd been working on a try-server using Chromium's try-change.py, but this seems like a much cleaner way to handle it, and ties into the Bugzilla workflow much better than my solution, and would be much easier to limit who can set the try bit, based on what we decide the policy to be. On Nov 12, 2009, at 12:41 PM, Jeremy Orlow wrote: It's so easy to have code that builds on one platform but not another. Even if the try servers were only builders to begin with, I think they'd provide a lot of value to the project. On Thu, Nov 12, 2009 at 11:43 AM, Kenneth Christiansen kenneth.christian...@openbossa.org wrote: I think that sounds like a really good idea, and I can see my self using that when touching cross platform code. Kenneth On Thu, Nov 12, 2009 at 4:37 PM, Adam Barth aba...@webkit.org wrote: As the project grows, we need to scale our processes to match. In large part, that means automating as much work as possible. Commit-queue has done a good job of solving the land patches from non-committers efficiently problem, effectively removing that as a pain point. I'd like to ask you to open your hearts and your minds to the idea of automating more of our processes. Currently, I see the biggest pain-point in our process as the always-burgeoning pending-review list. It's difficult to automate the process of accepting good patches because that requires attention from experts. Instead, I think we should make it easier to reject bad patches. As a first step, I've started extending bugzilla-tool to be a try server in https://bugs.webkit.org/show_bug.cgi?id=31422. Here's how this might work: 1) Contributor posts patch for review. 2) Committer marks patch with the try? flag. 3) The try-queue downloads, applies, builds, and tests the patch. 4) If all systems are go, the try-queue marks the patch as try+. Otherwise, it marks the patch as try- with an explanation of what went wrong. The try-queue will be purely optional and advisory. Hopefully a try- notation will encourage the contributor to post a new version of the patch that passes the try-queue. Further down the road, one can also imagine another bot that automates step (2) by scanning the pending-review list for untried patches and marking them as try? when the try-queue has unused bandwidth. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev -- Kenneth Rohde Christiansen Technical Lead / Software Engineer Qt Labs Americas, Nokia Technology Institute, INdT Phone +55 81 8895 6002 / E-mail kenneth.christiansen at openbossa.org ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
To clarify, I think abarth's proposal is great and we shouldn't block it on other things a try-server could provide. Would just be nice to keep the make writing patches more efficient use-case in mind as we add other infrastructure to avoid needing, for example, a whole different set of servers for that. On Thu, Nov 12, 2009 at 3:48 PM, Adam Roben aro...@apple.com wrote: On Nov 12, 2009, at 6:38 PM, Ojan Vafai wrote: 2. Can grab layout test results from the try servers. This would reduce the need/occurence of committing Mac expectations and then cleaning up other platforms post commit. Without being able to see the loaded page on that platform, how do you know if the results are correct? (If you're able to verify the results just based on the test output, maybe the test should be cross-platform!) This entirely depends on the change you're making. For example, with a change that makes buttons 1px wider it's easy to verify from the layout test diff that the new results are correct. It's also easy to spot tests with FAIL or PASS in the output. :) Ojan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
On Nov 12, 2009, at 6:38 PM, Ojan Vafai wrote: 2. Can grab layout test results from the try servers. This would reduce the need/occurence of committing Mac expectations and then cleaning up other platforms post commit. Without being able to see the loaded page on that platform, how do you know if the results are correct? (If you're able to verify the results just based on the test output, maybe the test should be cross-platform!) -Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
I agree that this design doesn't solve the problem of writing patches more efficiently. That's not the goal. The goal is to reduce review latency by automating the mechanical parts of the review process. Adam On Thu, Nov 12, 2009 at 3:38 PM, Ojan Vafai o...@chromium.org wrote: This approach doesn't lend itself as well to trying patches before putting them up for review. Specifically, I want to be able to try patches without spamming everyone with bugzilla mail. This is solvable in this bugzilla-based approach, but it doesn't lend itself to this as naturally, e.g. presumably there's a way to tell bugzilla not to send mail for a given comment. Also, it would be great if the commit-queue, try-server, whatever, had a UI like the buildbot waterfall. There's a couple advantages: 1. Can see the stdio as the tests run and get better information about why it failed. 2. Can grab layout test results from the try servers. This would reduce the need/occurence of committing Mac expectations and then cleaning up other platforms post commit. Ojan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
On Thu, Nov 12, 2009 at 3:51 PM, Eric Seidel esei...@google.com wrote: I think Ojan is used to Chromium's world where there is a layout-test rebaseling tool which knows how to suck expected results off of the chromium try-bots, including new pixel test results. So he (and others) are used to posting their patch for try and then getting back the new results for all the platforms. You can see how a page looks on that platform by looking at the pixel tests for that platform which come back to you from the rebaselining tool/try-bots. As abarth mentioned, a very nice-to-have, but not part of his initial stated goal. -eric On Thu, Nov 12, 2009 at 3:48 PM, Adam Roben aro...@apple.com wrote: On Nov 12, 2009, at 6:38 PM, Ojan Vafai wrote: 2. Can grab layout test results from the try servers. This would reduce the need/occurence of committing Mac expectations and then cleaning up other platforms post commit. Without being able to see the loaded page on that platform, how do you know if the results are correct? (If you're able to verify the results just based on the test output, maybe the test should be cross-platform!) -Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
I think the main reason why we don't yet have a try server is that we block it on stuff like this...which is nice to have. It seems like we could get something basic up that worked for 90% of cases and then iterate on something more featureful. I think Adam has the right idea here. J On Thu, Nov 12, 2009 at 3:38 PM, Ojan Vafai o...@chromium.org wrote: This approach doesn't lend itself as well to trying patches before putting them up for review. Specifically, I want to be able to try patches without spamming everyone with bugzilla mail. This is solvable in this bugzilla-based approach, but it doesn't lend itself to this as naturally, e.g. presumably there's a way to tell bugzilla not to send mail for a given comment. Also, it would be great if the commit-queue, try-server, whatever, had a UI like the buildbot waterfall. There's a couple advantages: 1. Can see the stdio as the tests run and get better information about why it failed. 2. Can grab layout test results from the try servers. This would reduce the need/occurence of committing Mac expectations and then cleaning up other platforms post commit. Ojan On Thu, Nov 12, 2009 at 12:58 PM, Brian Weinstein bweinst...@apple.comwrote: Seconded (or Thirded). I'd been working on a try-server using Chromium's try-change.py, but this seems like a much cleaner way to handle it, and ties into the Bugzilla workflow much better than my solution, and would be much easier to limit who can set the try bit, based on what we decide the policy to be. On Nov 12, 2009, at 12:41 PM, Jeremy Orlow wrote: It's so easy to have code that builds on one platform but not another. Even if the try servers were only builders to begin with, I think they'd provide a lot of value to the project. On Thu, Nov 12, 2009 at 11:43 AM, Kenneth Christiansen kenneth.christian...@openbossa.org wrote: I think that sounds like a really good idea, and I can see my self using that when touching cross platform code. Kenneth On Thu, Nov 12, 2009 at 4:37 PM, Adam Barth aba...@webkit.org wrote: As the project grows, we need to scale our processes to match. In large part, that means automating as much work as possible. Commit-queue has done a good job of solving the land patches from non-committers efficiently problem, effectively removing that as a pain point. I'd like to ask you to open your hearts and your minds to the idea of automating more of our processes. Currently, I see the biggest pain-point in our process as the always-burgeoning pending-review list. It's difficult to automate the process of accepting good patches because that requires attention from experts. Instead, I think we should make it easier to reject bad patches. As a first step, I've started extending bugzilla-tool to be a try server in https://bugs.webkit.org/show_bug.cgi?id=31422. Here's how this might work: 1) Contributor posts patch for review. 2) Committer marks patch with the try? flag. 3) The try-queue downloads, applies, builds, and tests the patch. 4) If all systems are go, the try-queue marks the patch as try+. Otherwise, it marks the patch as try- with an explanation of what went wrong. The try-queue will be purely optional and advisory. Hopefully a try- notation will encourage the contributor to post a new version of the patch that passes the try-queue. Further down the road, one can also imagine another bot that automates step (2) by scanning the pending-review list for untried patches and marking them as try? when the try-queue has unused bandwidth. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev -- Kenneth Rohde Christiansen Technical Lead / Software Engineer Qt Labs Americas, Nokia Technology Institute, INdT Phone +55 81 8895 6002 / E-mail kenneth.christiansen at openbossa.org ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] A bot-filled future?
On Nov 12, 2009, at 6:59 PM, Ojan Vafai wrote: On Thu, Nov 12, 2009 at 3:48 PM, Adam Roben aro...@apple.com wrote: On Nov 12, 2009, at 6:38 PM, Ojan Vafai wrote: 2. Can grab layout test results from the try servers. This would reduce the need/occurence of committing Mac expectations and then cleaning up other platforms post commit. Without being able to see the loaded page on that platform, how do you know if the results are correct? (If you're able to verify the results just based on the test output, maybe the test should be cross-platform!) This entirely depends on the change you're making. For example, with a change that makes buttons 1px wider it's easy to verify from the layout test diff that the new results are correct. That's true. It's also easy to spot tests with FAIL or PASS in the output. :) I'd hope that those tests would have cross-platform results, since clearly the condition is testable without having access to a full render tree dump. -Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
