[webkit-dev] Criteria for becoming a core builder
Hi all- I'd like to understand better what the criteria are for a builder to become a core builder. We've been putting lot of effort into keeping the Windows testers green lately (especially the Windows 7 Release (Tests) builder), and it seems like it would be good for them to eventually join the core group so they can reap the benefits of sheriffbot, an increased number of people looking at them, etc. Most of the failures that happen on the Windows bots these days are either due to: 1) a new test being added which needs results on Windows 2) an existing test with Windows-specific results being modified without the Windows results being updated 3) test flakiness Being in the core group seems like it would help with (1) and (2) because the people who are making these changes would notice that they forgot Windows. It wouldn't help with (3), of course. (We're obviously trying to track down all causes of flakiness we can.) So what does it take for a builder to be a core builder? Once we know this, we can hopefully focus our Windows efforts toward satisfying the criteria. Thanks! -Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Criteria for becoming a core builder
I'm somewhat sad at having helped create this concept of Core, since it's generated much confusion. It was/is intended more as a way to get non-stable buildbots off the main page, and away from distracting developers and bots. IMO the core list should be *descriptive* rather than *prescriptive*. If a bot is regularly green, it should be core. If its not, it should be non-core. We should not remove bots we don't like, or add bots we hope to stay green. So if you believe the windows bots are regularly green, you should make them Core. We can always remove them. Ideally all bots would be core! Along the same logic, we should move bots like Leopard to non-core, since they haven't been green in weeks (even though I'm sure we wish it would be green). Hopefully that helps. -eric On Wed, Mar 2, 2011 at 9:51 AM, Adam Roben aro...@apple.com wrote: Hi all- I'd like to understand better what the criteria are for a builder to become a core builder. We've been putting lot of effort into keeping the Windows testers green lately (especially the Windows 7 Release (Tests) builder), and it seems like it would be good for them to eventually join the core group so they can reap the benefits of sheriffbot, an increased number of people looking at them, etc. Most of the failures that happen on the Windows bots these days are either due to: 1) a new test being added which needs results on Windows 2) an existing test with Windows-specific results being modified without the Windows results being updated 3) test flakiness Being in the core group seems like it would help with (1) and (2) because the people who are making these changes would notice that they forgot Windows. It wouldn't help with (3), of course. (We're obviously trying to track down all causes of flakiness we can.) So what does it take for a builder to be a core builder? Once we know this, we can hopefully focus our Windows efforts toward satisfying the criteria. Thanks! -Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Criteria for becoming a core builder
On Mar 2, 2011, at 1:38 PM, Eric Seidel wrote: So if you believe the windows bots are regularly green, you should make them Core. Well, they are regularly green except for the cases I mentioned. The biggest problem is people making changes that cause tests to fail on Windows and not noticing that this has happened, so I think making them core will help keep them quite a bit greener. So it sounds to me like we should go ahead and do this. I'll wait a day or two to allow other opinions to be heard. Along the same logic, we should move bots like Leopard to non-core, since they haven't been green in weeks (even though I'm sure we wish it would be green). The Leopard Debug bots are non-core now. The Release ones are still core for the moment. -Adam On Wed, Mar 2, 2011 at 9:51 AM, Adam Roben aro...@apple.com wrote: Hi all- I'd like to understand better what the criteria are for a builder to become a core builder. We've been putting lot of effort into keeping the Windows testers green lately (especially the Windows 7 Release (Tests) builder), and it seems like it would be good for them to eventually join the core group so they can reap the benefits of sheriffbot, an increased number of people looking at them, etc. Most of the failures that happen on the Windows bots these days are either due to: 1) a new test being added which needs results on Windows 2) an existing test with Windows-specific results being modified without the Windows results being updated 3) test flakiness Being in the core group seems like it would help with (1) and (2) because the people who are making these changes would notice that they forgot Windows. It wouldn't help with (3), of course. (We're obviously trying to track down all causes of flakiness we can.) So what does it take for a builder to be a core builder? Once we know this, we can hopefully focus our Windows efforts toward satisfying the criteria. Thanks! -Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Criteria for becoming a core builder
On Wed, Mar 2, 2011 at 7:38 PM, Eric Seidel e...@webkit.org wrote: IMO the core list should be *descriptive* rather than *prescriptive*. If a bot is regularly green, it should be core. If its not, it should be non-core. We should not remove bots we don't like, or add bots we hope to stay green. While this is pretty reasonably I must note that keeping bots green is substantially more difficult if they are not core, introducing a chicken-egg factor in your theory: green bots should (or are) core, but it's complex to be green most of the time without being core. That's why I think that adding bots to the core set because you hope they'll stay green is not as unreasonable as it might sound. Not that I have any perfect solution for this, but perhaps the only criteria for being in the list should be whether people are actively trying to keep a bot green. At least we could accept their intentions at face value at first, and only remove them from core if they fail to keep up (which we already do). Xan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
