On Mon, Nov 16, 2009 at 3:33 PM, Gustavo Noronha Silva <g...@gnome.org> wrote: > On Mon, 2009-11-16 at 05:24 -0800, Adam Barth wrote: >> Eric and I are working on a bot that might help this situation. >> Essentially, the bot will try out patches on Qt and GTK and add a >> comment to the bug if the patch regresses the build. Our plan is to >> start with compiling, but we'd eventually like to run the tests as >> well. > > That sounds like an awesome idea. Thanks for working on it. Build > breakages themselves have become less of an issue for us in recent times > - people are definitely more aware of our bots, and are acting on fixing > them when they break. > > I think such an automated approach to running the build, and tests for > upcoming patches will surely help with giving this a second step > forward.
This is nice to see, but as Gustavo says build breakage is not too much an issue at the moment for us: the build does not break very often, and when it does people usually take the time to figure out what happened and either do fix it themselves or poke us about it. That being said, this could be improved in any number of ways and I'm happy to see it getting ever better. What is effectively a black hole with respect to our time is the tests breakage, though. We get new tests failing very regularly (either through new tests or through new code making old tests fail), and once the bots are red people tend to pay even less attention to them, so they spiral out of control in a positive feedback loop until we have tests failing in the double digits in a matter of days (or hours!). Of course in an ideal world we'd have a team big enough to always have at least one person looking at this and fixing the problems the moment they arise, but unfortunately this is not the case. I realize this is a very complicated problem to fix unless we throw more people at it from our side, but maybe we could all agree in some basic guidelines to make things a bit better: - If you are adding new functionality + tests that you *know* will fail in some bots, open a bug describing what we should implement to get it working, and skip all new tests with a reference to the bug number. This should be relatively easy to do for the person who implemented the functionality in the first place, and would save everyone lots of time in the long run. - If your new code makes a previously passing test(s) fail in a port other than your own, you have two choices: * If you know what's going on and think this is a bug in the port, you can propose a patch yourself, or open a bug with your ideas and skip the tests with a reference to the bug number. Get the port maintainers on the loop ASAP. * If you have no idea of what's going on, maybe Eric's proposed hard-line policy kicks in: you either figure it out quick and propose something, or the patch is rolled out until things are clear; regressions in tests in any port are as bad as build breakages, so early roll out is preferred to having the bots red for a long time. As I said, this is a complex problem, but I think at the very least having a clear picture of what is expected of everyone with respect to failing tests in the bots would be a good start to make things easier. Xan > > See you, > > -- > Gustavo Noronha Silva <g...@gnome.org> > GNOME Project > > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev