On Mon, Nov 16, 2009 at 5:56 AM, Xan Lopez <x...@gnome.org> wrote: > On Mon, Nov 16, 2009 at 3:33 PM, Gustavo Noronha Silva <g...@gnome.org> wrote: >> On Mon, 2009-11-16 at 05:24 -0800, Adam Barth wrote: >>> Eric and I are working on a bot that might help this situation. >>> Essentially, the bot will try out patches on Qt and GTK and add a >>> comment to the bug if the patch regresses the build. Our plan is to >>> start with compiling, but we'd eventually like to run the tests as >>> well. >> >> That sounds like an awesome idea. Thanks for working on it. Build >> breakages themselves have become less of an issue for us in recent times >> - people are definitely more aware of our bots, and are acting on fixing >> them when they break. >> >> I think such an automated approach to running the build, and tests for >> upcoming patches will surely help with giving this a second step >> forward. > > This is nice to see, but as Gustavo says build breakage is not too > much an issue at the moment for us: the build does not break very > often, and when it does people usually take the time to figure out > what happened and either do fix it themselves or poke us about it. > That being said, this could be improved in any number of ways and I'm > happy to see it getting ever better. > > What is effectively a black hole with respect to our time is the tests > breakage, though. We get new tests failing very regularly (either > through new tests or through new code making old tests fail), and once > the bots are red people tend to pay even less attention to them, so > they spiral out of control in a positive feedback loop until we have > tests failing in the double digits in a matter of days (or hours!). Of > course in an ideal world we'd have a team big enough to always have at > least one person looking at this and fixing the problems the moment > they arise, but unfortunately this is not the case.
This is a huge issue with the Chromium port as well. We spend quite a bit of effort tracking down failing tests, only to discover that the failure is due to one-port baselines or new functionality added to DRT. I wonder if the approach we have today in regards to tests is not sustainable with multiple vibrant ports, each spending way to much time catching up. :DG< _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev