On Wed, Mar 17, 2010 at 2:16 AM, Maciej Stachowiak <[email protected]> wrote: > > On Mar 17, 2010, at 1:49 AM, Kenneth Russell wrote: > >> On Wed, Mar 17, 2010 at 12:44 AM, Maciej Stachowiak <[email protected]> wrote: >>> >>> On Mar 17, 2010, at 12:02 AM, Adam Barth wrote: >>> >>>> >>>> Manual investigation seems to implicate >>>> <http://trac.webkit.org/changeset/56074> for at least some of the >>>> brokenness in SnowLeopard Intel Release (Tests). There's a patch in >>>> <https://bugs.webkit.org/show_bug.cgi?id=36194> that claims to fix >>>> things, but the brokenness has been polluting the tree for 12 hours. >>>> Is there some reason we didn't roll out the offending patch? >>> >>> Kenneth intended to temporarily disable the failing tests: >>> <http://trac.webkit.org/changeset/56093>. That seems to have cleaned up >>> the >>> failures on other bots, but there is still a great number of failures on >>> SnowLeopard release. I'm not sure what we can do other than revert the >>> patch until the correct fix is ready. >> >> I apologize for the widespread WebGL test breakage from this patch. I >> sidestepped the commit queue because other broken tests had blocked >> it, and this patch was blocking other work. While the patch worked >> fine on our local machines (both Leopard and Snow Leopard), it seems >> that OpenGL driver bugs on the build bots caused many tests to fail, >> in particular on the Snow Leopard bot. This afternoon we skipped the >> tests that were blocking the bots the cq cared about. > > I don't think it's that great to leave breakage on some of the bots solely > because the commit queue doesn't pay attention to them. > > (BTW do we know if the change might tickle similar driver bugs on any other > hardware/software combos that we haven't tested yet? Do we have a plan for > finding out? Has anyone reported the relevant driver bugs to the appropriate > vendor?)
Our best current plan is more widespread testing. We will file a Radar bug as soon as we have more information about the nature of the failure -- by virtue of working around the bugs. If we knew the precise hardware configuration of the bots, including graphics cards, that would help. >> Mo worked as quickly as possible today to come up with a workaround >> for the broken drivers, and a patch is out for review, but perhaps the >> original patch should be rolled out after all, and a revised version >> re-committed through the queue. > > If it's going to be more than a couple of hours before the proper fix can > land, then I would recommend that. I'm afraid I personally can't do anything at this point until probably 9 or 10 AM Pacific. If someone wants to revert the patch in the interim, that's fine. I hope that we will have the proper workaround for the driver bugs by early afternoon Pacific. >> Again, I apologize for the breakage. It would be best for everyone, I >> think, if we got the tree to a green state and all committed through >> the queue, thereby having a line of defense against unexpected test >> failures on the bots. > > If we really want everyone to use the commit queue for most normal work, we > really have to fix it so that it puts a meaningful value in SVN's committer > field. > > That being said, the mechanism I'd really like to see first is better > notification of when the bot goes red (I suspect a number of people involved > in today's redness didn't notice right away because there is no active > notification system). For what it's worth, the tree was red before I committed due to at least two or three flaky tests which have been showing up for days. An active notification system would not have helped here. I think what should be done is to get the tree green by skipping these flaky tests; file high-priority bugs against the test authors to fix the flakiness; and then figure out a way the commit queue can be used for the vast majority of patches. (A secure repository of committer username/passwords on the machine actually executing the commit?) -Ken > Regards, > Maciej > > _______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
