I forgot to say that I did *not* defer tests that crash or hang. Those seem like they should be fixed. Security tests also seem like they should never be deferred. Pam, is there a reasonable way to tell if we've ever passed a test?
So, here's my proposal then. From here on, when doing webkit merges, we should defer any new tests that we add to the tests_fixable lists unless they crash, timeout or are security tests. Seem reasonable? Ojan On Thu, Feb 19, 2009 at 3:16 PM, Adam Barth <[email protected]> wrote: > Please also exclude security tests. > > Thanks, > Adam > > > On 2/19/09, Pam Greene <[email protected]> wrote: > > Perhaps this should exclude tests that we've ever passed, since those are > > also regressions, albeit more recent ones. > > - Pam > > > > On Thu, Feb 19, 2009 at 2:49 PM, Ojan Vafai <[email protected]> wrote: > > > >> > >> In the spirit of trying to fix all the layout tests that represent > >> real regressions since our initial launch, I've just committed a > >> changelist that defers the tests that are failing that are new to > >> webkit since the revision of our launch or whose expectations changed > >> upstream (the latter was only a couple tests, the former was almost 80 > >> tests). If people think this is an awful idea, I can rollback. > >> > >> I don't think it makes sense to block releases to the stable channel > >> on new tests that we've never passed. My biggest concern with each > >> release is breaking sites that previously used to work in Chrome. New > >> tests, for the most part, don't represent regressions like that. Does > >> it make sense to make a policy of deferring new, failing tests from a > >> webkit merge by default. The person doing the merge should put a good > >> faith effort into getting it fixed, but it shouldn't block cutting a > >> release. > >> > >> Eventually we'll get to a point where we only have deferred tests left > >> and we can start tackling that long list of tests without having it > >> hinder our ability to push releases to the stable channel. > >> > >> Ojan > >> > >> > > >> > > > > > > > > > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
