On Sep 28, 2009, at 4:47 PM, David Levin wrote:
I don't believe that the test was checked in a flaky state. It was
solid for a long time and then something happened...
What's "the test" in this context? The network / credentials one?
I'll try to add more logging to this test this evening (after my
turn at helping chromium stay up to date with WebKit is over).
I'll ping Drew about the other test.
It sounds like that test was always buggy, if Drew is right about the
cause.
- Macie
Dave
On Mon, Sep 28, 2009 at 4:40 PM, Eric Seidel <e...@webkit.org> wrote:
On Fri, Sep 25, 2009 at 2:42 PM, Maciej Stachowiak <m...@apple.com>
wrote:
> I like Dave Levin's idea that the first action should be to
instrument the
> tests so we can find out why they intermittently fail. Especially
if the
> failure is reproducible on the bots but not on developer systems.
I like this idea too. I don't like the reality that flakey tests are
a burden on all developers caused by one.
> Using the
> skip list should be a last resort, because that hides the failure
instead of
> helping us diangose the cause.
I (respectfully) disagree. I think we shouldn't be so afraid to skip
tests. We don't allow people to check in compiles which fail. We
don't allow people to check in tests which fail on other platforms
(without skipping them) or on every other run. Why should we allow
people to check in tests which fail every 10 runs? Or worse, why
should we leave a known flakey test checked in/un-attended which fails
every 10 runs?
If we can't easily roll-out the failing tests (or the commit which
cause them to start failing), we should skip them to keep the bots (a
shared resource) green, so as not to block other work on the project.
No?
I very much like WebKit's "everyone is responsible for the whole
project" culture, but I disagree that the burden of diagnosis should
be on the person trying to make a completely unrelated checkin (as is
the case when we leave flakey tests enabled in the tree).
-eric
p.s. I now have two "skipping flakey tests" changes up for review:
https://bugs.webkit.org/show_bug.cgi?id=29322
https://bugs.webkit.org/show_bug.cgi?id=29344
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
