Recording to a local file the order in which tests are run on each thread should be relatively trivial. I filed https://bugs.webkit.org/show_bug.cgi?id=37396. That will help with flakiness that is simply order dependent, which I think is most of it. It won't help with flakiness that has to do with contention for other resources (e.g. ipv6 lookups as in https://bugs.webkit.org/show_bug.cgi?id=37104).
Ojan On Fri, Apr 9, 2010 at 9:34 PM, Dirk Pranke <dpra...@chromium.org> wrote: > On Fri, Apr 9, 2010 at 4:43 PM, Brady Eidson <beid...@apple.com> wrote: > > > > On Apr 9, 2010, at 4:41 PM, Alexey Proskuryakov wrote: > > > >> > >> On 09.04.2010, at 16:32, Brady Eidson wrote: > >> > >>>> - Exposes more flaky tests due to running tests in a non-deterministic > order. > >>> > >>> This sounds like a pro to me. > >> > >> > >> Existing run-webkit-tests can also do that, via a non-default option. > Unreproducible results with default options do sound like a con. > > > > Don't get me wrong - if a test fails reliably due to a specific run > order, then the tool needs to be able to record the run-order so the failure > can be explored. > > > > But finding more flakey tests by mixing up the run order is - in > principle - a good thing. > > > > I'd forgotten that the current tool can mixup the run-order. > > > > We should make the new tool record the order so order-specific failures > can be explored. > > > > There are two sources of non-determinism. > > One, which is easy to deal with, is the result of shuffling the > testing order. The new script supports this but doesn't use it by > default. > > The other is the result of timing and concurrency issues, and is not > so easily dealt with. The default code path shards by directory, and > as a result it is reasonably predictable which tests will run on which > thread. However, even in this situation you can get contention on the > filesystem and (much more common) the web server that can produce > timeouts and other unpredictable events. The > --experimental-fully-parallel code path makes the situation worse by > not even being able to easily say that any two tests will or won't be > executed sequentially by the same thread (the code uses a threadsafe > Python Queue object with N consumers). We also (I think) see a fair > amount of contention on the queue itself that contributes to timing > issues. > > -- Dirk > > > ~Brady > > > >> - WBR, Alexey Proskuryakov > >> > >> > > > > > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev