On Fri, Apr 9, 2010 at 4:43 PM, Brady Eidson <[email protected]> wrote: > > On Apr 9, 2010, at 4:41 PM, Alexey Proskuryakov wrote: > >> >> On 09.04.2010, at 16:32, Brady Eidson wrote: >> >>>> - Exposes more flaky tests due to running tests in a non-deterministic >>>> order. >>> >>> This sounds like a pro to me. >> >> >> Existing run-webkit-tests can also do that, via a non-default option. >> Unreproducible results with default options do sound like a con. > > Don't get me wrong - if a test fails reliably due to a specific run order, > then the tool needs to be able to record the run-order so the failure can be > explored. > > But finding more flakey tests by mixing up the run order is - in principle - > a good thing. > > I'd forgotten that the current tool can mixup the run-order. > > We should make the new tool record the order so order-specific failures can > be explored. >
There are two sources of non-determinism. One, which is easy to deal with, is the result of shuffling the testing order. The new script supports this but doesn't use it by default. The other is the result of timing and concurrency issues, and is not so easily dealt with. The default code path shards by directory, and as a result it is reasonably predictable which tests will run on which thread. However, even in this situation you can get contention on the filesystem and (much more common) the web server that can produce timeouts and other unpredictable events. The --experimental-fully-parallel code path makes the situation worse by not even being able to easily say that any two tests will or won't be executed sequentially by the same thread (the code uses a threadsafe Python Queue object with N consumers). We also (I think) see a fair amount of contention on the queue itself that contributes to timing issues. -- Dirk > ~Brady > >> - WBR, Alexey Proskuryakov >> >> > > _______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
