I'd propose an --ordering option, with three possible values: - random, randomizes the test list - natural, orders the test list in natural order, which is the current behavior - none, keeps the order that was specified in the arguments or test list, default (not certain about the name, though)
The --randomized option would be kept for backwards compatibility, but perhaps deprecated. I'll build up a patch that's based on the consensus. -Z On Thu, Nov 8, 2012 at 10:43 AM, Balazs Kelemen <kbal...@webkit.org> wrote: > On 11/08/2012 02:51 AM, Ojan Vafai wrote: > > On Wed, Nov 7, 2012 at 12:41 PM, Dirk Pranke <dpra...@chromium.org>wrote: > >> On Tue, Nov 6, 2012 at 11:58 PM, Žan Doberšek <zandober...@gmail.com> >> wrote: >> > Hi WebKitties, >> > >> > I'd like to see in what scale the --test-list option in NRWT is used and >> > whether anyone would object a change in how its usage behaves. >> > >> > Currently the test gathering takes into account any paths that were >> passed >> > in through the command line and any paths listed in the file to which a >> > possible --test-file option points[1]. These paths are then expanded if >> > necessary (due to platform-specific tests and virtual test suites) and >> all >> > the tests to which these paths apply are gathered[2]. All the gathered >> tests >> > are then either sorted into a natural order or randomized (if the >> > --randomize-order option is used)[3]. >> > >> > I'd like to propose a change in the behavior when using the --test-list >> > option. When used, the tests would be gathered only from the paths >> listed in >> > the file to which the option points. Also, the gathered tests would be >> > sorted to follow the order of the paths in the test list file. For >> instance, >> > consider the following two entries being placed in the test list file: >> > >> > c/d/e.html >> > a/b/ >> > >> > The current behavior would gather all the tests to which these two paths >> > apply and sort them in this order (on the premise that no other paths >> are >> > passed through the command line arguments): >> > >> > a/b/1.html >> > a/b/2.html >> > c/d/e.html >> > >> > The new behavior would place the c/d/e.html test at the top of the list >> but >> > sort the other two tests in order, like this: >> > >> > c/d/e.html >> > a/b/1.html >> > a/b/2.html >> > >> > The idea behind this is to be able to specify the exact order in which >> the >> > tests should be run. I'm currently playing around with a tool that >> randomly >> > chooses a certain number of tests, runs them and bisects down the first >> > regression that occurs (if any) to the smallest possible ordered list of >> > tests that reproduces that regression. The proposed change makes this a >> > simple task from the test listing perspective and I'd argue that exact >> test >> > ordering is the main point of an option such as --test-list. >> > >> > Currently I'm using an untested workaround of adding a new option that >> > specifies the tests should be reordered back to the original order that >> the >> > test list file provided, but I'd like to move on with implementing the >> > change in behavior if everyone feels it's OK and won't tamper with >> his/her >> > workflow. >> > >> > -Z >> > >> > >> > [1] >> > >> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_finder.py#L46 >> > [2] >> > >> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py#L576 >> > [3] >> > >> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py#L309 >> > >> >> Hi, >> >> I have used the option from time to time, and I have also wanted from >> time to time the ability to run tests in a specific order. This change >> would be fine by me. If people want to ensure that tests run in the >> old order, they can always sort the test list. >> > > I have used this in the past to identify which test caused test flakiness > (e.g. I have 100 tests and I know the last test depends on one of the other > 99 to run first in order to pass). The problem with this last sentence is > that it's hard to know what order run-webkit-tests would choose, so it's > hard to replicate it. It would make that use-case a little harder if we > made this change. Otherwise, I don't care either way. > > > Exactly for the reason Ojan mentioned I think the best would be to add an > option for defining the desired ordering behavior. Paths on command line > and in --test-list should imply to use the ordering as it is passed but it > would be possible to force the current method. I hope it would not add too > much complexity. > > -kbalazs > > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo/webkit-dev > >
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
