On Thu, Nov 8, 2012 at 2:00 AM, Žan Doberšek <[email protected]> wrote:
> I'd propose an --ordering option, with three possible values: > - random, randomizes the test list > - natural, orders the test list in natural order, which is the current > behavior > - none, keeps the order that was specified in the arguments or test list, > default (not certain about the name, though) > > The --randomized option would be kept for backwards compatibility, but > perhaps deprecated. I'll build up a patch that's based on the consensus. > Sounds good to me. I don't think we need to keep the --randomized option though. Fewer flags == better. We already have too many flags. > > -Z > > > On Thu, Nov 8, 2012 at 10:43 AM, Balazs Kelemen <[email protected]>wrote: > >> On 11/08/2012 02:51 AM, Ojan Vafai wrote: >> >> On Wed, Nov 7, 2012 at 12:41 PM, Dirk Pranke <[email protected]>wrote: >> >>> On Tue, Nov 6, 2012 at 11:58 PM, Žan Doberšek <[email protected]> >>> wrote: >>> > Hi WebKitties, >>> > >>> > I'd like to see in what scale the --test-list option in NRWT is used >>> and >>> > whether anyone would object a change in how its usage behaves. >>> > >>> > Currently the test gathering takes into account any paths that were >>> passed >>> > in through the command line and any paths listed in the file to which a >>> > possible --test-file option points[1]. These paths are then expanded if >>> > necessary (due to platform-specific tests and virtual test suites) and >>> all >>> > the tests to which these paths apply are gathered[2]. All the gathered >>> tests >>> > are then either sorted into a natural order or randomized (if the >>> > --randomize-order option is used)[3]. >>> > >>> > I'd like to propose a change in the behavior when using the --test-list >>> > option. When used, the tests would be gathered only from the paths >>> listed in >>> > the file to which the option points. Also, the gathered tests would be >>> > sorted to follow the order of the paths in the test list file. For >>> instance, >>> > consider the following two entries being placed in the test list file: >>> > >>> > c/d/e.html >>> > a/b/ >>> > >>> > The current behavior would gather all the tests to which these two >>> paths >>> > apply and sort them in this order (on the premise that no other paths >>> are >>> > passed through the command line arguments): >>> > >>> > a/b/1.html >>> > a/b/2.html >>> > c/d/e.html >>> > >>> > The new behavior would place the c/d/e.html test at the top of the >>> list but >>> > sort the other two tests in order, like this: >>> > >>> > c/d/e.html >>> > a/b/1.html >>> > a/b/2.html >>> > >>> > The idea behind this is to be able to specify the exact order in which >>> the >>> > tests should be run. I'm currently playing around with a tool that >>> randomly >>> > chooses a certain number of tests, runs them and bisects down the first >>> > regression that occurs (if any) to the smallest possible ordered list >>> of >>> > tests that reproduces that regression. The proposed change makes this a >>> > simple task from the test listing perspective and I'd argue that exact >>> test >>> > ordering is the main point of an option such as --test-list. >>> > >>> > Currently I'm using an untested workaround of adding a new option that >>> > specifies the tests should be reordered back to the original order >>> that the >>> > test list file provided, but I'd like to move on with implementing the >>> > change in behavior if everyone feels it's OK and won't tamper with >>> his/her >>> > workflow. >>> > >>> > -Z >>> > >>> > >>> > [1] >>> > >>> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_finder.py#L46 >>> > [2] >>> > >>> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py#L576 >>> > [3] >>> > >>> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py#L309 >>> > >>> >>> Hi, >>> >>> I have used the option from time to time, and I have also wanted from >>> time to time the ability to run tests in a specific order. This change >>> would be fine by me. If people want to ensure that tests run in the >>> old order, they can always sort the test list. >>> >> >> I have used this in the past to identify which test caused test >> flakiness (e.g. I have 100 tests and I know the last test depends on one of >> the other 99 to run first in order to pass). The problem with this last >> sentence is that it's hard to know what order run-webkit-tests would >> choose, so it's hard to replicate it. It would make that use-case a little >> harder if we made this change. Otherwise, I don't care either way. >> >> >> Exactly for the reason Ojan mentioned I think the best would be to add an >> option for defining the desired ordering behavior. Paths on command line >> and in --test-list should imply to use the ordering as it is passed but it >> would be possible to force the current method. I hope it would not add too >> much complexity. >> >> -kbalazs >> >> _______________________________________________ >> webkit-dev mailing list >> [email protected] >> http://lists.webkit.org/mailman/listinfo/webkit-dev >> >> > > _______________________________________________ > webkit-dev mailing list > [email protected] > http://lists.webkit.org/mailman/listinfo/webkit-dev > >
_______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo/webkit-dev
