+1
Do you have commit privileges on Apache-Test?
On Fri, May 20, 2011 at 1:51 PM, Rainer Jung <rainer.j...@kippdata.de> wrote:
> I noticed, that the SMOKE change in 1.36 also introduced a new sorting
> behaviour. In the change the test suite was running by default using the
> repeat order, that means the tests were ordered alpabetically.
>
> Now the default order is random.
>
> Furthermore the -order=rotate (A B C A B C) is gone, and -order=repeat
> now behaves like previously -order=rotate (it seems it previously was A
> A B B C C).
>
> I suggest:
>
> - keep behaviour and name of repeat as it is now
> - don't try to reimplement old repeat, because it might be tricky in
> combination with the new way iterations are done (SMOKE)
> - switch back to the new *repeat* (=old rotate) as the default for
> non-SMOKE tests, i.e. run the tests by default in sorted order and only
> do random if -order=random is explicitely given.
> - stay with random as default for SMOKE
>
> The following one line change should do that:
>
> Index: Apache-Test/lib/Apache/TestSort.pm
> ===================================================================
> --- Apache-Test/lib/Apache/TestSort.pm (revision 1125303)
> +++ Apache-Test/lib/Apache/TestSort.pm (working copy)
> @@ -54,7 +54,7 @@
> sub run {
> my($self, $list, $args) = @_;
>
> - my $order = $args->{order} || 'random';
> + my $order = $args->{order} || 'repeat';
> if ($order =~ /^\d+$/) {
> #dont want an explicit -seed option but env var can be a pain
> #so if -order is number assume it is the random seed
>
>
> Furthermore the READMEs and Changelog should get updated:
>
> ´Index: README
> ===================================================================
> --- README (revision 1125303)
> +++ README (working copy)
> @@ -77,18 +77,20 @@
>
> It's also possible that a test will pass when it's run after a
> particular test, but if moved to run after a different state it may
> -fail. For this reason it's important to ran the randomization smoking
> -testing with -order=random option.
> +fail. For this reason by default the tests run in random order.
>
> Since it's important to be able to reproduce the problem with the
> random testing, whenever -order=random is used, the used seed is
> printed to STDERR. Which can be then fed into the future tests with:
> via APACHE_TEST_SEED environment variable.
>
> +By adding the option -order=repeat, the tests will be run in
> +alphabetical order.
> +
> Combining these two important smoke testing techiques, one can run
> tests with:
>
> - t/SMOKE -times=N -order=(repeat|rotate|random)
> + t/SMOKE -times=N -order=(repeat|random)
>
> For example, to run the mod_rewrite tests 5 times, one would:
>
> @@ -98,8 +100,7 @@
> modes:
>
> - randomize all tests
> -- repeat the same test N times in sequence
> -- rotate the whole tests suite N times
> +- repeat the whole tests suite N times
>
> For configuration options and default settings run:
>
> Index: Apache-Test/README
> ===================================================================
> --- Apache-Test/README (revision 1125303)
> +++ Apache-Test/README (working copy)
> @@ -203,18 +203,15 @@
>
> run all tests 10 times in a random order (the seed is autogenerated
> and reported)
> -% t/TEST -times=10 -order=random
> +% t/SMOKE -times=10 -order=random
>
> run all tests 10 times in a random order using the seed obtained from
> the previous random run (e.g. 2352211):
> -% t/TEST -times=10 -order=2352211
> +% t/SMOKE -times=10 -order=2352211
>
> -rotate all tests 10 times (a, b, c, a, b, c)
> -% t/TEST -times=10 -order=rotate
> +repeat all tests 10 times (a, b, c, a, b, c)
> +% t/SMOKE -times=10 -order=repeat
>
> -repeat all tests 10 times (a, a, b, b, c, c)
> -% t/TEST -times=10 -order=repeat
> -
> When certain tests fail when running with -times option, you want to
> find out the minimal sequence of tests that lead to the
> failure. Apache::TestSmoke helps to ease this task, simply run:
> Index: Apache-Test/Changes
> ===================================================================
> --- Apache-Test/Changes (revision 1125303)
> +++ Apache-Test/Changes (working copy)
> @@ -8,6 +8,9 @@
>
> =item 1.37-dev
>
> +Run t/TEST tests by default in alphabetical order and
> +only t/SMOKE tests by default in random order.
> +
> Add t_file_watch_for to Apache::TestUtil [Torsten Foertsch]
>
> Add $boolean parameter to Apache::TestHandler::ok and
> Apache::TestHandler::ok1
>
> Regards,
>
> Rainer
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@perl.apache.org
For additional commands, e-mail: dev-h...@perl.apache.org
