I noticed, that the SMOKE change in 1.36 also introduced a new sorting
behaviour. In the change the test suite was running by default using the
repeat order, that means the tests were ordered alpabetically.
Now the default order is random.
Furthermore the -order=rotate (A B C A B C) is gone, and -order=repeat
now behaves like previously -order=rotate (it seems it previously was A
A B B C C).
I suggest:
- keep behaviour and name of repeat as it is now
- don't try to reimplement old repeat, because it might be tricky in
combination with the new way iterations are done (SMOKE)
- switch back to the new *repeat* (=old rotate) as the default for
non-SMOKE tests, i.e. run the tests by default in sorted order and only
do random if -order=random is explicitely given.
- stay with random as default for SMOKE
The following one line change should do that:
Index: Apache-Test/lib/Apache/TestSort.pm
===================================================================
--- Apache-Test/lib/Apache/TestSort.pm (revision 1125303)
+++ Apache-Test/lib/Apache/TestSort.pm (working copy)
@@ -54,7 +54,7 @@
sub run {
my($self, $list, $args) = @_;
- my $order = $args->{order} || 'random';
+ my $order = $args->{order} || 'repeat';
if ($order =~ /^\d+$/) {
#dont want an explicit -seed option but env var can be a pain
#so if -order is number assume it is the random seed
Furthermore the READMEs and Changelog should get updated:
´Index: README
===================================================================
--- README (revision 1125303)
+++ README (working copy)
@@ -77,18 +77,20 @@
It's also possible that a test will pass when it's run after a
particular test, but if moved to run after a different state it may
-fail. For this reason it's important to ran the randomization smoking
-testing with -order=random option.
+fail. For this reason by default the tests run in random order.
Since it's important to be able to reproduce the problem with the
random testing, whenever -order=random is used, the used seed is
printed to STDERR. Which can be then fed into the future tests with:
via APACHE_TEST_SEED environment variable.
+By adding the option -order=repeat, the tests will be run in
+alphabetical order.
+
Combining these two important smoke testing techiques, one can run
tests with:
- t/SMOKE -times=N -order=(repeat|rotate|random)
+ t/SMOKE -times=N -order=(repeat|random)
For example, to run the mod_rewrite tests 5 times, one would:
@@ -98,8 +100,7 @@
modes:
- randomize all tests
-- repeat the same test N times in sequence
-- rotate the whole tests suite N times
+- repeat the whole tests suite N times
For configuration options and default settings run:
Index: Apache-Test/README
===================================================================
--- Apache-Test/README (revision 1125303)
+++ Apache-Test/README (working copy)
@@ -203,18 +203,15 @@
run all tests 10 times in a random order (the seed is autogenerated
and reported)
-% t/TEST -times=10 -order=random
+% t/SMOKE -times=10 -order=random
run all tests 10 times in a random order using the seed obtained from
the previous random run (e.g. 2352211):
-% t/TEST -times=10 -order=2352211
+% t/SMOKE -times=10 -order=2352211
-rotate all tests 10 times (a, b, c, a, b, c)
-% t/TEST -times=10 -order=rotate
+repeat all tests 10 times (a, b, c, a, b, c)
+% t/SMOKE -times=10 -order=repeat
-repeat all tests 10 times (a, a, b, b, c, c)
-% t/TEST -times=10 -order=repeat
-
When certain tests fail when running with -times option, you want to
find out the minimal sequence of tests that lead to the
failure. Apache::TestSmoke helps to ease this task, simply run:
Index: Apache-Test/Changes
===================================================================
--- Apache-Test/Changes (revision 1125303)
+++ Apache-Test/Changes (working copy)
@@ -8,6 +8,9 @@
=item 1.37-dev
+Run t/TEST tests by default in alphabetical order and
+only t/SMOKE tests by default in random order.
+
Add t_file_watch_for to Apache::TestUtil [Torsten Foertsch]
Add $boolean parameter to Apache::TestHandler::ok and
Apache::TestHandler::ok1
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@perl.apache.org
For additional commands, e-mail: dev-h...@perl.apache.org
