Hi tech@,
The patch below adds a target, "random", to bsd.regress.mk, as
well as a description of the new target to bsd.regress.mk(5).
The "random" target is the same as "regress", except that the
ordering of the regression targets is randomized each time.
My thinking is that statically ordered regression runs can mask
bugs in the software under test and the tests themselves.
For small utilities this is unlikely, but for long-running daemons,
kernel modules, etc., it is more likely because these programs are
more complex.
In general, a test can put your system into a state that allows a
subsequent test to pass when it would have otherwise failed.
Randomly ordered regression runs could help to weed these issues
out. Even if no such bugs exist, your randomly ordered regression
runs are stronger statements about your software's correctness and
the independence of each test case.
Any takers? Thoughts?
--
Scott Cheloha
Index: share/man/man5/bsd.regress.mk.5
===================================================================
RCS file: /cvs/src/share/man/man5/bsd.regress.mk.5,v
retrieving revision 1.11
diff -u -p -r1.11 bsd.regress.mk.5
--- bsd.regress.mk.5 16 Jan 2017 07:00:03 -0000 1.11
+++ bsd.regress.mk.5 18 Mar 2017 21:24:38 -0000
@@ -50,6 +50,10 @@ test programs.
.It Cm depend
Build any dependencies required to carry out the current set
of regression tests.
+.It Cm random
+Identical to
+.Cm regress ,
+but regression targets are executed in a random order.
.It Cm regress
Executes all of the regression targets defined in the Makefile.
If one of the tests fails, the line
Index: share/mk/bsd.regress.mk
===================================================================
RCS file: /cvs/src/share/mk/bsd.regress.mk,v
retrieving revision 1.13
diff -u -p -r1.13 bsd.regress.mk
--- bsd.regress.mk 19 Jul 2014 18:15:53 -0000 1.13
+++ bsd.regress.mk 18 Mar 2017 22:23:40 -0000
@@ -83,6 +83,11 @@ REGRESS_SKIP_TARGETS+=${REGRESS_ROOT_TAR
. endif
.endif
+.ifmake random
+REGRESS_TARGETS!=echo ${REGRESS_TARGETS} | tr ' ' '\n' | sort -R
+.endif
+random: regress
+
regress: .SILENT
.if ! ${REGRESS_LOG:M/*}
echo =========================================================
