The branch master has been updated
via c678f68a19638c1e2bbfee6a7a1d8d728976ce66 (commit)
via a21a1c23c97163d341d1caf07c2160fcb68506b0 (commit)
from 2f06c34b0eb658e80d82be8f7f0b54305a9cf68b (commit)
- Log -----------------------------------------------------------------
commit c678f68a19638c1e2bbfee6a7a1d8d728976ce66
Author: Pauli <paul.d...@oracle.com>
Date: Sat Dec 12 21:38:17 2020 +1000
test: document the random test ordering env variable
Reviewed-by: Tomas Mraz <tm...@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/13672)
commit a21a1c23c97163d341d1caf07c2160fcb68506b0
Author: Pauli <paul.d...@oracle.com>
Date: Sat Dec 12 21:25:40 2020 +1000
test: print OPENSSL_TEST_RAND_ORDER=x when a randomised test fails.
The previous message "random seed x" is a lot less descriptive.
Reviewed-by: Tomas Mraz <tm...@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/13672)
-----------------------------------------------------------------------
Summary of changes:
test/README.md | 20 ++++++++++++++++++++
test/testutil/driver.c | 2 +-
2 files changed, 21 insertions(+), 1 deletion(-)
diff --git a/test/README.md b/test/README.md
index 43f8471120..9a5c91953d 100644
--- a/test/README.md
+++ b/test/README.md
@@ -148,3 +148,23 @@ for `TAP::Harness` to know more.
To run up to four tests in parallel at any given time:
$ make HARNESS_JOBS=4 test
+
+Randomisation of Test Ordering
+------------------------------
+
+By default, the test harness will execute tests in the order they were added.
+By setting the `OPENSSL_TEST_RAND_ORDER` environment variable to zero, the
+test ordering will be randomised. If a randomly ordered test fails, the
+seed value used will be reported. Setting the `OPENSSL_TEST_RAND_ORDER`
+environment variable to this value will rerun the tests in the same
+order. This assures repeatability of randomly ordered test runs.
+This repeatability is independent of the operating system, processor or
+platform used.
+
+To randomise the test ordering:
+
+ $ make OPENSSL_TEST_RAND_ORDER=0 test
+
+To run the tests using the order defined by the random seed `42`:
+
+ $ make OPENSSL_TEST_RAND_ORDER=42 test
diff --git a/test/testutil/driver.c b/test/testutil/driver.c
index 9343dfb016..0b4332b492 100644
--- a/test/testutil/driver.c
+++ b/test/testutil/driver.c
@@ -256,7 +256,7 @@ PRINTF_FORMAT(2, 3) static void test_verdict(int verdict,
test_flush_stderr();
if (verdict == 0 && seed != 0)
- test_printf_tapout("# random seed: %d\n", seed);
+ test_printf_tapout("# OPENSSL_TEST_RAND_ORDER=%d\n", seed);
test_printf_tapout("%s ", verdict != 0 ? "ok" : "not ok");
va_start(ap, description);
test_vprintf_tapout(description, ap);
