Re: running tests in parallel

[Akim Demaille]

It would be nice if it were possible to run tests in parallel using

on a single CPU is annoying.  As most 'make' implementations already
support running in parallel (-j), automake could just utilize this
functionality to runt tests in parallel.

automake. Waiting for tests to complete on a 4-way box while they are run

I very much agree.

I have been using for quite a while the following framework which
brings several benefits over the current framework.  I wanted to
have time to write bits in Automake to avoid GNU Make syntax,
but I didn't find enough time.  Maybe someone will want to pick
up the ball.

The features are:

- It can be run in parallel.

- A single run produces the quiet version on screen and
  the verbose version in a log, so you no longer have to
  run the tests twice.  A message similar to Autotest's
  invites the user to send the log to the reporting address.

- It can rely on dependencies, so if you have tests that need
  not be rerun if their sources are not updated, then they are
  not rerun at all.  On a project of mine, this is a significant
  speed up

- It is meant to produce reStructured Text, so if you produce
  logs in RST too, the resulting log file is easy to convert
  into clickable html.

- The display on screen uses colors to catch attention on
  failures.

- It gives the path to the directory that had the failure,
  which is quite handy when there are several test suites.

- The log file is not created if everything went right,
  so a call to find can easily reveal the directories
  where the failures occurred.

- It is meant to support several test-suites in a single
  directory, but this requires help from Automake.  I had in
  mind something like

  TEST_SUITES = test-suite-1 test-suite-2
  test_suite_1_SOURCES = test1.tst test2.chk test3.tst
  test_suite_1_XFAILS = test1.tst

  etc.  But this requires hacking Automake, and I did not take
  time to do it.


Here is the file that provides these features:

check.mk
Description: Binary data

Here is an example of use that supports another extension
than .test or $(EXEEXT):

----------------------------------------------------------
# The test suite is composed of *.chk files.  Each one must create a
# *.log file that we want to keep.
TESTS_ENVIRONMENT = srcdir=$(srcdir) USE_VALGRIND=$(USE_VALGRIND)
TESTS = $(notdir $(wildcard $(srcdir)/checkfiles/*.chk))
XFAIL_TESTS =                                   \
        events.chk                              \
        lazy-test-eval.chk                      \
        tag.chk
TEST_LOGS = $(TESTS:.chk=.log)
# .PRECIOUS: $(TEST_LOGS)

# Parallel test framework.
include $(top_srcdir)/build-aux/check.mk

# From a test file to a log file.
%.log: $(srcdir)/checkfiles/%.chk
        @$(am__check_pre) $(srcdir)/uconsole-check $${dir}$< $(am__check_post)
EXTRA_DIST += uconsole-check

.PHONY: clean-test-dirs
clean-local: clean-test-dirs
clean-test-dirs:
        rm -rf $(TESTS:.chk=.dir)
----------------------------------------------------------


Here is an example of textual log created this way, followed
by its HTML conversion (when running make check-html).  The
idea to use RST was made by Alexandre Duret-Lutz.

test-suite.log
Description: Binary data

Calculator 1.0: tests/test-suite.log

   1 of 34 tests failed.

   Contents
     * [1]FAIL: events.chk (exit: 1)
          + [2]Input
          + [3]output

[4]FAIL: events.chk (exit: 1)

[5]Input

   Input:
1 + 2;
1 + 2;
1 + 2;

[6]output

   Expected output:
3
3
3

   Effective output:
1
2
3

   Diffs on output:
--- Expected output (output.exp)
+++ Effective output (output.eff)
@@ -1,3 +1,3 @@
-3
-3
+1
+2
 3

References

   1. file://localhost/tmp/tmprGS3qn.html#fail-events-chk-exit-1
   2. file://localhost/tmp/tmprGS3qn.html#input
   3. file://localhost/tmp/tmprGS3qn.html#output
   4. file://localhost/tmp/tmprGS3qn.html#id1
   5. file://localhost/tmp/tmprGS3qn.html#id2
   6. file://localhost/tmp/tmprGS3qn.html#id3


I'd be happy to help if I can do something for this to be
included in Automake.