CMSP 25. Controlling test suite runs

[David Golden]

25. Controlling test suite runs

Proposal:

The META spec should add flags to indicate that distribution's
tests may be shuffled or run in parallel. (SlavenRezic)

Comments:

* The prove utility provides the interesting options --shuffle and --jobs.
  With --shuffle the test suite is not run in alphabetic but random order.
  With --jobs multiple test scripts run in parallel. This could speed up
  testing, especially on multi-processor machines.

* Unfortunately authors may write their test suites that neither one or the
  other option may be used: authors may have some "initialization" and
  "cleanup" script in the test suite (so shuffle cannot be used), or may
  use a file or other resource in many test scripts (so parallel runs
  cannot be used). So we cannot assume that --shuffle and --jobs may be
  used by default.

* A sample definition could look like this (please think of better key
  names):

    test_run:
      may_be_shuffled: yes
      may_run_in_parallel: yes

  Maybe even a control file to manage chunks/blocks that the module authors
  already set up (Tux).

    test_conf_file: t/t_conf

* I quite like this idea in principle, mostly because I've never seen a
  better suggestion for how the Makefile.PL or Build.PL could make parallel
  testing work. That said, it seems like it SHOULD be possible to make this
  happen without having to go via metadata (which is for the configuration
  to communicate to the CPAN client). Could the CPAN client really do
  anything about this? --Adam K