(The current 'metadata of interest' being simply whether or not to run 
the test and possibly which resources are required for an 'optional' 
test, but there could certainly be more to it than that.)

The current scheme for "don't run this test" is like:

  use Test::More;
  plan skip_all => "bye" unless($ENV{SOME_RANDOM_VARIABLE});

Alternatively, you have to do something like Test::Manifest or 
chromatic's custom disttest() action where the to-be-tested files are 
determined (or not-to-be-tested files are filtered out) and a list of 
files is fed to the harness.

With the first approach, only the tests know whether they should run.  
With the latter, only the makefile (roughly) knows.

The second requires extra tools or code, the first requires pasting code 
directly into the tests.

The first can use arbitrarily complex conditionals, while the second is 
(in current implementations) limited to 1 or 2 variables.

Neither makes much (or any) information available to external systems.

The 'skip unless $ENV...' approach is not very discoverable.  Answering 
"Why are those tests skipping?" requires you to open the test in your 
editor and read the code.  This ad-hoc approach is likely to vary from 
one distro to another even if the skip reason is exactly the same.

But putting all of the info in a config file might mean that it will be 
ignored if the test is run by a tool which doesn't support that.

So, let's put all of the info in a config file and have the test look at 
that iff it needs to.  Test::Manifest does the first bit, albeit only 
with a one-dimensional "Level" (afaict), but the test itself doesn't 
know about this.


  # xt/misc/foo.t
  use skiptest;

Where skiptest->import() checks caller(), loads testconfig.yml, and 
decides whether to 'skip all' if $some_conditions are not met.

This puts the test metadata in the hands of external tools, but allows 
the test itself to 'skip all' if it happens to get run by some 
'uneducated' tool.  It does require an external dependency.

For those eschewing dependencies, I'll note that the test in this 
example is not in a location which would be executed by default.  You 
only need the dependency if you are doing something complicated enough 
that directory layout can't cover it.

As for limitations of the config file -- I did not state any.  It could 
feasibly run some turing-complete code to determine some information.

The important thing is that an external tool can *somehow* acquire some 
information about the test (vs the $ENV{THING}, where the only 
information available is whether or not the test skipped -- you can't 
learn from that whether or not it *will* skip or what is required to 
make it not do so.)

hobgoblin n 1: (folklore) a small grotesque supernatural creature that
          makes trouble for human beings

Reply via email to