(The current 'metadata of interest' being simply whether or not to run
the test and possibly which resources are required for an 'optional'
test, but there could certainly be more to it than that.)
The current scheme for "don't run this test" is like:
use Test::More;
plan skip_all => "bye" unless($ENV{SOME_RANDOM_VARIABLE});
...
Alternatively, you have to do something like Test::Manifest or
chromatic's custom disttest() action where the to-be-tested files are
determined (or not-to-be-tested files are filtered out) and a list of
files is fed to the harness.
With the first approach, only the tests know whether they should run.
With the latter, only the makefile (roughly) knows.
The second requires extra tools or code, the first requires pasting code
directly into the tests.
The first can use arbitrarily complex conditionals, while the second is
(in current implementations) limited to 1 or 2 variables.
Neither makes much (or any) information available to external systems.
The 'skip unless $ENV...' approach is not very discoverable. Answering
"Why are those tests skipping?" requires you to open the test in your
editor and read the code. This ad-hoc approach is likely to vary from
one distro to another even if the skip reason is exactly the same.
But putting all of the info in a config file might mean that it will be
ignored if the test is run by a tool which doesn't support that.
So, let's put all of the info in a config file and have the test look at
that iff it needs to. Test::Manifest does the first bit, albeit only
with a one-dimensional "Level" (afaict), but the test itself doesn't
know about this.
maybe:
# xt/misc/foo.t
use skiptest;
...
Where skiptest->import() checks caller(), loads testconfig.yml, and
decides whether to 'skip all' if $some_conditions are not met.
This puts the test metadata in the hands of external tools, but allows
the test itself to 'skip all' if it happens to get run by some
'uneducated' tool. It does require an external dependency.
For those eschewing dependencies, I'll note that the test in this
example is not in a location which would be executed by default. You
only need the dependency if you are doing something complicated enough
that directory layout can't cover it.
As for limitations of the config file -- I did not state any. It could
feasibly run some turing-complete code to determine some information.
The important thing is that an external tool can *somehow* acquire some
information about the test (vs the $ENV{THING}, where the only
information available is whether or not the test skipped -- you can't
learn from that whether or not it *will* skip or what is required to
make it not do so.)
--Eric
--
hobgoblin n 1: (folklore) a small grotesque supernatural creature that
makes trouble for human beings
---------------------------------------------------
http://scratchcomputing.com
---------------------------------------------------
