Note: This reply is being sent to all 4 mailing lists that the
original was, though I am only subscribed to the first 2 myself, for
the benefit of completeness to the readers of the original.
------------------
As my initial reply, I'll bring up 2 recent discussions or Pugs
changes that affect this, so please keep them in mind when continuing
this thread.
1. First, I started a "making v6 test suite its own distro" thread on
just perl6-compiler which overlapped a lot with the intent of your
email, though went into some other details, and there were 4 replies:
* http://www.nntp.perl.org/group/perl.perl6.compiler/1434 - me
* http://www.nntp.perl.org/group/perl.perl6.compiler/1435 - Gaal
* http://www.nntp.perl.org/group/perl.perl6.compiler/1442 - me
* http://www.nntp.perl.org/group/perl.perl6.compiler/1448 - Gaal
* http://www.nntp.perl.org/group/perl.perl6.compiler/1449 - Jerry
Aside from proposing v6::tests as a separate distro that
implementation distros have as an external dependency, ...
In summary, I had proposed that unilateral SKIPs/TODOs etc would be
eliminated, and that a language-level feature, eg, a global %FEATURES
could be queryable at run time by programs, including the test suite,
to see if the current implementation claims to correctly implement
some features they want to use or test. A test suite would query
this for a feature, and output a SKIP if it returns false, and tests
the feature if it returns true, with PASS or FAIL.
At the very least, some language tests would have to be
unconditional, such as the ability to reference a predefined hash,
use basic conditionals, and strings, but once those are passing, it
is possible to use %FEATURES to conditionalize everything else.
We realized that, while this should work well long-term, and should
certainly work well for a framework written in a language that has
interchangeable backends and common tests for all, like DBI (or my
Rosetta), this may be too difficult to implement short-term for Perl
6 itself.
2. And so, within the last week, some commits to Pugs added the
ability for a SKIP/TODO to hardcode in the tests that it is just for
a particular version or version range of a particular implementation,
so it would either skip or not skip based on the declared version of
the interpreter, which would increment at release time. That is,
unconditional TODOs are conditional TODOs.
It seems that this is the current plan to do what you suggest in the
short term.
The short term would have the tests referencing specific
implementation versions, and the long term would have the tests
completely agnostic to what implementations or versions they run
under.
-- Darren Duncan
