A few months ago Larry proposed to add some testing
facilites to the language itself, because we want to
culturally encourage testing, and because the test
suite defines the language, so we need to specify the
behaviour of our testing facilities anyway.

We also discussed some possible changes to the current
testing syntax, please read the IRC discussion starting
from here:

There are some problems with the current approach,
especially if we make it built-in:

    * the word 'is' is overloaded in Perl 6, it is used
      for traits (class A is rw { ... }),
      implementation types (my @a is TiedArray) and
      inheritance (class A { is B; ... }). Especially
      the the last one can appear in the same position
      as an is() test, and means something completely

   * if we export subs is() and ok(), we clutter the
     namespace with subs with short names - not very nice

   * is() is rather imprecise; it doesn't say *how*
     things are compared. Currently two options seem to
     be open for the comparison semantics: either
     string based, or deep structural equality (with
     infix:<eqv>).  Both are very problematic: The
     stringification of some structures can be
     implementation specific (for example for Ranges),
     and it often looses lot of information. Also pugs
     used to stringify hashes in insertion order, but
     that's not forced by the spec, so many tests
     relied on that behaviour, ie they were wrong.
     Comparison by infix:<eqv> is dangerous, because it
     is very strict, and things that were identical in
     the early stages of the compiler become distinct
     later on. Consider 'sub f { return 1, 2, 3}; say
     f() eqv (1, 2, 3)'. Pugs says this is true,
     because it doesn't implement return() as returning
     a capture (or in turn thinks that lists are eqv to
     captures), and testers would rely on this.
     Future versions of a compiler would contradict the
     previous results, and thus put an additional burden
     on the test suite maintainers (ie mostly me). There
     are other subtle distinctions (like between List
     and Array) that make it hard to write tests
     with infix:<eqv> correctly.

So we want to get rid if the is() test function. The
current workaround is to use ok() and an explicit
comparison operator like this:

    ok $x == 1e5, 'with explicit numeric comparison';

However this just tells us if a test fails, not how it
fails - it would be nice to have something like
Test::More's output "Expected 1e5, got 1e4".  The only
way that ok() could produce such diagnostics would be
to declare it a macro that peeks into the AST of its
first argument and tries to find the two values. But
that's very advanced magic, and might not even be
possible in a DWIMmy way in the general case.

So Larry and Patrick developed the idea of creating an
adverb on the test operator instead:

    $x == 1e5   :ok('the :ok makes this is a test');

This is an adverb on the infix:<==> operator, and might
desugar to something like this:

multi sub infix:<==>($left, $right, :$ok) {
    $*TEST_BACKEND.proclaim($left == $right, $ok)
        or $*TEST_BACKEND.diag(
                "Got: «$left.perl()»; Expected: «$right.perl»");

(Daniel Ruoso also proposed to call the adverb :test
instead of :ok, making it easier to read but a bit
longer; my happiness doesn't depend on the exact name,
but of course we can discuss it once we have settled
on this scheme, if we do so).

So every operator that returns a boolean would get a
multi with the named argument :ok, which could be
autogenerated for most operators, but hand-written for
others that need more verbose diagnostics (for example
the smart match operator could tell you which of its
hundred possible comparisons it used). That's not just
limited to infix operators, ok($x, '$x is true') could
be written as ?$x :ok('$x is true'), where the :ok is
an adverb on prefix:<?>;

This approach gives us
 * good and easy diagnostics
 * exactness by forcing the explicit choice of
   comparison semantics
 * nice integration into the Perl 6 syntax
 * no cluttering of the namespace with short subs

However nothing in life is free, we pay for it with a
few disadvantages:
 * We nearly double the number of built-in operators
   by adding an :ok multi
 * We force implementors to handle operator adverbs
   and named arguments very early in their progress
   (don't know how easy or hard that is)
 * Testing of operators becomes somewhat clumsy. If you
 * want to test infix:<==>, you won't write
   '2 == 2 :ok("== works")', because you test a
   different multi there. Instead you'd have to write
   something like '?(2 == 2) :ok("== works")', where
   :ok is an adverb on prefix:<?>.

So I'd like to hear your opinions: do you think
adverb-based testing is a good idea? If you don't like
it, do you see any other good way to tackle the
problems I mentioned above?

I'll send another mail on the subject of pluggable
testing backends in order to allow different emitters
(TAP output, storage into databases, whatever)


Reply via email to