A few months ago Larry proposed to add some testing facilites to the language itself, because we want to culturally encourage testing, and because the test suite defines the language, so we need to specify the behaviour of our testing facilities anyway.
We also discussed some possible changes to the current testing syntax, please read the IRC discussion starting from here: http://irclog.perlgeek.de/perl6/2008-10-09#i_613827 There are some problems with the current approach, especially if we make it built-in: * the word 'is' is overloaded in Perl 6, it is used for traits (class A is rw { ... }), implementation types (my @a is TiedArray) and inheritance (class A { is B; ... }). Especially the the last one can appear in the same position as an is() test, and means something completely different * if we export subs is() and ok(), we clutter the namespace with subs with short names - not very nice * is() is rather imprecise; it doesn't say *how* things are compared. Currently two options seem to be open for the comparison semantics: either string based, or deep structural equality (with infix:<eqv>). Both are very problematic: The stringification of some structures can be implementation specific (for example for Ranges), and it often looses lot of information. Also pugs used to stringify hashes in insertion order, but that's not forced by the spec, so many tests relied on that behaviour, ie they were wrong. Comparison by infix:<eqv> is dangerous, because it is very strict, and things that were identical in the early stages of the compiler become distinct later on. Consider 'sub f { return 1, 2, 3}; say f() eqv (1, 2, 3)'. Pugs says this is true, because it doesn't implement return() as returning a capture (or in turn thinks that lists are eqv to captures), and testers would rely on this. Future versions of a compiler would contradict the previous results, and thus put an additional burden on the test suite maintainers (ie mostly me). There are other subtle distinctions (like between List and Array) that make it hard to write tests with infix:<eqv> correctly. So we want to get rid if the is() test function. The current workaround is to use ok() and an explicit comparison operator like this: ok $x == 1e5, 'with explicit numeric comparison'; However this just tells us if a test fails, not how it fails - it would be nice to have something like Test::More's output "Expected 1e5, got 1e4". The only way that ok() could produce such diagnostics would be to declare it a macro that peeks into the AST of its first argument and tries to find the two values. But that's very advanced magic, and might not even be possible in a DWIMmy way in the general case. So Larry and Patrick developed the idea of creating an adverb on the test operator instead: $x == 1e5 :ok('the :ok makes this is a test'); This is an adverb on the infix:<==> operator, and might desugar to something like this: multi sub infix:<==>($left, $right, :$ok) { $*TEST_BACKEND.proclaim($left == $right, $ok) or $*TEST_BACKEND.diag( "Got: «$left.perl()»; Expected: «$right.perl»"); } (Daniel Ruoso also proposed to call the adverb :test instead of :ok, making it easier to read but a bit longer; my happiness doesn't depend on the exact name, but of course we can discuss it once we have settled on this scheme, if we do so). So every operator that returns a boolean would get a multi with the named argument :ok, which could be autogenerated for most operators, but hand-written for others that need more verbose diagnostics (for example the smart match operator could tell you which of its hundred possible comparisons it used). That's not just limited to infix operators, ok($x, '$x is true') could be written as ?$x :ok('$x is true'), where the :ok is an adverb on prefix:<?>; This approach gives us * good and easy diagnostics * exactness by forcing the explicit choice of comparison semantics * nice integration into the Perl 6 syntax * no cluttering of the namespace with short subs However nothing in life is free, we pay for it with a few disadvantages: * We nearly double the number of built-in operators by adding an :ok multi * We force implementors to handle operator adverbs and named arguments very early in their progress (don't know how easy or hard that is) * Testing of operators becomes somewhat clumsy. If you * want to test infix:<==>, you won't write '2 == 2 :ok("== works")', because you test a different multi there. Instead you'd have to write something like '?(2 == 2) :ok("== works")', where :ok is an adverb on prefix:<?>. So I'd like to hear your opinions: do you think adverb-based testing is a good idea? If you don't like it, do you see any other good way to tackle the problems I mentioned above? I'll send another mail on the subject of pluggable testing backends in order to allow different emitters (TAP output, storage into databases, whatever) Cheers, Moritz
