This is a more or less random collection of thoughts and questions about
the rakudo testing infrastructure.
First of all I feel that it's in a rather good shape.
I put up a chart of number of tests on http://rakudo.de/ and plan to
update it regularly (via cron job once the admin installs Text::CSV_XS).
"make localtest" is very handy, among other things for fudging new test
files.
But it would be useful to have an option that makes localtest more
verbose. Specifically if a script dies, it's not obvious after which
test it died. Is there an easy way to pass an option for more verbosity?
Or should we introduce localtest-verbose make target?
I have two scripts in my local tools/ directory, both of which aren't
platform independent (only tested on linux so far). One writes a patch
for automagically unfudging tests (aka autounfudge.pl), the other
generates the chart on rakudo.de. What should I do with them? Are they
of general interest, and should live in the parrot repo? if yes, should
I send patches and ask for testing on other platforms before adding? Or
should I put them somewhere else (pugs repo, entirely different repo,
whatever)
isa_ok in pugs and rakudo seem to work differently. Many tests call
isa_ok with a string as the second argument:
isa_ok(0, 'Int');
while rakudo only understands the bare type name:
isa_ok(0, Int);
It seems to me that the latter is more intuitive, but I'd still like
some feedback before changing the tests on a larger scale.
There seems to be a thorough confusion about numeric types. for example
some tests read like this: is (1.1).WHAT, 'Num'; and then in a different
file is (1.1).WHAT, 'Rat'; That raises two questions for me
1) should we test for the value of .WHAT at all (unless for testing WHAT
of course)? It seems to me that a compiler should have the freedom to
return a different subtype (for example 1 could be a PositiveInt as long
as PositiveInt is an Int; that's a contrived example, but you get the
picture)
2) How do we know which numeric type is a class and which is a role? Is
there an explicit spec about the types of number literals? That could
have some impact on type checking in the tests. (Maybe I should prod p6l
with that).
I've attached a list of skip messages from spectest_regression, roughly
sorted into categories, some messages commented by me (in parenthesis).
Finally I'd like to point out some annoyances with rakudo. It's rather
frustrating (for example while fudging a large test file) to get an
error message that can't easily be mapped to a location in the source file.
The two most frequent ones are the Null PMC access in the type checker
(as reported by Ovid), and "Method 'lvalue' not found for invocant of
class 'PAST::Stmts'", which I finally tracked down to ($x, $y) = (1, 2).
The latter seems to fit into the category "not yet implemented". If it
is indeed, it would be nice to have it fail with line number information
(I guess the various Past:: nodes already have line information attached
- would it be possible to print that whenever something fails that
involves Past nodes? Perhaps that information wouldn't be too accurate,
but it's already a success if I know which part of the file I should be
looking at).
Sorry for being a bit lengthy, I'd welcome any comments.
Cheers,
Moritz
--
Moritz Lenz
http://moritz.faui2k3.org/ | http://perl-6.de/
Feature
4 skipped: calling postcircumfix:<()> with a dot
9 skipped: whatever star
1 skipped: whatever star not implemented
7 skipped: prefix:<\>
21 skipped: prefix:<\> not implemented
4 skipped: int, num, undef assign not implemented
1 skipped: Can't compare to undef (?)
3 skipped: {} hash composer not implemented
6 skipped: eqv and {} as hash composer not implemented
1 skipped: @() should be the same as @($/)
9 skipped: CATCH blocks not implemented
4 skipped: No pointy blocks on while loops yet
20 skipped: method fallback to sub unimpl
100 skipped: No method unpolar yet
20 skipped: No method unpolar
5 skipped: promote Int to Num
2 skipped: colon invocant syntax
2 skipped: no hashes at the moment
18 skipped: adverbial closure
6 skipped: adverbial closure
2 skipped: adverbial closure
5 skipped: adverbial closure
2 skipped: method fallback unimplemented
12 skipped: colon invocant syntax
1 skipped: %p doesn't yet throw exception
3 skipped: Inf and NaN not yet implemented
6 skipped: module Num not implemented
2 skipped: NaN not implemented
42 skipped: can't parse pi (actually exp() and prefix:<-> for Complex)
1 skipped: unicode
2 skipped: unicode
2 skipped: unicode
Bugs
2 skipped: lexicals bug RT#56184
4 skipped: weird errors
2 skipped: underscores cause weird IMCC errors
3 skipped: type checking bug
4 skipped: try{} does not keep test from failing
1 skipped: zip and @@ are broken
1 skipped: test passes but Parrot bug gets in the way
1 skipped: Arity problem
4 skipped: no sub srand() (actually a bug in type checker)
Parse
10 skipped: only the return type "of" parses correctly
1 skipped: can't parse
3 skipped: $ $a parsefail
1 skipped: '(@ 3)' parsefail
1 skipped: "(@ $a)" parsefail
2 skipped: parsefail: sigil coercion
5 skipped: parsefail: meta cross op not implemented
1 skipped: parsefail: meta cross op not implemented, eval dies
9 skipped: parse failure: pointy block as an expression
7 skipped: parsefail
1 skipped: can't parse
2 skipped: parsefail
24 skipped: parsefail
3 skipped: don't parse
1 skipped: can't parse
4 skipped: parsefail
Dunno/Dubious/Unclassified
1 skipped: zip does not work
1 skipped: zip is broken (probably missing slice context)
4 skipped: Won't test &connect as environment variable "PERL_TESTS_ALLOW_NETWORK" is not true.
1 skipped: feature
1 skipped: eval not implemented
5 skipped: unspecced
