Re: todo tests in the TAP Plan
- Original Message
From: Shlomi Fish [EMAIL PROTECTED]
t/sample-tests/todo in the Test-Harness distribution reads:
print DUMMY_TEST;
1..5 todo 3 2;
ok 1
ok 2
not ok 3
ok 4
ok 5
DUMMY_TEST
As one can see, the 1..5 plan is followed by the todo 3 2; directive.
This is supposed to indicate something about plan ahead todo tests. (Instead
of the # TODO directives in the individual tests' outputs. Now:
1. t/sample-tests/todo is being run by Test-Harness, which seems to think
the not ok 3 is a todo test.
2. This todo-enabled plan is not documented in:
http://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod
3. TAPx-Parser version 0.20 cannot handle it.
--
I'd like to know what I should do about this feature, because right now I'm
trying to convert Test-Run to use TAPX::Harness, and this is giving me
problems.
Hmm, that's curious. However, if it's undocumented I would argue against
supporting it right now. What benefit does it gain us?
It also makes the grammar a tad more complicated (though not overly so). This
no longer works:
tap::= plan tests | tests plan
plan ::= '1..' positiveInteger \nInstead we'd have to do this:
tap ::= plan todoList \n tests | tests plan \n?
plan ::= '1..' positiveInteger
todoList ::= 'todo ' positiveInteger { ' ' positiveInteger }
We can only allow the todoList on a leading plan because one on a trailing plan
means we can't know the test results until the test run has completed (which
would suck in the case of infinite streams).
However, even allowing this on a leading plan is still problematic because
either the programmer needs to know the test numbers in advance to add that
list or the code somehow would need to deduce them. Then, we'd have to started
defining the semantics of what happens if we have that leading todoList in case
of one of those tests having a SKIP or TODO directive. I'm guessing that means
that leading todoList test numbers would be ignored in that case? I would
suggest that this is not a feature we want unless we can nail down its intent
and semantics quite carefully.
Does anyone know the rational behind it?
Cheers,
Ovid
--
Buy the book -- http://www.oreilly.com/catalog/perlhks/
Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
post-YAPC::Europe CPANTS news
Hi!
During YAPC::Europe in Birmingham I did a (rather short and un-hackish)
hackathon. Slides are available here:
http://domm.plix.at/talks/2006_birmingham_cpants/
During YAPC, I added a few new metrics:
- manifest_matches_dist
Check if the stuff listed in MANIFEST matches what's in the dist
- buildtool_not_executable
Check if the buildtool (Makefile.PL, Build.PL) are not executable
(and thus need to be called with 'perl Build.PL' thereby specifying
which exact version of Perl you want)
- has_example
An optional metric that checks if the author included a dir called
'eg|ex|example(s?)' which in turn includes at least on *.pl
This resulted in quite some movement in the CPANTS game...
I've also changed the display of the authors page
http://cpants.perl.org/author/DOMM
optional metrics (is_prereq and has_example) are now visually seperated
from the core metrics.
The site cpants.perl.org is still not very stable due to a
Catalyst/DBIx::Class-Bug (or more likely a bug in my usage of them)
Oh, and if you want to join the fun and help a bit, here's a (probably
incomplete) list of tasks:
- Metrics:
- has_license
- no_boilerplate
- has_valid_filenames
- declares_dependencies
- no_open_bugs
- Signature checking ?
- has_rating ?
- relation of subs to lines of code ?
- Scripts
- Generate a SQLite DB dump from the postgres DB
- Catalyst Frontend
- sqash bug
- Split up dist/view into several sub-pages
- API / Webservice
--
#!/usr/bin/perl http://domm.zsi.at
for(ref bless{},just'another'perl'hacker){s-:+-$-gprint$_.$/}
Re: todo tests in the TAP Plan
Selon Ovid [EMAIL PROTECTED]: As one can see, the 1..5 plan is followed by the todo 3 2; directive. This is supposed to indicate something about plan ahead todo tests. [...] I'd like to know what I should do about this feature, because right now I'm trying to convert Test-Run to use TAPX::Harness, and this is giving me problems. Hmm, that's curious. However, if it's undocumented I would argue against supporting it right now. What benefit does it gain us? This comes from the Good Old Test.pm module: $ perl -MTest -e 'plan tests = 10, todo = [2,4];' 1..10 todo 2 4; As there are quite some test scripts out there that use it, staying compatible with it sounds like a prerequisite. We can only allow the todoList on a leading plan because one on a trailing plan means we can't know the test results until the test run has completed (which would suck in the case of infinite streams). However, even allowing this on a leading plan is still problematic because either the programmer needs to know the test numbers in advance to add that list or the code somehow would need to deduce them. Then, we'd have to started defining the semantics of what happens if we have that leading todoList in case of one of those tests having a SKIP or TODO directive. I'm guessing that means that leading todoList test numbers would be ignored in that case? I would suggest that this is not a feature we want unless we can nail down its intent and semantics quite carefully. Test.pm could only handle predefined test numbers (it doesn't have a no_plan option like Test::More). -- Sébastien Aperghis-Tramoni Close the world, txEn eht nepO.
Re: todo tests in the TAP Plan
- Original Message From: Sébastien Aperghis-Tramoni [EMAIL PROTECTED] Hmm, that's curious. However, if it's undocumented I would argue against supporting it right now. What benefit does it gain us? This comes from the Good Old Test.pm module: $ perl -MTest -e 'plan tests = 10, todo = [2,4];' 1..10 todo 2 4; As there are quite some test scripts out there that use it, staying compatible with it sounds like a prerequisite. Ah, crud. I need to support it then. Bummer. I'll try to get a release out there when I can, then. Cheers, Ovid -- Buy the book -- http://www.oreilly.com/catalog/perlhks/ Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Namespace question (was Re: Counting files, lines, packages, subs in Perl)
What namespace should I use? So I've added a simplified counting of McCabe Complexity to my Perl::Code::Analayze - I copied the complexity measuring code from Perl::Critic (http://search.cpan.org/dist/Perl-Critic/lib/Perl/Critic/ Policy/Subroutines/ProhibitExcessComplexity.pm) I'm wondering if I should go ahead and put this in the CPAN, but with a different namespace - perhaps Perl::Metric::Simple or something? The (still alpha) code is at: http://g5-imac.matisse.net/~matisse/Perl-Code-Analyze/Perl-Code- Analyze-0.011/ There is an example script using the module: http://g5-imac.matisse.net/~matisse/Perl-Code-Analyze/Perl-Code- Analyze-0.011/examples/analyze.pl Results might be: % analyze.pl path/to/code perl files:76 lines: 21070 packages: 39 subs: 381 min. sub size: 1 lines max. sub size: 679 lines avg. sub size: 27.04 lines min. sub complexity: 1 max. sub complexity: 209 avg. sub complexity: 5.58 --- Matisse Enzer [EMAIL PROTECTED] http://www.matisse.net/ - http://www.eigenstate.net/ smime.p7s Description: S/MIME cryptographic signature
Re: Terrible diagnostic failure
Its been doing that for the last 10 years or so. Try an espresso. yeah, ok. Apache::Test, by default, sends diagnostics to STDERR. This is because by default it uses Test.pm which sends its errors to STDERR. right. I haven't actually used the Test.pm interface in ages. but most other people do I guess. seems like we should change the Test.pm behavior though - when running tests on the server we probably want STDERR to appear on the command line as well, not be lost in the error_log. Only when you explicitly say -withtestmore does it go to STDOUT. right. -withtestmore seems like it changes the behavior of Apache::Test yes. A-T's dependence on Test.pm predates my involvement, and there has been significant pushback to make it rely solely on T::B and friends. so, T::B support is a side venture for people like me who care. and seems a little broken so I doubt anyone's using it, I use it every day, and it's not broken for me... the tests don't, which is why you haven't noticed. yeah, very few tests for it, unfortunately. $ perl -Mblib -wle 'use Apache::Test qw(-withtestmore); plan tests = 1; ok 1, 2' 1..1 # Using Apache/Test.pm version 1.28 # You named your test '2'. You shouldn't use numbers for your test names. # Very confusing. ok 1 - 2 your turn for coffee :) $ perl -e 'use Test::More; plan tests = 1; ok 1,2' $ perl -e 'use Apache::Test qw(-withtestmore); plan tests = 1; is 1,2' --Geoff
Re: post-YAPC::Europe CPANTS news
On Wednesday 06 September 2006 02:53, Thomas Klausner wrote: - buildtool_not_executable Check if the buildtool (Makefile.PL, Build.PL) are not executable (and thus need to be called with 'perl Build.PL' thereby specifying which exact version of Perl you want) I'm not sure of the value of this one; how does an author make the buildtool executable on Windows, for example? I have the impression (not using Windows) that users must always call the tool with 'perl Build.PL' on such platforms. Am I mistaken? -- c
Re: post-YAPC::Europe CPANTS news
On 9/6/06, chromatic [EMAIL PROTECTED] wrote: On Wednesday 06 September 2006 02:53, Thomas Klausner wrote: - buildtool_not_executable Check if the buildtool (Makefile.PL, Build.PL) are not executable (and thus need to be called with 'perl Build.PL' thereby specifying which exact version of Perl you want) I'm not sure of the value of this one; how does an author make the buildtool executable on Windows, for example? I have the impression (not using Windows) that users must always call the tool with 'perl Build.PL' on such platforms. Am I mistaken? on windows the command interpreter uses file associations instead of C#!. files with extension C.pl are usually associated with Cperl.exe, and invoked without specifying Cperl on the command line. this is user-modifiable, so it's possible to configure files with the C.pl extension to be associated with another executable, or none at all. ~jerry
Re: post-YAPC::Europe CPANTS news
I could be wrong here, but I think the check is to make sure that tar doesn't set +x on Makefile.PL or Build.PL, thus forcing the user to run the proper version of perl instead of automagically running the perl that shebang points to. (Example: Makefile.PL says #!/usr/bin/perl, but you really want to run /home/jon/blead/bin/perl. Forcing you to type this out is a good thing.) I personally don't see the value of this, I always run perl Makefile.PL anyway. Do +x *.PL files cause problems with CPAN installs? (Or am I completely wrong in my interpretation of this test? :) Regards, Jonathan Rockway chromatic wrote: On Wednesday 06 September 2006 02:53, Thomas Klausner wrote: - buildtool_not_executable Check if the buildtool (Makefile.PL, Build.PL) are not executable (and thus need to be called with 'perl Build.PL' thereby specifying which exact version of Perl you want) I'm not sure of the value of this one; how does an author make the buildtool executable on Windows, for example? I have the impression (not using Windows) that users must always call the tool with 'perl Build.PL' on such platforms. Am I mistaken? -- c
Re: post-YAPC::Europe CPANTS news
On Wednesday 06 September 2006 10:27, Jonathan Rockway wrote: I could be wrong here, but I think the check is to make sure that tar doesn't set +x on Makefile.PL or Build.PL, thus forcing the user to run the proper version of perl instead of automagically running the perl that shebang points to. (Example: Makefile.PL says #!/usr/bin/perl, but you really want to run /home/jon/blead/bin/perl. Forcing you to type this out is a good thing.) Oh, right. Yes, that's obvious now. I misinterpreted it. -- c
Re: TAPx::Parser 0.20
On Sat, 2006-09-02 at 05:43 -0700, Ovid wrote: Further, though I don't know Tk or other windowing systems well enough, it should now be fairly easy to slap a GUI on TAPx::Parser. Here's an initial attempt at a Gtk2 interface. From running it against a few test suites and getting some strange results, it seems like TAPx::Parser doesn't like lines like ok 3 # comment It flags the corresponding result as being of type unknown. Also, this bit at the top of the program might be of interest if you wonder about performance: # Unfortunately, autoflush-ing seems to be a big performance problem. If you # don't care about real-time progress bars, turn this off. $writer - autoflush(1); -- Bye, -Torsten tprove_gtk Description: Perl program
Re: todo tests in the TAP Plan
Shlomi Fish wrote: t/sample-tests/todo in the Test-Harness distribution reads: print DUMMY_TEST; 1..5 todo 3 2; ok 1 ok 2 not ok 3 ok 4 ok 5 DUMMY_TEST As one can see, the 1..5 plan is followed by the todo 3 2; directive. This is supposed to indicate something about plan ahead todo tests. (Instead of the # TODO directives in the individual tests' outputs. Now: 1. t/sample-tests/todo is being run by Test-Harness, which seems to think the not ok 3 is a todo test. 2. This todo-enabled plan is not documented in: http://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod 3. TAPx-Parser version 0.20 cannot handle it. -- I'd like to know what I should do about this feature, because right now I'm trying to convert Test-Run to use TAPX::Harness, and this is giving me problems. These old style todo tests were never really documented until I stumbled on them inside Test::Harness and Test.pm, nobody really understood, aren't very useful and weren't really used. They were deprecated a while ago. Since they were never really documented or used I just hid the deprecation notice inside Test::Harness as it was only interesting to folks hacking on it. This notice used to appear in Test::Harness but it appears to have been removed. =begin _deprecated Alternatively, you can specify a list of what tests are todo as part of the test header. 1..23 todo 5 12 23 This only works if the header appears at the beginning of the test. This style is Bdeprecated. =end _deprecated Don't push them forward into TAP. Any feature which relies on static test numbering is broken. Just let them die.
Re: todo tests in the TAP Plan
Ovid wrote: $ perl -MTest -e 'plan tests = 10, todo = [2,4];' 1..10 todo 2 4; As there are quite some test scripts out there that use it, staying compatible with it sounds like a prerequisite. By it do you mean Test.pm or Test.pm's todo feature? The former I can believe. The latter... I'd be interested to know of a single use in the wild. And if there is anyone using it they'd likely be better served by moving to Test::Legacy and inline TODO tests. Or backporting inline TODO tests to Test.pm. Ah, crud. I need to support it then. Bummer. I'll try to get a release out there when I can, then. Don't bother, its a poorly designed feature and likely unused. I don't want to see it pushed forward into TAP.
Re: Terrible diagnostic failure
Geoffrey Young wrote: and seems a little broken so I doubt anyone's using it, I use it every day, and it's not broken for me... By broken I mean I was surprised to find the behavior of Apache::Test changed by changing the backend. I would have expected the interface to remain the same. Looking at it again it is clearly documented otherwise (I guess its not a backend after all) but nevertheless surprising to me. $ perl -Mblib -wle 'use Apache::Test qw(-withtestmore); plan tests = 1; ok 1, 2' 1..1 # Using Apache/Test.pm version 1.28 # You named your test '2'. You shouldn't use numbers for your test names. # Very confusing. ok 1 - 2 your turn for coffee :) $ perl -e 'use Test::More; plan tests = 1; ok 1,2' $ perl -e 'use Apache::Test qw(-withtestmore); plan tests = 1; is 1,2' --Geoff
Re: Test::Builder: mixing use_numbers on and off
Yitzchak Scott-Thoennes wrote: Is the avoid mixing warning out of date? Or in accord with future TAP plans? That dates back from the original version of Test::Builder in 2001. Test::Harness has handled no numbers and mixing of with and without numbers for as long as I know, but not always gracefully. Here's the TH that shipped with 5.5.4 (TH 1.1604 or something) $ perl -wle 'require /usr/local/src/perl5.005_04/lib/Test/Harness.pm; Test::Harness::runtests(@ARGV)' ~/devel/Test-Harness/t/nonumbers.t /Users/schwern/devel/Test-Harness/t/nonumbers. Argument isn't numeric in numeric gt () at /usr/local/src/perl5.005_04/lib/Test/Harness.pm line 118, GEN0 line 2. Argument isn't numeric in numeric gt () at /usr/local/src/perl5.005_04/lib/Test/Harness.pm line 118, GEN0 line 3. Argument isn't numeric in numeric gt () at /usr/local/src/perl5.005_04/lib/Test/Harness.pm line 118, GEN0 line 5. Argument isn't numeric in numeric gt () at /usr/local/src/perl5.005_04/lib/Test/Harness.pm line 118, GEN0 line 6. Argument isn't numeric in numeric gt () at /usr/local/src/perl5.005_04/lib/Test/Harness.pm line 118, GEN0 line 7. ok All tests successful. Files=1, Tests=6, 0 wallclock secs ( 0.00 cusr + 0.00 csys = 0.00 CPU) And its possible t/TEST (the Perl core harness) didn't handle the mixing well. Anyhow, I don't remember explicitly fixing it, must have just fixed itself. The warning against mixing in TB can probably go away.
Re: soliciting TAP comments from language designers (where to begin?)
jerry gay wrote: as i work on parrot, i happen to know a few language designers. i know they're interested in providing feedback on the proposed TAP changes, but (perhaps because i'm new to the list) i don't see a good thread to point them to. is there a summary of changes, or something i've missed that would be a good link to send to folks for their feedback? About all we've got that's organized is: http://perl-qa.yi.org/index.php/TAP_Proposals but its incomplete.
Re: post-YAPC::Europe CPANTS news
Thanks for the updates, Thomas. And now on with the complaining! Thomas Klausner wrote: - has_example An optional metric that checks if the author included a dir called 'eg|ex|example(s?)' which in turn includes at least on *.pl IMO examples in an example directory are a detriment, not a bonus. Any uninstalled documentation in the source directory will likely never been seen by the user (think about a CPAN shell installation) or even if they do see it it will be thrown out with the source tree once the module is installed and thus lost. Examples go in the installed docs or they are lost. Additionally, if you're going to keep this thing around, restricting the examples to .pl files penalizes HTML (Mason or other templating system), POD and module examples. [1] It would be best to just say that the directory contains something. [1] Also people like me who don't like to put a .pl extension onto their Perl executables. You don't run ls.c or cpp.sh. The user does not care and should not know what language their executable is in.
Re: todo tests in the TAP Plan
On Wednesday 06 September 2006 22:50, Michael G Schwern wrote: Shlomi Fish wrote: t/sample-tests/todo in the Test-Harness distribution reads: print DUMMY_TEST; 1..5 todo 3 2; ok 1 ok 2 not ok 3 ok 4 ok 5 DUMMY_TEST As one can see, the 1..5 plan is followed by the todo 3 2; directive. This is supposed to indicate something about plan ahead todo tests. (Instead of the # TODO directives in the individual tests' outputs. Now: 1. t/sample-tests/todo is being run by Test-Harness, which seems to think the not ok 3 is a todo test. 2. This todo-enabled plan is not documented in: http://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod 3. TAPx-Parser version 0.20 cannot handle it. -- I'd like to know what I should do about this feature, because right now I'm trying to convert Test-Run to use TAPX::Harness, and this is giving me problems. These old style todo tests were never really documented until I stumbled on them inside Test::Harness and Test.pm, nobody really understood, aren't very useful and weren't really used. They were deprecated a while ago. Since they were never really documented or used I just hid the deprecation notice inside Test::Harness as it was only interesting to folks hacking on it. This notice used to appear in Test::Harness but it appears to have been removed. =begin _deprecated Alternatively, you can specify a list of what tests are todo as part of the test header. 1..23 todo 5 12 23 This only works if the header appears at the beginning of the test. This style is Bdeprecated. =end _deprecated Don't push them forward into TAP. Any feature which relies on static test numbering is broken. Just let them die. OK, thanks. I'll forward-port this sample-tests script to the new # TODO syntax , and would not take measurements to handle this situation. Thanks for the heads up. Regards, Shlomi Fish - Shlomi Fish [EMAIL PROTECTED] Homepage:http://www.shlomifish.org/ Chuck Norris wrote a complete Perl 6 implementation in a day but then destroyed all evidence with his bare hands, so no one will know his secrets.
Re: TAPx::Parser 0.20
Torsten Schoenfeld wrote: Here's an initial attempt at a Gtk2 interface. From running it against a few test suites and getting some strange results, it seems like TAPx::Parser doesn't like lines like ok 3 # comment It flags the corresponding result as being of type unknown. And its correct. # is not the comment character, # is for directives. You might recognize this better like so: ok 4 # SKIP because I said so TAPx::Parser is seeing # comment and trying to interpret comment as a directive. Currently the only directives are skip and todo. Test::Harness silently ignores unknown directives. TAPx::Parser apparently does not. That's ok. To accomplish what you want, use - ok 3 - comment
Re: post-YAPC::Europe CPANTS news
Jonathan Rockway wrote: I could be wrong here, but I think the check is to make sure that tar doesn't set +x on Makefile.PL or Build.PL, thus forcing the user to run the proper version of perl instead of automagically running the perl that shebang points to. (Example: Makefile.PL says #!/usr/bin/perl, but you really want to run /home/jon/blead/bin/perl. Forcing you to type this out is a good thing.) I personally don't see the value of this, I always run perl Makefile.PL anyway. Do +x *.PL files cause problems with CPAN installs? (Or am I completely wrong in my interpretation of this test? :) There's a previous thread on this that explains the details. In short, the perl you call Makefile.PL with, is the one that automagically gets used later when calling make test, make install etc. So for cases where there is more than one Perl installed (and there's more than you'd think) it ensures that people are installing modules to the correct Perl, and prevents accidents. Adam K
Re: post-YAPC::Europe CPANTS news
Thomas Klausner wrote: Hi! During YAPC::Europe in Birmingham I did a (rather short and un-hackish) hackathon. Slides are available here: http://domm.plix.at/talks/2006_birmingham_cpants/ During YAPC, I added a few new metrics: - manifest_matches_dist Check if the stuff listed in MANIFEST matches what's in the dist - buildtool_not_executable Check if the buildtool (Makefile.PL, Build.PL) are not executable (and thus need to be called with 'perl Build.PL' thereby specifying which exact version of Perl you want) Glad to see this included, it caught about 20 modules of mine that had executable Makefile.PLs... although working out how the hell I managed to do that might take a little while, I've added another step to my release automation to stop it. - has_example An optional metric that checks if the author included a dir called 'eg|ex|example(s?)' which in turn includes at least on *.pl I thought we were generally negative on this one, because it would encourage people to spuriously add trivial example directories to their distributions... As a secondary point, if we keep it you might also want to include /sample(s)/ and /demo(s)/. This resulted in quite some movement in the CPANTS game... I've also changed the display of the authors page http://cpants.perl.org/author/DOMM optional metrics (is_prereq and has_example) are now visually seperated from the core metrics. I like the new style, not being red is a good thing. The site cpants.perl.org is still not very stable due to a Catalyst/DBIx::Class-Bug (or more likely a bug in my usage of them) Oh, and if you want to join the fun and help a bit, here's a (probably incomplete) list of tasks: - Metrics: - has_license - no_boilerplate - has_valid_filenames - declares_dependencies It might be an interesting idea to also add a dependencies_exist metric, that makes sure that all the dependencies that are declared actually exist in the CPAN. Dunno, could be of dubiously little value, but I just managed to somehow upload something with bad deps that I had installed on my local machine, but that weren't all on CPAN yet... - no_open_bugs If we are going to do this, we have to make sure that we DON'T include wishlist bugs in this metric, or it blows the validity all to hell. - Signature checking ? I looked into this a bit more recently, with all the Module::Signature changes. I think this is a bad idea. Signature checking HAS to be done by the CPAN client, and NOT ever by the module itself, because it limits the ability of the client to modify or tweak files (for example, we might potentially want to have Makefile.PL alter the META.yml to localise the deps for the current platform). By the time make test runs, it's WAY too late to be checking signatures. has_signature is fine, but another test along the lines of the Test::Pod and Test::Pod::Coverage ones would be a bad idea. - has_rating ? Bad idea I think as well, because it will encourage spurious ratings. This is the same reason that ratings aren't factored into the search.cpan result order. CPAN Ratings currently has most of it's value because it doesn't matter much, and so it encourages high quality reviews. Anything that would make the quality of the reviews worse (or malicious) isn't a negative thing. - relation of subs to lines of code ? Dubious. Makes a naive and blanket judgement call. Adam K
Re: post-YAPC::Europe CPANTS news
Anything that would make the quality of the reviews worse (or malicious) isn't a negative thing. Or rather, it IS a negative thing. :) Adam K
