Re: v0.3 [Was: CPAN Upload: R/RB/RBS/Test-Differences-0.2.tar.gz]
On Sat, 15 Dec 2001 02:41:13 -0500 Barrie Slaymaker <[EMAIL PROTECTED]> wrote: > > IMHO Test::Differences' benefit is its mix-in nature. Changing > > each is_deeply() to eq_or_diff_data() hand by hand "if not ok" > > seems a little hassle to me. > > Why is it more effort than doing something like: > > perl -ipe 's/is_deeply/eq_or_diff' t/*.t Again, IMHO we can get the benefit of Test::Differences like this: * First, test the code with Test::More::is() is $complicated_long_text, $expected; * but it fails. Test::More's output is too long for me to find what's the differecne with the two. * Then we want the output switch to Text::Diff's one. Add something like: use Test::Differences qw(:globally_override_test_more); Or with command line: perl -Mblib -MTest::Differences=:globally... t/foo.t Switching each function line by line, or adding magical C shortcut in the top of script will do. But I wish there's a handy way to do it. Thanks. -- Tatsuhiko Miyagawa <[EMAIL PROTECTED]>
Re: v0.3 [Was: CPAN Upload: R/RB/RBS/Test-Differences-0.2.tar.gz]
On Sat, Dec 15, 2001 at 06:52:40PM +0900, Tatsuhiko Miyagawa wrote: > Again, IMHO we can get the benefit of Test::Differences like this: > > * First, test the code with Test::More::is() > > is $complicated_long_text, $expected; > > * but it fails. > > Test::More's output is too long for me to find what's the > differecne with the two. > > * Then we want the output switch to Text::Diff's one. Why not just use Test::Differences's function in the first place? -- Michael G. Schwern <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/ Perl Quality Assurance <[EMAIL PROTECTED]> Kwalitee Is Job One "You can't set a generic cow." -- Randal Schwartz
Re: v0.3 [Was: CPAN Upload: R/RB/RBS/Test-Differences-0.2.tar.gz]
On Sat, 15 Dec 2001 05:26:01 -0500 Michael G Schwern <[EMAIL PROTECTED]> wrote: > Why not just use Test::Differences's function in the first place? Two cases for using Test::* 1) Testing our application (for paying work) it's good to use T::D in the first place. 2) Testing my CPAN module I don't like letting go Test::More for it ;) -- Tatsuhiko Miyagawa <[EMAIL PROTECTED]>
Re: v0.3 [Was: CPAN Upload: R/RB/RBS/Test-Differences-0.2.tar.gz]
Now I see what you're after with the -M approach, thanks for the
example.
On Sat, Dec 15, 2001 at 07:54:17PM +0900, Tatsuhiko Miyagawa wrote:
> Two cases for using Test::*
>
> 1) Testing our application (for paying work)
>
> it's good to use T::D in the first place.
We're agreed here :).
> 2) Testing my CPAN module
>
> I don't like letting go Test::More for it ;)
You can still have Test::More. Do you mean you don't want the extra
dependency like:
PREREQ_PM => {
"Test::More" => 0,
"Test::Differences" => 0,
}
?
I can understand that :) since CPAN is/seems a non-started for a
significant number of folks (I have a Win32 system that it's
nonfuncitonal on; haven't debugged it yet).
How about having Test::More use eq_or_diff_data() in is_deeply() if it
cane eval "use Test::Differences;" and use it's current routine if it
can't? That's as seamless as you can get.
- Barrie
Re: v0.3 [Was: CPAN Upload: R/RB/RBS/Test-Differences-0.2.tar.gz]
On Sat, Dec 15, 2001 at 01:56:19AM -0500, Michael G Schwern wrote:
> On Sat, Dec 15, 2001 at 02:41:13AM -0500, Barrie Slaymaker wrote:
> > There's also a test script for the escaping now :). I'd love to test
> > characters for codepoints > 0xff, but they're borken / unsupported in
> > all released perls, I think.
>
> Broken is such an ugly word. "Experimental" :)
:)
> 5.6.1's unicode implementation should be usable enough for your
> purposes. You might want to have a file full of Unicode tests that
> only runs if you've got a Perl >= 5.6.1
Well, when trying to descape chr( 0x0100 ) and chr( 0xfffd ) without
"use utf8", I get:
not ok 12
# Test 12 got: '\xc4\x80' (t/00escape.t at line 23)
#Expected: '\x{0100}'
not ok 13
# Test 13 got: '\xef\xbf\xbd' (t/00escape.t at line 24)
#Expected: '\x{fffd}'
When I "use utf8" in Differences.pm and try to escape chr( 0xff ), I get:
ok 10
Malformed UTF-8 character (byte 0xff) in substitution (s///) at
blib/lib/Test/Differences.pm line 216.
Malformed UTF-8 character (byte 0xff) in ord at
blib/lib/Test/Differences.pm line 216.
Segmentation fault (core dumped)
The core dump is speaking to me. When I channel it, a low, demented
voice (well, more demented than normal) comes from me saying "DON'T USE
UTF8 HERE". So I just don't test for that and I don't "use utf8". I
haven't had time to poke and prod at it to see if I can work around it.
Here's a small demo of the "Malformed" error on chr(0xff) from test 11:
$ perl -le ' print join ",", map sprintf( "%02x", ord), chr( 0xff ) =~
/([\000])/'
$ perl -le 'use utf8; print join ",", map sprintf( "%02x", ord), chr( 0xff ) =~
/([\000])/'
Malformed UTF-8 character (byte 0xff) in pattern match (m//) at -e line 1.
It really seems to be matching against
And here's a one liner for the core dump:
$ perl -le 'use utf8; ( $s = chr( 0xff )) =~ s/([^\000])/sprintf "f"/e'
Malformed UTF-8 character (byte 0xff) in substitution (s///) at -e line 216.
Segmentation fault (core dumped)
Throwing a split at it seems to be better than a regexp if I use utf8:
$ perl -le 'use utf8; print join ",", map sprintf( "%02x", ord), ( split //, chr(
0xff ) . chr( 0x100 ) . chr( 0xfffd ) . chr( 0x ) )'
Malformed UTF-8 character (character 0x) in ord at -e line 1. ff,100,fffd,00
but horks if I don't:
$ perl -le 'print join ",", map sprintf( "%02x", ord), ( split //, chr( 0xff ) .
chr( 0x100 ) . chr( 0xfffd ) . chr( 0x ) )'
Malformed UTF-8 character (unexpected non-continuation byte 0x00 after start byte
0xc3) in ord at -e line 1.
Malformed UTF-8 character (unexpected continuation byte 0xbf) in ord at -e line 1.
Malformed UTF-8 character (unexpected non-continuation byte 0x00 after start byte
0xc4) in ord at -e line 1.
Malformed UTF-8 character (unexpected continuation byte 0x80) in ord at -e line 1.
Malformed UTF-8 character (unexpected non-continuation byte 0x00 after start byte
0xef) in ord at -e line 1.
Malformed UTF-8 character (unexpected continuation byte 0xbf) in ord at -e line 1.
Malformed UTF-8 character (unexpected continuation byte 0xbd) in ord at -e line 1.
Malformed UTF-8 character (unexpected non-continuation byte 0x00 after start byte
0xef) in ord at -e line 1.
Malformed UTF-8 character (unexpected continuation byte 0xbf) in ord at -e line 1.
Malformed UTF-8 character (unexpected continuation byte 0xbf) in ord at -e line 1.
00,00,00,00,00,00,00,00,00,00
I'll need to dig in to this a bit more with unpack, etc. I guess, unless
somebody who knows and loves 5.6's utf8-ness can see some happy place
I'm missing.
For reference, the regex causing the grief in the first place is:
sub _escape($) {
my $s = shift ;
$s =~ s{([^\040-\177])}{
exists $escapes{$1}
? $escapes{$1}
: sprintf( "\\x{%04x}", ord $1 ) ;
}ge;
$s;
}
If anyone can see a good way to walk this string byte by byte on older
perls and do either that or get automatic adjustments to utf8 mode to
work on 5.6.1, I'll be quite grateful.
- Barrie
Re: v0.3 [Was: CPAN Upload: R/RB/RBS/Test-Differences-0.2.tar.gz]
On Sat, Dec 15, 2001 at 07:54:17PM +0900, Tatsuhiko Miyagawa wrote: > 2) Testing my CPAN module > > I don't like letting go Test::More for it ;) If I understand you, they work together fine. -- Michael G. Schwern <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/ Perl Quality Assurance <[EMAIL PROTECTED]> Kwalitee Is Job One I know you get this a lot, but what's an unholy fairy like you doing in a mosque like this?
Re: v0.3 [Was: CPAN Upload: R/RB/RBS/Test-Differences-0.2.tar.gz]
On Sat, 15 Dec 2001 07:10:33 -0700, Barrie Slaymaker wrote: > How about having Test::More use eq_or_diff_data() in is_deeply() if it cane > eval "use Test::Differences;" and use it's current routine if it can't? That's > as seamless as you can get. Wild idea: How about splitting out the error-reporting, with a nice default in Test::Builder or wherever, using optional modules if possible? use Test::More tests => 100, report => Test::Differences; -- c
Re: v0.3 [Was: CPAN Upload: R/RB/RBS/Test-Differences-0.2.tar.gz]
On Sat, Dec 15, 2001 at 10:52:07AM -0700, chromatic wrote: > On Sat, 15 Dec 2001 07:10:33 -0700, Barrie Slaymaker wrote: > > > How about having Test::More use eq_or_diff_data() in is_deeply() if it cane > > eval "use Test::Differences;" and use it's current routine if it can't? That's > > as seamless as you can get. > > Wild idea: > > How about splitting out the error-reporting, with a nice default in > Test::Builder or wherever, using optional modules if possible? > > use Test::More > tests => 100, > report => Test::Differences; The two modules can already work together in the same script. So, if you have Test::Differences, which has superior complex data structure handling, why are you calling is_deeply() in the first place? -- Michael G. Schwern <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/ Perl Quality Assurance <[EMAIL PROTECTED]> Kwalitee Is Job One my mother once said "never eat paste my darling" would that I heeded -- stimps
Re: v0.3 [Was: CPAN Upload: R/RB/RBS/Test-Differences-0.2.tar.gz]
> On Sat, 15 Dec 2001 13:18:23 -0500, Michael G Schwern <[EMAIL PROTECTED]> said: MGS> On Sat, Dec 15, 2001 at 10:52:07AM -0700, chromatic wrote: >> Wild idea: >> >> How about splitting out the error-reporting, with a nice default in >> Test::Builder or wherever, using optional modules if possible? >> >> use Test::More >> tests => 100, >> report => Test::Differences; MGS> The two modules can already work together in the same script. So, if MGS> you have Test::Differences, which has superior complex data structure MGS> handling, why are you calling is_deeply() in the first place? Still it is quite intersting idea to separate tests and results printing. Imaging that you can have pluggable output modules. So with one module you can have compact output with just ok/not ok, with another module you can build detailed report. I have already something similar in unreleased version of new HTTP::WebTest. It supports various report plugins so test results can be represented in several ways. -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- | Ilya Martynov (http://martynov.org/) TIV.net (http://tiv.net/) | | GnuPG 1024D/323BDEE6 D7F7 561E 4C1D 8A15 8E80 E4AE BE1A 53EB 323B DEE6 | -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Re: v0.3 [Was: CPAN Upload: R/RB/RBS/Test-Differences-0.2.tar.gz]
On Sat, Dec 15, 2001 at 01:18:23PM -0500, Michael G Schwern wrote:
> On Sat, Dec 15, 2001 at 10:52:07AM -0700, chromatic wrote:
> > On Sat, 15 Dec 2001 07:10:33 -0700, Barrie Slaymaker wrote:
> >
> > > How about having Test::More use eq_or_diff_data() in is_deeply() if it cane
> > > eval "use Test::Differences;" and use it's current routine if it can't? That's
> > > as seamless as you can get.
> >
> > Wild idea:
> >
> > How about splitting out the error-reporting, with a nice default in
> > Test::Builder or wherever, using optional modules if possible?
> >
> > use Test::More
> > tests => 100,
> > report => Test::Differences;
>
> The two modules can already work together in the same script. So, if
> you have Test::Differences, which has superior complex data structure
> handling, why are you calling is_deeply() in the first place?
The reason I was suggesting an automatic remapping of is_deeply() to be
eq_or_diff_data() inside Test::More is so that you can write a script
that only uses Test::More, but if you install Test::Differences, you get
the fancier output.
This would address Tatsuhiko Miyagawa's issue of wanting to minimize the
prerequisites (and thus the hassle) for CPANed modules, yet leverage
Test::Differences when needed.
A snippet of code is worth a kiloword, so this is what I mean. First,
rename sub is_deeply to sub _is_deeply in More.pm, then something like:
eval <<'EOCODE1' || eval <<'EOCODE2' ;
require Test::Differences;
sub is_deeply {
goto &eq_or_diff_data;
}
1;
EOCODE1
sub is_deeply {
goto &_is_deeply
}
EOCODE2
or equivalent is what I mean.
Seems very DWIMmerly to me, but YMMV, of course :).
Whether or not you want to alter Test::More's API to offer pluggable
reporting modules is another question; I'm not deeply concerned about
that.
- Barrie
Re: Untested modules update: There's more than we thought
On Fri, Dec 14, 2001 at 05:20:24PM -0700, chromatic wrote: > On Fri, 14 Dec 2001 15:18:43 -0700, Michael G Schwern wrote: > > > ExtUtils::MM_Cygwin > > Here's a test for that. It could use someone on Cygwin testing it. Tels said > he would, but apparently got busy. Thanks, applied. -- $jhi++; # http://www.iki.fi/jhi/ # There is this special biologist word we use for 'stable'. # It is 'dead'. -- Jack Cohen
Good news/bad news about perlcc
The bad news is perlcc is completely shot, and has been since 5.6.1. The bytecode and Perl->C compiler is completely busted and nobody's stepped forward to fix it. The last people to do so was Simon Cozens and tchrist. Simon's busy with Parrot and Tom's gotten a life outside Perl now. Malcolm Beattie, the original author, has also gone to live in the big blue room. The good news is, if its broken and nobody can fix it, we can't do much to test it! :) After throwing some basic perlcc sanity checks into the core, I'm scratching B::Assembler, B::Bytecode, B::C, B::CC and B::Disassembler (can't disassemble something you can't assemble) off the list. The magic number is 28. #B::Assembler [perlcc irreparably broken, can't test] #B::Bytecode[ditto] #B::C [ditto ditto] #B::CC [again] #B::Concise [[EMAIL PROTECTED]] #B::Disassembler[can't disassemble what you can't assemble] #B::Lint[[EMAIL PROTECTED]] #B::Stackobj[[EMAIL PROTECTED]] #B::Xref[[EMAIL PROTECTED]] Byteloader CPAN Dynaloader ExtUtils::MM_NW5 ExtUtils::MM_OS2 ExtUtils::MM_Unix ExtUtils::MM_VMS ExtUtils::MM_Win32 ExtUtils::Install ExtUtils::Installed ExtUtils::Liblist ExtUtils::Mksymlists #lib [Brent Dax] Net::Cmd Net::Domain Net::Ping Net::POP3 O Pod::Functions (no docs!) Pod::Html Pod::InputObjects Pod::Select Pod::Usage re -- Michael G. Schwern <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/ Perl Quality Assurance <[EMAIL PROTECTED]> Kwalitee Is Job One Milk is timeless.
is_op -> cmp_ok. is() remains as it is.
Ok, here's what I've decided. The is_ok() thing will be cmp_ok(). Best name I could think of. The current behavior of is() wrt undef, 0 and '' being considered different will remain. I've decided is() is its own thing and the small bit of magical behavior is Good. unlike() and diag() will be added next version. -- Michael G. Schwern <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/ Perl Quality Assurance <[EMAIL PROTECTED]> Kwalitee Is Job One "His plagiarism was limited only by this faulty technique." -- Peter Schickele
Re: Untested modules update: There's more than we thought
On Sat, Dec 15, 2001 at 11:46:29PM +, Nicholas Clark wrote: > On Sat, Dec 15, 2001 at 05:59:22PM -0500, Michael G Schwern wrote: > > On Sat, Dec 15, 2001 at 11:48:29PM +0100, Tels wrote: > > > Te"One day Math::Big* will have more tests than Perl;)"ls > > > > One day Perl will have more tests than Bit::Vector. 67255 at last count. > > Ilya chose not to attempt that for 5.6, by make op/numconvert.t only report > 1 ok (or not) for a cluster of 15 numbers passed through the same conversions. > We could always change op/numconvert.t to report 15 times as many tests > (about another 20,000 I think) Finer granularity is often better than lumping them all together, makes it easier to figure out which one is failing. numconvert.t gets around this by the way it reports its failures with the big blat of diagnostics. In the interests of myself never wanting to touch numconvert.t for fear of it getting angry, digitizing and being thrown onto the game grid, do whatever you feel is best with it. :) > But that's cheating, isn't it? It may be useless, but its fun to see who can pee the farthest. ;) > More on topic - I like Test::More. > I like test.pl in the core. > But I fear that we aren't bootstrapping our tests carefully enough. > In that I'm pretty sure we shouldn't be using Test::More until we're > sure we've tested all the ops Already done. By the Treaty Of t/, Test::More's domain is limited to ext/ and lib/. And Andy's fears about its widespread use leading to a cascade failure has come true, as it collapsed on djgpp and took a good chunk of lib/ with it. Still working out my djgpp environment to look at that. > and we shouldn't be using test.pl until we've tested all the ops it > uses (and be careful to limit the ops and the complexity of ops it > uses) Sort of. t/test.pl is written to avoid the more twitchy things which Test::More plays with (like duping filehandles) and does keep the "peculiar style" axiom in mind. However, trying to limit the use of t/test.pl to only things which it doesn't use at all can get complicated and somewhat tedious as it leaves us without a test library for good chunk of the ops. It also gets into the tricky problem of finely ordering the tests to they run in some sort of dependency order, and I really don't want to get into that right now. t/test.pl using, say, caller() and it also being used in t/op/caller.t (which, umm... I can't seem to find) is fine as long as t/test.pl limits itself to the basic, rock-solid functionality of caller. If t/test.pl started using things like the more obscure bits of caller() that's a bit more likely to break (say, the $hints) I'd be worried. > TEST starts: > > # This is written in a peculiar style, since we're trying to avoid > # most of the constructs we'll be testing for. Which is sort of a white lie these days, but its still quite a bit simpler than Test::Harness. -- Michael G. Schwern <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/ Perl Quality Assurance <[EMAIL PROTECTED]> Kwalitee Is Job One It sure is fun masturbating. http://www.unamerican.com/
Re: Untested modules update: There's more than we thought
On Sat, Dec 15, 2001 at 02:19:21PM -0800, Kurt D. Starsinic wrote: > On Dec 15, Dave Rolsky wrote: > > Ok, so that's a bit off the topic of "why use isa_ok()" but I just don't > > see why people seem to object to the use of Test::More in the core Perl > > tests. I can't see how it couldn't help improve the quality of the tests > > while providing a standardized way to do the things that all the tests do > > anyway. > > I, for one, don't object to the use of Test::More in the core Perl > tests. I'm just being responsibly skeptical about the promotion of many > different test harness modules with many different functions. It runs > the risk of _seeming_ like the People in Charge don't know what testing > is for, and of not providing effective guidance to those people writing > their first tests. We have Test::Simple, Test::Tutorial and eventually Test::FAQ for such guidance. Work on those if you're worried about providing effective guidance. Beyond that, I'm not going to cut down my toolbox to only a hammer just because it looks tidy. (And its PERL! Of course you're going to have 25 different types of bevels in the toolbox.) Its also been "discovered" just recently that its now possible to inter-mix several different Test::* modules in the same program, as long as they all use the Test::Builder backend (and Barrie found a way to cheat with Test.pm). http:[EMAIL PROTECTED]/msg01091.html This means its no longer necessary to write monolithic test modules. You can write special purpose modules for special purpose things and combine them as you please. After the next release, which will complete the interface with cmp_ok() (aka. is_op), unlike() and eval_ok(), Test::More's growth is pretty much over. I'll leave the rest to seperate modules. > I also think that we have a real testing problem, which is that it > isn't _trivially_ easy to run a huge test suite (like Alzabo's, or Perl's) > and to find out at the end which tests failed and what their output was. > 3000 lines and 20 minutes later, a message that 3 out of 1 tests > failed is not maximally useful. Greater variety of test functions is not > (IMNSHO) nearly as important as addressing that. Test::More outputs its diagnostics to STDERR, so they appear as you run 'make test'. PERL_DL_NONLAZY=1 /usr/local/bin/perl -Iblib/arch -Iblib/lib -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.t t/Module-InfoNOK 3# Failed test (t/Module-Info.t at line 15) # got: 'foo' # expected: 'bar' t/Module-Infook 73/74# Looks like you planned 74 tests but ran 1 extra. t/Module-Infodubious Test returned status 2 (wstat 512, 0x200) DIED. FAILED test 3 Failed 1/74 tests, 98.65% okay Failed Test Stat Wstat Total Fail Failed List of Failed --- t/Module-Info.t2 512741 1.35% 3 Failed 1/1 test scripts, 0.00% okay. 0/74 subtests failed, 100.00% okay. Even in the core: $ ./perl TEST ../lib/foo.t t/../lib/foo# Failed test (../lib/foo.t at line 10) # got: 'foo' # expected: 'bar' # Looks like you failed 1 tests of 1. FAILED at test 1 Failed 1 test script out of 1, 0.00% okay. Its a little messy, but it works. I'd welcome help making it look nicer. However t/test.pl (the low-fat Test::More substitute which almost everything in t/ uses) puts its diagnostics on STDOUT which gets sucked up by Test::Harness. That could be changed. As for providing further information at the end of a Perl core test run, that requires modifying t/TEST which has its own set of mental health problems. Finally, if you want to see real movement in improving the state of 'make test' output, help out finishing up Test::Harness 2.00 http://www.pobox.com/~schwern/src/Test-Harness-2.00_01.tar.gz and finishing the ExtUtils::* tests so we can work on that safely (the lives of MakeMaker and Test::Harness being intertwined). -- Michael G. Schwern <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/ Perl Quality Assurance <[EMAIL PROTECTED]> Kwalitee Is Job One An official "I want James Earl Jones' cock up my ass" t-shirt. http://www.goats/com/archive/010303.html
Re: Untested modules update: There's more than we thought
On Sat, Dec 15, 2001 at 11:46:29PM +, Nicholas Clark wrote: > On Sat, Dec 15, 2001 at 05:59:22PM -0500, Michael G Schwern wrote: > > On Sat, Dec 15, 2001 at 11:48:29PM +0100, Tels wrote: > > > Te"One day Math::Big* will have more tests than Perl;)"ls > > > > One day Perl will have more tests than Bit::Vector. 67255 at last count. > > Ilya chose not to attempt that for 5.6, by make op/numconvert.t only report > 1 ok (or not) for a cluster of 15 numbers passed through the same conversions. > We could always change op/numconvert.t to report 15 times as many tests > (about another 20,000 I think) > > But that's cheating, isn't it? My Gedcom module has 7671 tests. Most of these are from checking a 1476 line file five times. I could easily make that file bigger ... Although really I'd prefer to make it smaller. It used to take about 15 mins to run on my old Dec. Nowadays that's not such a priority :-) > More on topic - I like Test::More. > I like test.pl in the core. > But I fear that we aren't bootstrapping our tests carefully enough. > In that I'm pretty sure we shouldn't be using Test::More until we're > sure we've tested all the ops, and we shouldn't be using test.pl until > we've tested all the ops it uses (and be careful to limit the ops > and the complexity of ops it uses) > > TEST starts: > > # This is written in a peculiar style, since we're trying to avoid > # most of the constructs we'll be testing for. > > Maybe it should print that out on each run, to remind everyone :-) I agree. That is, I share your fears. I also wonder about TEST. Last time I did a patch for it I was surprised at just what constructs it did include. Maybe all the constructs it uses are tested earlier, but after all the fingers that have been in the pie, it's probably time for someone to take a good look at what gets tested where to make sure that the various test programs do only use previously tested constructs as far as possible. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Untested modules update: There's more than we thought
On Sat, Dec 15, 2001 at 11:48:29PM +0100, Tels wrote: > Te"One day Math::Big* will have more tests than Perl;)"ls One day Perl will have more tests than Bit::Vector. 67255 at last count. -- Michael G. Schwern <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/ Perl Quality Assurance <[EMAIL PROTECTED]> Kwalitee Is Job One Is there an airport nearby or is that just my tae-kwon-do taking off?
Re: Untested modules update: There's more than we thought
-BEGIN PGP SIGNED MESSAGE- Moin, On 15-Dec-01 Jarkko Hietaniemi tried to scribble about: >> Ok, so that's a bit off the topic of "why use isa_ok()" but I just don't >> see why people seem to object to the use of Test::More in the core Perl >> tests. I can't see how it couldn't help improve the quality of the >> tests >> while providing a standardized way to do the things that all the tests >> do >> anyway. > > Mainly it's a question of bootstrapping. Test::More is quite fancy > and unless all the Perl it needs is working, you are not testing > what you think you are testing. Requiring poetry before you know > how to spell is unfair. Try meditating on the word "core". But it should be okay for additional modules, like Math::BigInt. One day I need to switch to Test::More, since the "add debugging print statements" get's on my nerves. Cheers, Te"One day Math::Big* will have more tests than Perl;)"ls - -- perl -MDev::Bollocks -e'print Dev::Bollocks->rand(),"\n"' confidentially innovate market-driven customers http://bloodgate.com/perl My current Perl projects PGP key available on http://bloodgate.com/tels.asc or via email -BEGIN PGP SIGNATURE- Version: 2.6.3i Charset: latin1 iQEVAwUBPBvTNncLPEOTuEwVAQGW0wf7BafFa25z0gTcGQ89DRh140yevRcuxmFg 4x43kdy9NiBtal7k2RMrkkyglhTJCd6ytIihE0Bsg6/tufjm5nHRyHqT+Anx1L+9 0BX7tPzfcTJ68oPV24enczbtLv3hsjcCDipMPrz/gcE/Ua6I2Q/kitagxV44hh4p Tl8sDVHeuHNskkooOAPobo99URwjBdGckfiMn1Q9kSg8T7wuyUG8+bvdkshM0w5v tQMt8pwarzmHhR+dzrSOxIaEfhr28ldC/anWbhPsH9rzjsty7cjnIzLm/LRDLH8s moirvu4UFFj5aToqHqSiEwtQyKzYfbX3TIUQkeKz5yL/qKGibpoT1g== =TeV5 -END PGP SIGNATURE-
Re: Untested modules update: There's more than we thought
On Dec 14, Michael Schwern wrote:
> Constructors fail often enough that I felt it worthwhile to write a
> test function for them.
>
> You could leave off the test entirely and just trust that you use $foo
> later on down in the test and it'll explode there, but its better to
> put the point of failure as close to where the bug is as possible.
> And you get no diagnostic information (see below).
>
> The equivalent code without isa_ok() would be:
>
> my $foo = Foo->new;
> ok( $foo->isa('Foo') );
>
> except should $foo be unblessed or undef that will explode. You have
> to start doing things like:
>
> my $foo = Foo->new;
> ok( defined $foo && $foo->isa('Foo') );
>
> which rapidly gets tiresome.
Or ok(UNIVERSAL::isa(Foo->new, 'Foo'));
- Kurt
Re: Untested modules update: There's more than we thought
On Sat, 15 Dec 2001, Kurt D. Starsinic wrote:
> > my $foo = Foo->new;
> > ok( defined $foo && $foo->isa('Foo') );
> >
> > which rapidly gets tiresome.
>
> Or ok(UNIVERSAL::isa(Foo->new, 'Foo'));
Under 5.005 it spits out warnings if Foo->new returns undef. Plus you
still don't get any useful diagnostics. You just know it failed and then
you'll have to go back and add debugging print statements or use the
debugger to figure out _why_ it failed (did Foo->new return undef,
something blessed into a different class, an unblessed thing, etc).
I recently converted a rather large test suite (for Alzabo, 600+ unique
tests or so) to use Test::More. Doing so helped me do several things:
1. Break down what I was testing into smaller units. Where I used to
have things like:
ok( @foo && defined $foo[0] && $foo[0]->name eq 'bob' );
I now have:
is( @foo, 2, "expect two elements from bar() call" );
isa_ok( $foo[0], "Frobnigator" );
is( $foo[0]->name, 'bob', "the frobnigator should be named 'bob'" );
The presence of these similar but slightly different utility functions
helped me really think about exactly what I should be testing.
2. Diagnosing new failures. The diagnostics are such that I rarely have
to go back and add print statements or use the debugger. I know _why_
things are failing right away, which is better than simply knowing that
they failed for some reason or other.
3. Removed liable to break support code bits that were being repeated
everywhere.
So in the end I have more tests, which are easier to diagnose, with less
support code that I have to maintain. This is good because Alzabo is very
complicated as it is and the tests have enough weird life support to do
things like test against different RDBMS backends and such. Anything I
don't have to write makes me happier.
And if there's a problem with the testing code, I can report the bug to
Schwern rather than fixing it in my five test suites where I copied the
code around.
Ok, so that's a bit off the topic of "why use isa_ok()" but I just don't
see why people seem to object to the use of Test::More in the core Perl
tests. I can't see how it couldn't help improve the quality of the tests
while providing a standardized way to do the things that all the tests do
anyway.
-dave
/*==
www.urth.org
We await the New Sun
==*/
Re: Untested modules update: There's more than we thought
On Dec 15, Dave Rolsky wrote: > Ok, so that's a bit off the topic of "why use isa_ok()" but I just don't > see why people seem to object to the use of Test::More in the core Perl > tests. I can't see how it couldn't help improve the quality of the tests > while providing a standardized way to do the things that all the tests do > anyway. I, for one, don't object to the use of Test::More in the core Perl tests. I'm just being responsibly skeptical about the promotion of many different test harness modules with many different functions. It runs the risk of _seeming_ like the People in Charge don't know what testing is for, and of not providing effective guidance to those people writing their first tests. I also think that we have a real testing problem, which is that it isn't _trivially_ easy to run a huge test suite (like Alzabo's, or Perl's) and to find out at the end which tests failed and what their output was. 3000 lines and 20 minutes later, a message that 3 out of 1 tests failed is not maximally useful. Greater variety of test functions is not (IMNSHO) nearly as important as addressing that. And I only chimed in here to point out that one can use isa() on undef, which Schwern claimed couldn't be done. - Kurt P.S. Schwern, not to deter you from doing what's important to you. I know that patches are welcome. YMMV.
Re: Untested modules update: There's more than we thought
> Ok, so that's a bit off the topic of "why use isa_ok()" but I just don't > see why people seem to object to the use of Test::More in the core Perl > tests. I can't see how it couldn't help improve the quality of the tests > while providing a standardized way to do the things that all the tests do > anyway. Mainly it's a question of bootstrapping. Test::More is quite fancy and unless all the Perl it needs is working, you are not testing what you think you are testing. Requiring poetry before you know how to spell is unfair. Try meditating on the word "core". -- $jhi++; # http://www.iki.fi/jhi/ # There is this special biologist word we use for 'stable'. # It is 'dead'. -- Jack Cohen
[ANNOUNCE] Test::Harness 2.00 release candiate 2
The second [1] Test::Harness 2.00 release candidate just went up to CPAN http://www.pobox.com/~schwern/src/Test-Harness-2.00_04.tar.gz As Test::Harness is a rather critical module, and this release is a fairly major reshaping of the internal code, I'm going to do a round of release candidates so people can shake out the bugs. The first one that lasts a week in the field wins. Simply install the module, let me know if it went ok and your perl -V and go about your normal business. If something goes wrong, if you suspect Test::Harness screwed up a 'make test', if you get a warning, etc... post the bug to me and/or [EMAIL PROTECTED] Users of non-Unix, or non-mainstream Unix, I'd particularly like to hear from. Now would be a good time to fire up the CPAN shell and install a bunch of modules you've been meaning to upgrade. CPAN module authors may want to check their modules against this new harness. Those interested in testing will want to look at the new Test::Harness::Straps programatic interface to the harness. Its not complete, but at least there is a way to tell if a test passed or failed without eyeballing the output. I've decided that if I wait until the Test::Harness::Straps interface is complete I'll never release the damned thing. So I'm planting my flag and starting the fight from here. [1] The first one lasted about 5 minutes thanks to Gil. -- Michael G. Schwern <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/ Perl Quality Assurance <[EMAIL PROTECTED]> Kwalitee Is Job One Our business in life is not to succeed but to continue to fail in high spirits. -- Robert Louis Stevenson
Re: Untested modules update: There's more than we thought
Thinks like use_ok, require_ok, isa_ok, can_ok, etc... are all functions that I threw into Test::More because I found myself needing them and suspected others did, too. If you prefer to use ok() instead of one of the above special purpose functions, I'm not going to jump up and down and fling poo at you (as enjoyable as that is). They're nice to have, which reflects the "More" nature of Test::More. If they didn't exist, I could see worrying about their utility weight against the effort involved in implementing them. But some crazy test monkey already implemented them, so there they are. Ooop ooop. -- Michael G. Schwern <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/ Perl Quality Assurance <[EMAIL PROTECTED]> Kwalitee Is Job One my anus yearns now warm paste fills me happily saturday morning -- imploded
Test::Harness::Straps interface changes
Forgot to mention, there are a few user visable interface changes to Test::Harness::Straps in 2.00_03 (and thus _04) that have changed since 2.00_01. * 'summary' is now 'details'. It wasn't much of a summary. * Test #1 is now element 0 on the details array. It works out better that way I think. * analyze_file() is more portable, but no longer taint clean -- Michael G. Schwern <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/ Perl Quality Assurance <[EMAIL PROTECTED]> Kwalitee Is Job One I'm going to have to hurt you on principle.
'make test_harness' for more test info as it happens
Kurt mentioned he'd like more verbose information about how the tests are progressing as it happens, rather than having to wait until the end. In the perl core, the normal 'make test' doesn't give you much information at all because it has to use a neutered harness. I just remembered, you can run "make test_harness" instead of "make test" and it will run the core tests through Test::Harness. Unless you're expecting a truly catastrophic failure, this is safe and it provides more information both about the progress of the test and exactly which tests failed at the end. -- Michael G. Schwern <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/ Perl Quality Assurance <[EMAIL PROTECTED]> Kwalitee Is Job One conway: unit of mind expansion. One Conway == ~20 lines of Perl code found in $CPAN/authors/id/D/DC/DCONWAY, which gives the sensation of your brain being wrapped around a brick, with kiwi juice squeezed on top. -- Ziggy
[PATCH MANIFEST, lib/ExtUtils/Installed.t] Add tests for ExtUtils::Installed
This passes all tests, gets the MANIFEST patch right for once, and shouldn't
have weird sorting bugs on EBCDIC platforms. Cross your fingers.
-- c
--- ~MANIFEST Sat Dec 15 23:17:25 2001
+++ MANIFESTSat Dec 15 23:17:42 2001
@@ -906,6 +906,7 @@
lib/ExtUtils/inst Give information about installed extensions
lib/ExtUtils/Install.pmHandles 'make install' on extensions
lib/ExtUtils/Installed.pm Information on installed extensions
+lib/ExtUtils/Installed.t See if ExtUtils::Installed works
lib/ExtUtils/Liblist.pmLocates libraries
lib/ExtUtils/MakeMaker.pm Write Makefiles for extensions
lib/ExtUtils/Manifest.pm Utilities to write MANIFEST files
--- /dev/null Thu Aug 30 03:54:37 2001
+++ lib/ExtUtils/Installed.tSat Dec 15 23:18:22 2001
@@ -0,0 +1,204 @@
+#!./perl
+
+use strict;
+use warnings;
+
+# for _is_type() tests
+use Config;
+
+# for new() tests
+use Cwd;
+use File::Path;
+
+# for directories() tests
+use File::Basename;
+
+BEGIN {
+ chdir 't' if -d 't';
+ @INC = '../lib';
+}
+
+use Test::More tests => 43;
+
+use_ok( 'ExtUtils::Installed' );
+
+# saves having to qualify package name for class methods
+my $ei = bless( {}, 'ExtUtils::Installed' );
+
+# _is_prefix
+is( $ei->_is_prefix('foo/bar', 'foo'), 1,
+ '_is_prefix() should match valid path prefix' );
+is( $ei->_is_prefix('\foo\bar', '\bar'), 0,
+ '... should not match wrong prefix' );
+
+# _is_type
+is( $ei->_is_type(0, 'all'), 1, '_is_type() should be true for type of "all"' );
+
+foreach my $path (qw( installman1dir installman3dir )) {
+ my $file = $Config{$path} . '/foo';
+ is( $ei->_is_type($file, 'doc'), 1, "... should find doc file in $path" );
+ is( $ei->_is_type($file, 'prog'), 0, "... but not prog file in $path" );
+}
+
+is( $ei->_is_type($Config{prefix} . '/bar', 'prog'), 1,
+ "... should find prog file under $Config{prefix}" );
+is( $ei->_is_type('bar', 'doc'), 0,
+ '... should not find doc file outside path' );
+is( $ei->_is_type('bar', 'prog'), 0,
+ '... nor prog file outside path' );
+is( $ei->_is_type('whocares', 'someother'), 0, '... nor other type anywhere' );
+
+# _is_under
+ok( $ei->_is_under('foo'), '_is_under() should return true with no dirs' );
+
+my @under = qw( boo bar baz );
+is( $ei->_is_under('foo', @under), 0, '... should find no file not under dirs');
+is( $ei->_is_under('baz', @under), 1, '... should find file under dir' );
+
+# new
+my $realei = ExtUtils::Installed->new();
+
+isa_ok( $realei, 'ExtUtils::Installed' );
+isa_ok( $realei->{Perl}{packlist}, 'ExtUtils::Packlist' );
+is( $realei->{Perl}{version}, $Config{version},
+ 'new() should set Perl version from %Config' );
+
+my $wrotelist;
+if (mkpath('auto/FakeMod')) {
+ if (open(PACKLIST, '>', 'auto/FakeMod/.packlist')) {
+ print PACKLIST 'list';
+ close PACKLIST;
+ if (open(FAKEMOD, '>', 'auto/FakeMod/FakeMod.pm')) {
+ print FAKEMOD <<'FAKE';
+package FakeMod;
+use vars qw( $VERSION );
+$VERSION = '1.1.1';
+1;
+FAKE
+
+ close FAKEMOD;
+ $wrotelist = 1;
+ }
+ }
+}
+
+
+SKIP: {
+ skip( "could not write packlist: $!", 3 ) unless $wrotelist;
+
+ # avoid warning and death by localizing glob
+ local *ExtUtils::Installed::Config;
+ %ExtUtils::Installed::Config = (
+ archlib => cwd(),
+ sitearch=> cwd() . 'auto/FakeMod',
+ );
+
+ # necessary to fool new()
+ push @INC, cwd() . '/auto/FakeMod';
+
+ my $realei = ExtUtils::Installed->new();
+ ok( exists $realei->{FakeMod}, 'new() should find modules with .packlists');
+ isa_ok( $realei->{FakeMod}{packlist}, 'ExtUtils::Packlist' );
+ is( $realei->{FakeMod}{version}, '1.1.1',
+ '... should find version in modules' );
+}
+
+# modules
+$ei->{$_} = 1 for qw( abc def ghi );
+is( join(' ', $ei->modules()), 'abc def ghi',
+ 'modules() should return sorted keys' );
+
+# files
+$ei->{goodmod} = {
+ packlist => {
+ $Config{installman1dir} . '/foo' => 1,
+ $Config{installman3dir} . '/bar' => 1,
+ $Config{prefix} . '/foobar' => 1,
+ foobaz => 1,
+ },
+};
+
+eval { $ei->files('badmod') };
+like( $@, qr/badmod is not installed/,'files() should croak given bad modname');
+eval { $ei->files('goodmod', 'badtype' ) };
+like( $@, qr/type must be/,'files() should croak given bad type' );
+my @files = $ei->files('goodmod', 'doc', $Config{installman1dir});
+is( scalar @files, 1, '... should find doc file under given dir' );
+like( $files[0], qr/foo$/, '... checking file name' );
+@files = $ei->files('goodmod', 'doc');
+is( scalar @files, 2, '... should find all doc files with no dir' );
+@files = $ei->files('goodmod', 'prog', 'fake', 'fake2');
+is( scalar @files, 0, '.
[PATCH MANIFEST, ext/re.t] Basic tests for re.pm
Here are tests for everything but the 'debug' subpragma of the re pragma.
That'll take a bit more scary regex-fu than I have at the moment. Anyone who
is daring enough to change re.pm, re.xs, or re.c, though, now has no excuse for
improving the tests.
-- c
--- ~MANIFEST Sat Dec 15 23:53:50 2001
+++ MANIFESTSat Dec 15 23:54:17 2001
@@ -494,6 +494,7 @@
ext/re/hints/mpeix.pl Hints for re for named architecture
ext/re/Makefile.PL re extension makefile writer
ext/re/re.pm re extension Perl module
+ext/re/re.tsee if re pragma works
ext/re/re.xs re extension external subroutines
ext/Safe/safe1.t See if Safe works
ext/Safe/safe2.t See if Safe works
--- /dev/null Thu Aug 30 03:54:37 2001
+++ ext/re/re.t Sat Dec 15 23:53:39 2001
@@ -0,0 +1,64 @@
+#!./perl
+
+use strict;
+
+BEGIN {
+ chdir 't' if -d 't';
+ @INC = '../lib';
+}
+
+use Test::More 'no_plan';
+require_ok( 're' );
+
+# setcolor
+$INC{ 'Term/Cap.pm' } = 1;
+local $ENV{PERL_RE_TC};
+re::setcolor();
+is( $ENV{PERL_RE_COLORS}, "md\tme\tso\tse\tus\tue",
+ 'setcolor() should provide default colors' );
+$ENV{PERL_RE_TC} = 'su,n,ny';
+re::setcolor();
+is( $ENV{PERL_RE_COLORS}, "su\tn\tny", '... or use $ENV{PERL_RE_COLORS}' );
+
+# bits
+my $warn;
+local $SIG{__WARN__} = sub {
+ $warn = shift;
+};
+eval { re::bits(1) };
+like( $warn, qr/Useless use/, 'bits() should warn with no args' );
+
+delete $ENV{PERL_RE_COLORS};
+re::bits(0, 'debug');
+is( $ENV{PERL_RE_COLORS}, '',
+ "... should not set regex colors given 'debug'" );
+re::bits(0, 'debugcolor');
+isnt( $ENV{PERL_RE_COLORS}, '',
+ "... should set regex colors given 'debugcolor'" );
+re::bits(0, 'nosuchsubpragma');
+like( $warn, qr/Unknown "re" subpragma/,
+ '... should warn about unknown subpragma' );
+ok( re::bits(0, 'taint') & 0x0010, '... should set taint bits' );
+ok( re::bits(0, 'eval') & 0x0020, '... should set eval bits' );
+
+local $^H;
+
+# import
+re->import('taint', 'eval');
+ok( $^H & 0x0010, 'import should set taint bits in $^H when requested' );
+ok( $^H & 0x0020, 'import should set eval bits in $^H when requested' );
+
+re->unimport('taint');
+isnt( $^H & 0x0010, 1, 'unimport should clear bits in $^H when requested' );
+re->unimport('eval');
+isnt( $^H & 0x0020, 1, '... and again' );
+
+package Term::Cap;
+
+sub Tgetent {
+ bless({}, $_[0]);
+}
+
+sub Tputs {
+ return $_[1];
+}
Re: [PATCH MANIFEST, ext/re.t] Basic tests for re.pm
Chromatic wrote:
[snip]
> +re->unimport('taint');
> +isnt( $^H & 0x0010, 1, 'unimport should clear bits in $^H when requested' );
> +re->unimport('eval');
> +isnt( $^H & 0x0020, 1, '... and again' );
These tests are wrong. $^H & 0x0010 will never be one, no matter
what is in $^H. It will either be 0, or it will be 0x0010.
Just write them as:
ok( !($^H & 0x0010), 'unimport should clear bits in $^H when requested' );
and:
ok( !($^H & 0x0020), '... and again' );
--
Klein bottle for rent - inquire within.
Re: [PATCH MANIFEST, ext/re.t] Basic tests for re.pm
On Sun, 16 Dec 2001 00:43:07 -0700, Benjamin Goldberg wrote:
> chromatic wrote:
>> +re->unimport('taint');
>> +isnt( $^H & 0x0010, 1, 'unimport should clear bits in $^H when
>> requested' ); +re->unimport('eval');
>> +isnt( $^H & 0x0020, 1, '... and again' );
> These tests are wrong. $^H & 0x0010 will never be one, no matter what is
> in $^H. It will either be 0, or it will be 0x0010.
Oops. You're absolutely right, thank you. How does Schwern program when he
should be asleep? I can't.
I'll repatch tomorrow, to give other people a chance to find errors.
-- c
