Re: QA for Perl
On Wed, Jul 31, 2013 at 08:46:53PM +, Tracy Radel wrote: > Hi Dag, > > It is more an issue of documentation of how quality assurance is done for > Perl. Our procedures for QA require that when doing QA on a model or program > we have created (my ultimate goal), any software used by that program must > also have gone through a sufficiently rigorous QA process. For other > programs I have been able to find documentation on their QA program that > includes standard practices, problem reporting, code control, records > management, etc. They also have documentation describing the V&V tests > performed with tolerances and results outlined. You mention "code control". I don't think that anyone has mentioned the following yet (although you might already be aware of it) The Perl 5 source code now uses a git repository for version control: http://perl5.git.perl.org/perl.git Previously Perl 5 was using Perforce, from March 1997 until December 2008. The git repository contains a complete import of the Perforce repository (byte for byte perfect. I checked) but additionally attempts to reconstruct history prior to March 1997 from any available sources - mostly stable and development release tarballs. So we have over 25 years of history in version control, although the resolution of changes isn't that good for changes more than 16 years old. Nicholas Clark
Re: TAP::Harness uses lots of RAM to store test results
On Thu, Apr 11, 2013 at 10:24:35AM -0700, Buddy Burden wrote: > Nicholas, > > tl;dr: TAP::Harness uses lots of RAM to store test results > > Particularly lots and lots for millions of little "ok\n"s > > It would be nice if it didn't use lots of RAM when storing things that > > mostly > > pass. > > > > Yes, I ran into this before.(*) I was referred to a portion of the > Test::Builder POD(**) which says that perhaps it will be fixed one day. > :-D I guess "one day" hasn't come yet ... > > Perhaps some of these references will inspire someone with enough tuits to > work on a patch ... ? > > > -- Buddy > > (*) http://code.activestate.com/lists/perl-qa/12708/ > (**) https://metacpan.org/module/Test::Builder#MEMORY I don't think that I hit the same problem as you. It wasn't actually Test::Builder that was bailing out from inside a test script. It was some part of Test::Harness or TAP::Parser in the harness process that was running the scripts, and collating the results of all the TAP, that was failing for me. And yes, I'm rather hoping that someone will think that as this is an optimisation project, it's fun. The code is here: https://github.com/Perl-Toolchain-Gang/Test-Harness Nicholas Clark
TAP::Harness uses lots of RAM to store test results
tl;dr: TAP::Harness uses lots of RAM to store test results
Particularly lots and lots for millions of little "ok\n"s
It would be nice if it didn't use lots of RAM when storing things that mostly
pass.
So, the long(er) story. I use to be able to build and test blead on my
Raspberry Pi. Now it runs of memory (well, swap), somewhere near the end
of the tests, and fails to finish. So, I wondered why, and tried bisecting
the problem to see if it could be narrowed down to a particular change.
Running this:
export TEST_JOBS=1; bisect.pl -j1 --start
9f68b0f7acd1bcb04e7baa4bdb7cfec8e5d985c8 --end blead^ -Doptimize=-g
-Dusethreads -Duse64bitint -- sh -c 'cd t && ./perl harness; echo $?; ls -l
test_state'
*eventually* produced this:
HEAD is now at 3283393 Restrict the valid identifier syntax, fix some
identifier bugs.
bad - non-zero exit from sh -c cd t && ./perl harness; echo $?; ls -l test_state
32833930e32dc619abdaaab54e88de2a2765fb86 is the first bad commit
commit 32833930e32dc619abdaaab54e88de2a2765fb86
Author: Brian Fraser
Date: Tue Mar 5 18:18:49 2013 -0300
Restrict the valid identifier syntax, fix some identifier bugs.
Fixes:
* Length-one identifiers are now restricted to
[\p{XIDS}\p{POSIX_Punct}\p{POSIX_Digit}\p{POSIX_Cntrl}]
plus, if under 'no utf8', the 128 non-ASCII characters in the
Latin1 range.
* Identifiers that start with ASCII letters can be followed with
XIDC characters
(The committer made some small edits in the pod)
:100644 100644 e8f5402a77dd011f065f7d91db354fdfd6da6830
8ac08abf50107cbfba3a8296df34556449134ba6 M gv.c
:04 04 9ae3a108394119259ca10d38ef44ef23c237fb91
3a55dc5feeea97da7346fc74df0601d3200e32e3 M pod
:04 04 695729610bd917237944558cfe729d529dfb6514
714c16587ae882fc635c17fd82362e5ff804f16b M t
:100644 100644 27485462ffca6939c895961092a743775401fa14
1ea0da7fde5aa50521ffe68378a87980048578cd M toke.c
bisect run success
That took 274968 seconds
(yes, that's just over 3 days).
If I reverse just the test cases added by that commit:
git show 32833930e32dc619abdaaab54e88de2a2765fb86^:t/uni/variables.t
>t/uni/variables.t
Test Summary Report
---
uni/variables.t (Wstat: 0
Tests: 5 Failed: 0)
TODO passed: 5
Files=2388, Tests=620014, 18222 wallclock secs (2020.18 usr 55.35 sys +
13406.55 cusr 290.67 csys = 15772.75 CPU)
Result: PASS
whereas with that test restored:
../lib/charnames.t ok
Could not execute (./perl -I.. -MTestInit=U1 ../lib/diagnostics.t): open3: fork
failed: Cannot allocate memory at ../lib/TAP/Parser/Iterator/Process.pm line
168.
make: *** [test_harness] Error 12
If I double the amount of swap:
All tests successful.
Files=2388, Tests=685775, 18699 wallclock secs (2216.11 usr 60.91 sys +
14095.49 cusr 300.45 csys = 16672.96 CPU)
Result: PASS
So, yes, it's to do with the amount of memory available. That test itself
is innocent. All it does wrong is to add a *lot* of testcases:
-plan (tests => 5);
+plan (tests => 65850);
Which makes me wonder - just how much memory is TAP::Parser using.
In particular, is TAP::Parser using the same amount of memory to store 65850
"ok"s as it would to store some mix of 65850 "ok"s and "not ok"s?
Which I'm starting to think, for large test suites, isn't that efficient.
Most tests pass most of the time. So is it possible for TAP::Parser to use
a more efficient format in memory to "archive" results for tests where
every single subtest was an "ok"?
Nicholas Clark
Re: [QA2012] IRC Meeting 2011-01-12
On Fri, Jan 13, 2012 at 04:14:28PM +0100, Philippe Bruhat (BooK) wrote: > On Fri, Jan 13, 2012 at 09:54:10AM -0500, Matthew Horsfall (alh) wrote: > > On Thu, Jan 12, 2012 at 6:22 PM, Philippe Bruhat (BooK) < > > philippe.bru...@free.fr> wrote: > > > > > - as a side remark, it would be great if the companies that use perl > > > would give back some of the money they saved by not having to pay > > > for a Perl license especially for an event like this one, that is > > > targetted at improving perl & cpan > > > > > > > Agreed, I just wouldn't word it quite like that. > > Oops. That wasn't meant to be read by sponsors. ;-) Welcome to the joys of publicly archived mailing lists. I think that the Facebook generation have solved this by no longer assuming that anything is private. Meanwhile the unwilling stars of Wikileaks have discovered that whilst it was always the case that anything you write down might end up becoming public, computers just make this a lot easier on an industrial scale. Nicholas Clark
Re: TB2::Mouse will be internal use only... with one hitch.
On Tue, Nov 29, 2011 at 10:33:42PM -0800, Michael G Schwern wrote:
> On 2011.11.29 9:49 PM, David E. Wheeler wrote:
> > How much slower will this make running the core tests,
> > since neither Mouse nor Mouse::XS will every be available
> > there (during development, at core installation time).
>
> Doing some back of the envelope calculations, it looks like an extra minute
> and a half on my laptop. I don't know how long core tests take right now.
>
> Running...
>
> use Test::More tests => 1;
> pass("ok");
>
> 100 times via prove...
>
> With 0.98: 2.2s
> With 1.5: 10s
>
> An extra 8 seconds per 100 tests. There's about 1100 core tests which use
> Test::More. 8 seconds x 11 is about 90 seconds.
Divide by the number of CPU cores.
Assuming one can hog them all, what probably matters is the duration of
the IO::Compress tests.
My laptop takes about 20 minutes to Configure, build and test the core.
dromedary and various other multicore servers I have access to usually take
about 5 minutes. Upshot - it saves a lot of time to hack locally, then
rsync the code off to a different machine to actually run the complete tests.
> That's worst case. I can probably get compile time down at least 20ms which
> shaves off about 30 seconds.
The IO::Compress tests being such a reliable serial bottleneck that I can set
off a complete build and test run while writing this e-mail just to check
that I had the right distribution, and it finishes before me :-)
Nicholas Clark
Re: Dual life t/test.pl?
On Wed, Nov 16, 2011 at 11:15:21PM -0800, Michael G Schwern wrote: > On 2011.11.16 5:58 PM, Father Chrysostomos via RT wrote: > > I think these patches are safe, but given the recent thread about > > dual-lifing test.pl I'm not even sure whether I should apply it or not. > > (Nicholas Clark does seem to have some good points.) > > This patch is not affected by that discussion. [Agree. Will try to get back to that in a different message] > > I don't really want to get involved in the thread, but let me ask you > > one thing (and I'll let others [e.g., rjbs] respond to your response): > > Why not simply fork test.pl? That way you can strip out the > > miniperl-specific stuff and have an even more lightweight testing > > library. You could even pop it in a .pm file, add package foo and > > @EXPORT, and with little effort you have a shiny new Test::More::Lite > > module. > > 1) It's not really my goal to make it distributable as a CPAN module, >but as just something you copy. It continues to be my goal to reduce the amount of effort needed to support the core. Exposing more of the internals runs counter to this. > 2) Test::More would lose the benefit of improvements from p5p. > > 3) p5p would lose the benefit of improvements from Test::More. The core is using Test::More for pretty much every core-maintained test outside of t/ One of the things I've been working on slowly is trying to turn "pretty much" to "every", by removing each impediment in turn. p5p continues to benefit from Test::More, and certainly I'm grateful for your continued work on that. (I assume everyone else is) > 4) I already maintain enough test modules, thanks. [referred to below] On Tue, Nov 15, 2011 at 06:16:37PM -0800, Michael G Schwern wrote: > First thing to clarify, I don't really envision this being turned into a CPAN > module. That would be directly against its purpose. > > The primary purpose here is to separate and focus development and provide a > point for module authors to copy it into their project. But that's not the point of t/test.pl It's not meant as a lightweight general purpose testing system. It's intended to test miniperl and low level Perl features, without assuming that everything is actually working. That it turns out to be a lightweight test system is, well, convenient. Convenient for other people. > It probably will *not* be released to CPAN and it probably will *not* be API > stable. The needs of the core outweigh the needs of the few people that use > it. Totally agree about the "needs of the core". But not on the interpretation. The continued experience of "the core" is that a: for any part that anyone can access, someone starts to rely on it. b: because anyone can read the core's source, they can see non-accessible parts that would be useful. But these parts were *never* written with any consideration to whether they make a sane *public* API. Historically what has typically happened is that someone has sent in a patch to expose it, "that's cool" (as everyone thinks about how they too could use it) and it's applied, as a "nuclear reactor" in bikeshed terms. Which ends up with ever more parts of "the core" exposed, replied upon, and implicitly supported. Right now, t/test.pl is not installed by the core. So it's an internal implementation detail of testing. Anyone else who uses it copied it, and by act of copying it to install it they've taken an action that acknowledges that they're violating a form of public/private encapsulation. > On 2011.11.15 6:34 AM, Nicholas Clark wrote: > > Other than people writing test modules, who would find it useful? > > Test::More is about to become a lot more complicated internally, so anyone who > does things which Test::More might be using. > > In addition, .t files which run a bazillion tests would find it useful both > for speed and memory usage. I just recommended Buddy Burden use it for some > huge Data::Random tests. Abigail could use it for her enormous Regexp::Common > test suites. > > But that's not really what it's about. > > The library upon which 372 core tests rely is undocumented and untested. > Moving it into its own repo and tracker allows it to be tested and stable > without having to go through the p5p memorial bike shed. That's the entire bloody point. It's not *meant* to be assumed to be stable. It's an internal implementation detail of testing the core. It does what *we* need quite nicely. We're not looking for patches for it to make it into something else. The "p5p memorial bike shed" *is* a problem (both for bikesheds and nuclear reactors, and everythin
Re: Dual life t/test.pl?
This was suggested previously in 2008 http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2008-06/msg00912.html On Mon, Nov 14, 2011 at 04:59:57PM -0800, Michael G Schwern wrote: > Having a parallel and featureful testing system is very useful. I use it to > test Test::More (in the Test-Builder1.5 branch). Others might find it useful > to do the same. Other than people writing test modules, who would find it useful? > t/test.pl is very important to the Perl core tests, but it is largely > undocumented and untested. Going through the p5p process to patch it makes > working on it difficult and open to bike shedding. If it were dual-lived it > could be worked on as its own thing with its own commits and tests. test.pl is a script, not a package. Our machinery for dealing with dual life things is really only set up to deal with modules. So doing this is not going to just drop in to what we have. > By having a dedicated feed of issues and commits, careful attention could be > paid to what code and features are going in. Close attention and discussion > can happen about whether the code is simple enough and the features apt for > the needs of the core. Surely the needs of the core *are* better determined by the core developers, not some third party group? Also, having upstream be anywhere other than the core actively gets in the way of t/test.pl actually being useful to core developers (or at least to me, as a core developer who has added things to it). Even having it being upstream blead gets in the way of it being useful. Right now, as all users of the core's t/test.pl are in the core, it's possible to a: add things to it without stopping to think "is this a decent, supportable API that can last forever", because if the API turns out to be wrong, it's entirely possible to have a "flag day" commit that changes it and all its users b: remove things from it, because nothing is using them any more c: not worry about what versions of perl it supports, or working round problems in them (Although to be fair, if a particular piece of perl syntax had been buggy before, it's possibly something fragile that should be avoided in case it breaks again for a different reason) Nothing in the above stops anyone taking a copy of test.pl and using it. Doing this also doesn't add a layer of process into working on the core. > If folks are cool with that, I can set up a Github repository for it. We'll > worry about whether CPAN release is appropriate later. I'm distinctly uncool about doing it that way. I'm unconvinced about doing it with blead upstream. This feels like a lot of work for little benefit, with the work outweighing the benefit. We did start work on trying to extract runperl() as Test::Builder module. There's a branch from blead where I attempted to convert module tests to use it and found which features were necessary for them: http://perl5.git.perl.org/perl.git/shortlog/refs/heads/smoke-me/runperl Beware that IPC::Cmd can't capture output on Win32 with core-only modules, and IPC::Run can't reliably capture output on Win32, thanks to Microsoft's socketpair implementation following advice from Klortho: #11912 Evidently it's important to you to get the wrong answer as quickly as possible. (The default for closing a socket on Win32 is to drop all unsent buffered data on the floor. No, I'm not kidding. Note To assure that all data is sent and received on a connection, an application should call shutdown before calling closesocket (see Graceful shutdown, linger options, and socket closure for more information). Also note, an FD_CLOSE network event is not posted after closesocket is called. http://msdn.microsoft.com/en-us/library/ms737582%28v=vs.85%29.aspx I've also found http://msdn.microsoft.com/en-us/library/ms738547%28v=vs.85%29.aspx Whereas the BSD socket() implementation works in such a way that a socket FD can be substituted for a file FD, pipe FD or tty FD, and the program continues to work just as it did before, without needing any special case code. Doesn't even need recompiling.) Nicholas Clark
Re: Perl QA Hackathon 2012
On Fri, Nov 04, 2011 at 09:54:12AM -0700, David E. Wheeler wrote: > ? Cité des Sciences et de l'Industrie is a terrific place. I took my daughter > to Cité des enfants a couple of times when we were in France for six weeks in > 2008. It was easy to get to via the metro from central Paris, too. It's walkable from la Gare du Nord. It's not exactly outskirts either. (Google says 40 minutes. That's certainly within my scale of walkable, although likely I'd prefer other transport in one hit. But I remember that when I went to Les Journées Perl 2004, I was staying in a hotel roughly midway, so walked to it from Gare du Nord, and walked from the hotel to the venue) I like things that are convenient for la Gare du Nord. I'm biased :-) Nicholas Clark
Re: Devel::Cover 'time' html column
On Tue, May 25, 2010 at 09:32:44PM +0200, Paul Johnson wrote: > On Mon, May 24, 2010 at 06:52:15AM -0400, James E Keenan wrote: > > In any event, you needn't worry too much about that. It may be > > profiling information, but if that is the sort of information you are > > looking for you are better off with tools specifically designed for that > > purpose such as Devel::Pprof and Devel NYTprof. > > Certainly you should be using NYTprof for anything serious in this area, > but it didn't exist when I first wrote this code. Now it does, I'm > considering removing the time coverage. Does anyone have any thoughts > one way or the other about that? Devel::NYTProf does this so much better. I'd suggest removing it. Longer term, is Devel::Cover's runtime capture doing anything that Devel::NYTProf doesn't do these days? ie - is it possible to refactor Devel::Cover to provide the same coverage reports, but generate them from an nytprof.out file? If not, what's missing from the nytprof.out file? Would this reduce the maintenance load on Devel::Cover? Nicholas Clark
Re: What's up with "No targets specified and no makefile found"?
On Tue, Dec 22, 2009 at 07:03:49AM -0800, Joshua ben Jore wrote:
> Where is EU::MM's version control? I'd like to submit a patch adding a bunch
> of:
>
> or croak("Can't write '$fn': $!");
Good plan. (At least, in my opinion)
http://search.cpan.org/~mschwern/ExtUtils-MakeMaker-6.56/
says "Repository http://github.com/schwern/extutils-makemaker "
Nicholas Clark
Re: Making TODO Tests Fail
On Tue, Jul 14, 2009 at 03:33:29AM -0700, Ovid wrote: > I know. I've thought about that, but truth be told, I'm really getting burnt > out with the Perl community right now. Lots of people are being rude, > thinking that being "right" is all they need to justify being arrogant and > it's sapping my energy relating to anything regarding Perl right now. If > people can't play nice, I don't want to play. You're not the only one fed up with this, with this result. On Sunday, given the choice of hacking on something useful, or just sitting in the garden, I opted for sitting in the garden. I would have done again last night, except that the sun had already gone away. Pesky daystar. :-) Nicholas Clark
Re: Please test Module::Build release candidate
On Mon, Feb 23, 2009 at 07:10:09AM -0500, David Golden wrote:
> (Apologies to those receiving this multiple times)
>
> Eric Wilhelm and I have put out Module::Build 0.31_04 as a release
> candidate for a stable 0.32 Module::Build.
>
> Initial CPAN Testers results look good. (So far, it's the first "all
> PASS" since 0.2804)
>
> However, before release (and before it gets pushed into blead), it
Oh, too late. I did that with 66e531b6b23122113bed8498baac1ef25a958d57
It's blead. It's allowed to break things. :-)
I had to make this tweak to get xs.t and ppm.t to pass:
--- ../Module-Build-0.31_04/t/lib/MBTest.pm 2009-02-20 19:06:45.0
+
+++ lib/Module/Build/t/lib/MBTest.pm2009-02-23 14:05:36.0 +
@@ -77,7 +77,7 @@
# We pass everything through to Test::More
use vars qw($VERSION @ISA @EXPORT %EXPORT_TAGS $TODO);
-$VERSION = 0.01;
+$VERSION = 0.01_01;
@ISA = qw(Test::More); # Test::More isa Exporter
@EXPORT = @Test::More::EXPORT;
%EXPORT_TAGS = %Test::More::EXPORT_TAGS;
@@ -124,7 +124,7 @@
# Setup a temp directory
sub tmpdir {
return File::Temp::tempdir( 'MB-',
-CLEANUP => 1, DIR => File::Spec->tmpdir
+CLEANUP => 1, DIR => $ENV{PERL_CORE} ? Cwd::cwd : File::Spec->tmpdir
);
}
Otherwise ExtUtils::CBuilder::Base::perl_src() gets very upset because it
can't find the source.
blead currently has this difference from Module::Build
http://perl5.git.perl.org/perl.git/commit/86bddcbfaf2555223ec8fc596416d13d7a1a1118
which relates to https://rt.cpan.org/Public/Bug/Display.html?id=42724
Nicholas Clark
Re: use-case for isnt()
On Sat, Feb 14, 2009 at 12:57:45PM +0200, Gabor Szabo wrote: > On Thu, Feb 12, 2009 at 3:18 PM, Nicholas Clark wrote: > > I find > > > >isnt($foo, undef); > > > > useful as it gives better failure diagnostics than > > > >ok(defined $foo); > > > > Wouldn't it be better to write this > > is_defined($foo); > > Which, on failure would print > # got: undef > # expected: anything else > > as it does your example. > > Schwern, can this be added to Test::More ? Assuming that isnt() isn't going to be removed from Test::More, I don't see this as a good idea. Your proposed is_defined($foo) is the same amount of typing as isnt($foo, undef), gives no better diagnostics, is no clearer in its intent than isnt($foo, undef), and increases the complexity of the interface to Test::More by 1 function. It feels like: Although the Perl Slogan is There's More Than One Way to Do It, I hesitate to make 10 ways to do something. :-) http://groups.google.com/group/comp.lang.perl/msg/b7b1650e90b89c0b Nicholas Clark
Re: use-case for isnt()
On Thu, Feb 12, 2009 at 02:20:13PM +0200, Gabor Szabo wrote: > Hi, > > With the obvious risk of being laughed at, I'd like to ask > you guys for a good use-case of isnt() in Test::More. > > the one in the docs looks like this: > > # $foo isn't empty > isnt( $foo, '', "Got some foo" ); > > > which feels such a weak test I don't think I'd write it. I find isnt($foo, undef); useful as it gives better failure diagnostics than ok(defined $foo); Nicholas Clark
Re: Perl 6 and Test.pm's skip() function
On Fri, Jan 23, 2009 at 01:00:59AM +0100, Aristotle Pagaltzis wrote: > * Eric Wilhelm [2009-01-22 18:55]: > > Pretend for a moment that the number of tests could > > automatically be counted by the interpreter (e.g. at > > the parse/compile stage.) > > There???s no need to pretend. Either you can tell us how to solve > the halting problem and then it???s possible, or you can???t and then > it???s not. Someone had put it up on getacoder, but unfortunately the enthusiastic bids from at least one outsourcing collective has been removed, leaving only the original request: http://www.getacoder.com/projects/bug_finder_92913.html (I wonder if the owners of getacoder.com realised that the clearly demonstrated competency level of some of the folks bidding was not a good advert for the quality of results you might get back from using their site. But the incoming link karma was worth keeping, hence the original bid remains. Buyer beware. You *are* getting what you paid for.) Nicholas Clark
malformed TAP?
Is this TAP valid? ok 1 - killed the keyd 1..1 ok 2 - Child exit of 3 was expected prove (v3.14) thinks so: All tests successful. Files=1, Tests=2, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.04 cusr 0.00 csys = 0.07 CPU) Result: PASS I'm not convinced. Nicholas Clark
Re: How to write a test for Test::Builder?
On Fri, Sep 19, 2008 at 03:35:36PM -0500, brian d foy wrote:
> In article <[EMAIL PROTECTED]>, Nicholas Clark
> <[EMAIL PROTECTED]> wrote:
>
> > But it seems that this bug is only fixed as a side effect of that change,
> > and
> > it's not actually tested for. What's the best way to write a test that fits
> > within the current frameworks to prevent any regression?
> > It's not obvious to me how to make use_ok() test for failure.
>
> Are you looking for Test::Builder::Tester?
Not sure.
>#!perl
>
>use Test::More tests=>1;
>use Test::Builder::Tester;
>
>test_out('not ok 1 - use Fcntl;');
>test_fail(+1);
>use_ok 'Fcntl', 'Pie';
that needs to be in a BEGIN block
>test_test( "Fails for bad export");
>__END__
Yes, but I then need it to do that with perl invoked with -c, and
$ENV{HARNESS_ACTIVE}=1
I think that this might be a job for system()...
Nicholas Clark
How to write a test for Test::Builder?
With Test::Builder 0.80
$ perl -c ~/test/bang.t
/home/netbanx/test/bang.t syntax OK
$ cat ~/test/bang.t
#!perl
use Test::More tests=>1;
BEGIN {use_ok 'Fcntl', 'Pie'};
__END__
$ perl -c ~/test/bang.t
/home/netbanx/test/bang.t syntax OK
$ HARNESS_ACTIVE=1 perl -c ~/test/bang.t
Can't use an undefined value as a symbol reference at
/usr/local/share/perl/5.8.8/Test/Builder.pm line 1275.
BEGIN failed--compilation aborted at /home/netbanx/test/bang.t line 5
I see that this is fixed in svn by r691:
http://code.google.com/p/test-more/source/diff?spec=svn691&r=691&format=side&path=/trunk/lib/Test/Builder.pm
Added note() and explain().
explain() does not print to allow you to choose if you want the results to
be a note or a diag.
But it seems that this bug is only fixed as a side effect of that change, and
it's not actually tested for. What's the best way to write a test that fits
within the current frameworks to prevent any regression?
It's not obvious to me how to make use_ok() test for failure.
Nicholas Clark
Re: PerlUnit
On Tue, Sep 16, 2008 at 04:23:16AM -0700, Eric Wilhelm wrote: > # from Josh Heumann > # on Tuesday 16 September 2008 02:37: > > >Shouldn't the people who post more-or-less agreeing solutions vote > > each other's solution up? > > Yes. Now try to do it ;-) > > You need a +15 modifier of nothingbettertodoism to vote. So everyone gainfully employed because they know what they are doing is automatically disqualified? Nicholas Clark
Re: Can't locate version/vpp.pm in @INC
On Thu, Sep 11, 2008 at 12:34:18PM -0700, Randy J. Ray wrote: > On Thu, Sep 11, 2008 at 4:56 AM, Gabor Szabo <[EMAIL PROTECTED]> wrote: > > > Can someone tell me what causes this failure? > > > > http://www.nntp.perl.org/group/perl.cpan.testers/2008/09/msg2187300.html > > > > the latest Module::Inspector (1.05) is installed > > http://search.cpan.org/dist/Module-Inspector/ > > and that module has not test failure at all. > > > > I ran into this a week or so ago... this is what I *think* happened: > * I built/updated version using the system-installed Perl > * I tried to use it from a second perl interpreter on the same system > * Second perl was no binary compatible (in this case, non-threaded vs. > threaded) > * As such, version.pm loaded but could not load the XS portion But that shouldn't happen, because modules containing XS should be installed to an architecture specific path, and the architecture name used differs for threaded versus unthreaded. Nicholas Clark
Re: Sub::Uplevel vs Test::More
On Wed, Sep 10, 2008 at 04:01:56PM +0200, Aristotle Pagaltzis wrote: > * Paul Johnson <[EMAIL PROTECTED]> [2008-09-10 13:50]: > > Oh, but they *could* have them. > > > > And I think that is a perfect solution. CPANTS should check > > whether modules have a shebang line, and if so whether it > > contains -w. If it does then the author has asserted that the > > module runs cleanly with warnings enabled and should receive > > the kwalitee point. > > Problem is, the shebang line doesn???t actually *do* anything. > > The correct solution (which also doesn???t require changes to > CPANTS) is called warnings::compat. Assuming you actually care > that much??? I don't agree. I'm with Paul. Why should I add a dependency to correct code to placate the CPANTS game? Particularly as warnings::compat *can't* do everything the warnings pragma can do, so by changing to use it I might be adding bugs. Nicholas Clark
Re: Sub::Uplevel vs Test::More
On Tue, Sep 09, 2008 at 10:12:24PM -0400, David Golden wrote: > On Tue, Sep 9, 2008 at 8:51 PM, Michael G Schwern <[EMAIL PROTECTED]> wrote: > Heh. You planted that bomb, dude. Any excuse I have to bump things > to 5.006 I've been taking and saying "because Schwern says so". > > (CPAN::Reporter and deps are probably the exception and only because > Slaven has sent me so many patches that I feel I owe it to him to > support his Quixotic mission to smoke 5.005.) > > Certainly, I'd have a higher CPANTS score if I could start adding "use > warnings" to my code. ;-) Wouldn't it be simpler to send a patch to domm that makes the warnings point credited if the module states a requirement on perl pre 5.006, and has a -w on the #! line? Nicholas Clark
passing the baton onwards (was Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it))
On Thu, Sep 04, 2008 at 02:19:26PM -0400, Greg Sabino Mullane wrote: > I recognize that CPAN is a volunteer effort, but it does seem to me there > is a implicit responsibility on the part of the author to maintain the > module going forward, or to pass the baton to someone else. Call it a Best Is there an easy central visible way to flag up a module as "up for adoption"? What should have been the right list to ask that question on? > Practice, if you will. The end-user simply wants the module to work. > Maintainers not paying attention, and the subsequent bitrot that is > appearing on CPAN, is one of Perl's biggest problems at the moment. > Careful attention and responsiveness to CPAN testers and to rt.cpan.org is > the best cure for this. Alternatively, I could just not upload code to CPAN, and not have this problem. You're right that it's a problem. I'm not convinced that your cure will work with all real world volunteers. Nicholas Clark
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Thu, Sep 04, 2008 at 09:50:01AM -0700, chromatic wrote: > On Thursday 04 September 2008 01:19:44 Eric Wilhelm wrote: > > > Let's pretend that I'm a real jerk of an author and I only care about > > whether my code installs on a perl 5.8.8+ (a *real* perl -- no funky > > vendor patches) with a fully updated and properly configured toolchain > > and the clock set to within 0.65s of NTP time. > > Oh great, yet another check I have to add to my Build.PL. What's the magic > cantrip for THAT one? > > (Why yes, I *have* seen bugs related to time skew on network-mounted paths), So have I. I think that there's at least one stat test in the core that will fail if you're testing on an NFS mount from a machine where the clock differs. IIRC it's because file creation time is stamped by the server, and file modification time is stamped with a time from the client, and if they're out the "impossible" can happen. Which when you're sanity testing the ops that read these sorts of things, you're looking out for. Although I suspect that there can be more general problems with make getting (correctly) confused by timestamps on files it touched. Nicholas Clark
Re: App::Prove::State Users?
On Fri, Aug 22, 2008 at 09:18:36AM -0700, Ovid wrote: > Does anyone use App::Prove::State, a module which ships with the new > Test::Harness? Outside of the Test::Harness distribution, the only code which > I know of which uses it is a quick "which tests are slow?" hack I put > together. > > I ask this because I'm making some internal changes to this module. I am > giving it a proper API to let people extend it in the future. I need to add > new features for it, but said features can't live in the core (due to > non-core dependencies) and as a result, I need to nail down the interface, > but if anyone else is using it, I'd like to not cause them too much pain. I had a play with using it in (of all places) the core, to sort a subset of the tests to run in slowness order, in t/harness You can see the (annotated) code here: http://public.activestate.com/cgi-bin/perlbrowse?filename=t%2Fharness&show_blame=Show+Annotated+File If you want to break it, it doesn't matter. I'll take it out again. Nicholas Clark
Re: [RFC] Dual-lifing test.pl
On Tue, Jul 01, 2008 at 09:31:22AM -0400, Jerry D. Hedden wrote: > If the functionality in test.pl (that does not currently > exist in other module) could be duplicated elsewhere using a > Test::Builder-based module, would there be a reason then to > maintain test.pl? Would it be better then to eliminate > test.pl entirely? No, it should stay. It intentionally doesn't use packages internally, and (at least) some other constructions. (specifically ++. Although a quick skim suggests that some use of -> for method calls has slipped in, with File::Spec and Config, in which_perl(), fresh_perl* and the isa/can tests) Nicholas Clark
Re: [RFC] Dual-lifing test.pl
On Mon, Jun 30, 2008 at 12:54:29PM -0400, Jerry D. Hedden wrote: > Jerry D. Hedden wrote: > > From what I can tell, there are well over a dozen dual-lived > > modules that incorporate test.pl into their distributions. > > Rather than continuing this trend, I would like to propose > > turning it into a module and dual-lifing it to CPAN. > > Nicholas Clark wrote: > > Why are they using it, rather than something built from > > Test::Builder? What does it offer that Test::Builder > > derived modules don't? > > Jerry D. Hedden wrote: > > In my case of dual-lifing threads, threads::shared, > > Thread::Queue and Thread::Semaphore, it was because the > > tests were already written to use test.pl. Thus, adding it > > to the distribution was easier than rewriting all the tests. But, logically, those tests in the modules in core can be re-written to not use it [if, see below] > > I would imagine that is also the case for the other > > dual-lived modules that use test.pl, but their maintainers > > could have other reasons. Anybody? > > Additionally, test.pl has certain functionality (e.g., > fresh_perl() and watchdog()) not found in the Test::* > modules. hence I think that a better solution would be to add just that code to a Test::Builder based module, either a new one, or an existing one if its maintainer is keen. The core's test.pl is meant to be a subset of the Test::More functionality for core regression tests in t/* (distinct from tests in lib/* for modules shipped with core). To me it feels like scope creep if it also ends up dual- lived. It also means that we would have *two* core-supported test frameworks, which can't be used in the same test. Nicholas Clark
Re: About tidying up Kwalitee metrics
On Tue, Jun 24, 2008 at 10:10:07AM +0200, Salve J Nilsen wrote: > Hello, folks > > I propose to split the current "main" and "optional" kwalitee scales into > topical ones, so we can allow for richer set of metrics while allowing > everyone that care mostly about certain types of metric access to > "untainted" versions. > > Let's remove the "optional" type, and instead create the following metrics > where we can place the existing tests: > > Disto Kwalitee > (most of the original test should go here) > Security Kwalitee > (checks for taint-mode or other security-related issues go here) > Community Support Kwalitee > (checks for supplied mailing list address, bugtracker, archives, etc. go > here) > Community Trust Kwalitee > (analysis of external acceptance of the module, including Debian use go > here) > > Thoughts? Certainly, I would like the metrics to be split into those I can control by what I upload to PAUSE, and those that I can't "fix" however much I upload. Which I think most obviously is those that you group here as "Community Trust Kwalitee". The previous 2 seem good, as they are likely to be categories that some people have legitimate disagreements with. ie I've not been paying close attention to CPANTS, but if I did, I suspect that it would annoy me that it expects me to have a POD coverage test, and that in turn to make it pass I could well spend more time bodging that than actually writing documentation. Which, I agree with chromatic, would be stupid, and not something that I'd like to see promoted. (Is "You have POD and it's well formed" is something that is already tested?) Nicholas Clark
Re: TODO Tests
On Mon, May 12, 2008 at 08:41:58PM +0200, Bram wrote: > I'm not saying that it shouldn't change. > What I'm saying is that the value may change but that if it changes I > would like to be informed about it. > > > >Leave your todo test as it was to start with. > > > >Create a new test file "development_values_that_shoulnot_change.t" in your > >developer test directory (that's not just for pod right). > > Which is not practical at all. > That means maintaining an extra test file for each file that defines a > todo test and that means duplicating every todo test. My view too. It adds a lot of complexity. (but) > What I'm suggesting is that it outputs something similar for TODO test > that return an unexpected result. > > Again, make test only outputs All test succesful. > Users will only run make test and thus can't be confused by the extra > output. I think that this problem could be solved without chances to TAP. Specifically, one is trying to make a way of specifying two alternate "expected" results for a given test. One is the right one. If this result appears, then TODO test unexpectedly succeeded. (Which is what we have currently) One is the current wrong one. If this result appears, then the current test failed, but it's TODO. (Which, at the TAP level, is what we have currently) The third case is "anything else". Right now, it's also "not ok (TODO)" and the TAP parser assumes that it's not noteworthy. What might be needed is a coupled way of passing *two* tests into the test running framework (Test::Builder or the like). One is the "desired" result. One is the "current expected fail" result. Right now the only thing we test is the "desired" result. The framework then spits out the results from both tests (run the desired result first) as current expected result oknot ok desiredok "can't happen""not ok TODO, not ok TODO" result not ok "not ok TODO, "not ok ok TODO" not ok TODO" "can't happen" is also reported as "not ok" - it's a critical logic error in the tests. (ie top left and bottom right are fails, bottom left and top right are the two expected cases, and so both pass, although bottom left is "unexpected TODO pass") Could this be done with a Test::Builder (subclass) function that takes two subroutine references - one that tests for the "desired" result (and returns it as the return value), and one that tests for the "current expected result" (and likewise returns it as the return value). With suitable coding it would be possible in the actual test to call it quite tersely, with minimal code duplication, possibly using closures. One will still get all the diagnostics from running the test twice. It would just be that the second test, the one we currently don't have a way to write, would loose its "TODO" and become a hard fail if things change in the wrong direction. Nicholas Clark
Re: OT: muttrc (was: User Supplied YAML Diagnostic Keys: Descriptive Version)
On Mon, Apr 14, 2008 at 01:54:02PM +0200, Aristotle Pagaltzis wrote: > * Nicholas Clark <[EMAIL PROTECTED]> [2008-04-14 08:00]: > > $ grep -c 'ignore X' ~/.muttrc > > 100 > > > > That's the ones I've collected that I don't care about. And > > some of those are common prefixes. > > You know that you can use wildcards to ignore everything (or just > big swathes of stuff) by default and then selectively unignore > some of non-gunk, right? No, I didn't, but I'm not looking to do that. I'm curious what the gunk is, without having to remember to look at every message. This way, the new gunk announces itself to me, without me having to go look for it. I think that this has some parallel to testing, but I'm not sure what. I guess it's like testing that no warnings are issued by code, rather than testing that a particular warning (that signals known trouble) is not issued. Nicholas Clark
Re: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version
On Sun, Apr 13, 2008 at 03:58:58PM -0700, chromatic wrote: > and you'll end up with the protocol equivalent of spaghetti. Anyone care to > guess how many X-* headers there are in all of the SMTP clients and servers > in the wild? How about HTTP headers? Maybe you don't have to care about $ grep -c 'ignore X' ~/.muttrc 100 That's the ones I've collected that I don't care about. And some of those are common prefixes. I guess that there are bazillions more. "There's one born every minute." Nicholas Clark
Re: TAP YAML Diagnostics
On Sun, Apr 06, 2008 at 08:41:11AM -0700, Ovid wrote: > Currently you can shove anything in there you want, but you must use > upper-case keys for your personal use and all lower-case keys are > reserved (and it's a parse error to use an unknown lower-case key). > Are there any strange Unicode issues where we might get confused about > what is upper and lower case?) I believe that there are code points which would be considered word characters but do not have distinct upper and lower case forms (or by implication title case either), but I hope that the good folks of perl-unicode will correct me if I'm wrong. Hence I'm not sure what the most efficient way of determining if something is all lower case is. If I'm right, one can't just test if ($string eq lc $string) because these code points would mess you up, and I *assume* that they are not those which you want to consider reserved. I guess that one needs to loop over all characters in the string, and verify that if $char eq lc $char then also $char ne uc $char. (But one could first short circuit the common pass case with the test above) Nicholas Clark
Re: My Perl QA Hackathon Wishlist
On Wed, Mar 26, 2008 at 10:41:18AM +0200, Gabor Szabo wrote: > > -Original Message- > > From: Michael G Schwern [mailto:[EMAIL PROTECTED] > > The downside is there's no pretty GUIs for TAP, but the potential exists. > > It's a simple matter of programing. The upside is that when a TAP GUI is > > created it will work with all existing TAP tests. > > Which might be a good project to work on in Oslo... It might also make a good Google Summer of Code project. It ticks lots of boxes: * It's self contained * It's decoupled from other projects * It doesn't matter the the main project if it fails * It can expand (or contract) to fit the student's abilities and whatever "end" is reached it can still be declared a success * It doesn't have a high barrier of entry due to needing knowledge of existing code bases. Now, does anyone know a student? Nicholas Clark
Re: CPANTesters considered harmful
On Mon, Mar 03, 2008 at 05:21:47PM +, Smylers wrote: > $ man chattr > No manual entry for chattr > > $ man lsattr > No manual entry for lsattr > > That's on a FreeBSD server. Let's try Linux. Ah, yes: > > $ whatis chattr > chattr (1) - change file attributes on a Linux second > extended file system > > So you can make this test work even when running as root, so long as you > don't mind it now failing on Unix-like OSes that aren't Linux. I'm not > sure that's a nett win! Well, it depends on whether the file system supports chattr... :-) The FreeBSD equivalent: $ cd /tmp/ $ date >Pie $ chflags uchg Pie $ date >Pie bash: Pie: Operation not permitted $ cd ~/test $ date >Pie $ chflags uchg Pie chflags: Pie: Operation not supported $ df . Filesystem 1K-blocks UsedAvail Capacity Mounted on nas:/data 258199522 203419366 3412419686%/filer Nicholas Clark
Re: CPANTesters considered harmful
On Mon, Mar 03, 2008 at 02:19:23PM +, Smylers wrote: > demerphq writes: > > > It turned out the problem is that when the tests are root it seems to > > be not possible to create a directory that is not writeable by root. > > I think that can be reduced to: It isn't possible to create a directory > that is not writeable by root. The whole point of root is that as the > super-user it can do anything! > > > Im not really sure how to tackle this better than simply skipping the > > tests as root which is what the most recent release does. > > That's plausible. It could also temporarily drop privileges to be some > other user for running that test, but I don't know how you'd work out > which user to do it as. My guess would be "nobody" if that user exists, else give up. But I agree that skipping is better, because the tests run as non-root already prove that the module's functionality worked. Adding a lot of complex logic to the test to swap user when running as root would actually make the test as much a test of the user ID swapping code, and introduce code that isn't usually tested, and generally introduce fragility and cause false positive failures. Nicholas Clark
Re: Is there even a C compiler?
On Mon, Feb 25, 2008 at 03:59:14PM +, Andy Armstrong wrote:
> Is there a generally approved way for an XS module to test for the
> existence of a C compiler before attempting to build?
Personally I'd like such a test to be standard in MakeMaker, because I am
of the (old fashioned) opinion that if %Config says that there is a C
compiler and there isn't one, then it's lying (and the instalation is broken,
for a small value of broken)
Oh, and I don't want wheels re-invented.
> I guess I could just try running whatever $Config{cc} suggests. Are
> there any edge cases that that misses? Does $(CC) -o foo.o foo.c work
> just about everywhere?
Not sure. Certainly don't test that there is a file of that name in $PATH,
as it's acceptable for it to be a series of shell commands.
Also, you're already making assumptions about object file extension.
You can get that from $Config:
$ perl -V:obj_ext
obj_ext='.o';
Nicholas Clark
Re: Wide character support for Test::More
On Sat, Feb 23, 2008 at 08:35:15PM -0800, Michael G Schwern wrote:
> I just merged together a number of tickets having to do with Test::More not
> liking wide characters.
> Wide character in print at lib/Test/Builder.pm line 1252.
> ok 1 - Testing ??
>
>
> I know almost nothing about Unicode. How do I make this Just Work? Is it
You can't. It might be that the user's terminal simply cannot display that
character.
One could get round it by escaping (in some way) all characters that are non-
ASCII, for example with \x{} encoding, and also escaping \ (at least) so that
all the diagnostic output would be valid for a "" string.
> safe to just set binmode to always be ':utf8' if perl > 5.8?
Yes, safe from a Perl point of view. But not correct if the user's terminal
isn't expecting UTF-8. And wrong if the user's terminal is expecting the
same (8 bit) encoding as the script already is in.
However, it might be safe enough to invoke the testing Perl with -CLS
(set STDIN/STDOUT/STDERR to UTF-8 if the user's locale has UTF-8 in it)
which is probably going to be more right more often than anything else.
Not sure if -CLS only came in with 5.8.1
Nicholas Clark
Re: Change 33313 causing failures
On Fri, Feb 15, 2008 at 10:50:51AM +0100, Rafael Garcia-Suarez wrote:
> On 14/02/2008, Jerry D. Hedden <[EMAIL PROTECTED]> wrote:
> > Change 33313 is causing test suite failures:
> >
> >
> > t/op/pat..FAILED--unexpected
> > output at test 1308
> >
> > t/op/pat_thr..FAILED--unexpected
> > output at test 1308
>
> That happens with TEST, not with harness, which explains why I didn't saw it
> :(
>
> > Capturing the relevant test output:
> >
> > ok 1307 - $+{C} Line 3788
> > not ok 1308 - $+{D} Line 3788 # TODO 1
> > not # Failed test at line 3788
> > # expected: '???xfoo'
> > # result: ''
> > ok 1309 - 18446744073709551615~=/(?(?foo)s+(?bar)?s+(?baz))/
> > Line
> > 3793
> >
> > The 'failure' is the extra 'not' before the pound sign.
> > I tried to figure out what's causing it, but couldn't.
Is it possible to put the TAP parser into a strict mode where it would detect
and fault these sorts of things?
(I think most specifically "these things" are lines that aren't /^ok/,
/^not ok/ or /^#/ )
Nicholas Clark
Re: Testing print failures
On Mon, Jan 07, 2008 at 03:59:18PM +, Adrian Howard wrote: > Actually, I've had some nasty bugs in the past with prints failing > with dodgy NFS mounts. Caused some important data to go away. Not my > code fortunately :-) In these circumstances print failed but the close on the file did not? I know that it happens the other way, because I clearly remember that a certain firm's compiler got caught with its pants down when a certain firm's NFS implementation went multithreaded in release 2.4 of the OS. With 2.4 write failures (here caused by hitting quotas) were not reported as errors on the writes, but instead only spotted at close time (when everything resynchronised). The compiler wasn't checking the return value of close. Then again, close time might be too late for the application you're describing, if it is holding files open for long periods. Nicholas Clark
Re: Testing print failures
On Mon, Jan 07, 2008 at 04:19:17PM +, David Cantrell wrote: > On Sat, Jan 05, 2008 at 10:24:47PM +0000, Nicholas Clark wrote: > > > Not tested, but, can you > > > > 1: grab the address of print's op from PL_ppaddr > > 2: store it somewhere useful > > 3: replace it in PL_ppaddr with your own function > > > > Your own function calls the original, and then before returning, checks > > the return value on the stack. If it indicates fatal, then check the calling > > context. If that's void, croak. > > > > Otherwise return normally. > > Please for the love of god don't do that. Global changes to perl's But I can for the love of Mammon? :-) [ http://www.miltonbrewery.co.uk/beers/mammon.html Not that I had any last night, because Christmas was still in bottled form: http://www.individualpubs.co.uk/pembury/drinks.html ] > behaviour propagating out of some obscure module several steps away in > your application's dependency tree are a Bad Idea. See UNIVERSAL::* for > examples of why. Good point. I think that it's already been suggested that it can be done lexically in 5.10 (and later) by inspecting $^H, and only croaking if it's void context and and a user defined lexical pragma has enabled it. Nicholas Clark
Re: Testing print failures
On Sat, Jan 05, 2008 at 11:00:41PM +0100, nadim khemir wrote: > day an the answer was 'no'. Do you happend to know something about > replacing 'print' with XS code short of patching perl (which doesn't sound > like a good idea) Not tested, but, can you 1: grab the address of print's op from PL_ppaddr 2: store it somewhere useful 3: replace it in PL_ppaddr with your own function Your own function calls the original, and then before returning, checks the return value on the stack. If it indicates fatal, then check the calling context. If that's void, croak. Otherwise return normally. Nicholas Clark
Re: Auto: Your message 'FAIL IO-AIO-2.51 i386-freebsd-thread-multi 6.2-release' has NOT been received
On Wed, Dec 19, 2007 at 12:38:46PM +, Andy Armstrong wrote: > On 19 Dec 2007, at 12:32, Michael G Schwern wrote: > >>Obviously that's my (probably extremely unprofessional) impression of > >>his views. He did mention religion and ego though :) > > > >CPAN Testers does mug his modules pretty badly, just look at all > >that red. > >http://cpantesters.perl.org/author/MLEHMANN.html > > > >He does an awful lot of XS which is always going to be problematic. > > > From what he's said to me off-list I'd guess that if he used > Devel::Check* many of those failures would go away. > > I'm trying to persuade him to let me archive our exchange somewhere so > people can read it if they're interested. It would be interesting. He's a smart guy and usually right. So are you. Without seeing it, it makes me think of the exchanges between IlyaZ and TomC on p5p. Both were right, but both contradicted each other. The cause was that each based their reasoning on a different set of assumptions, and what one thought important the other thought was not, and vice versa. Nicholas Clark
Re: Impotent disclaimers (was: Milton Keynes PM coding collaboration)
On Fri, Dec 14, 2007 at 09:09:12AM +0100, A. Pagaltzis wrote: > Hi Tony, > > * Edwardson, Tony <[EMAIL PROTECTED]> [2007-12-14 08:55]: > > This is a commercial communication from Commerzbank AG. > > > > [20 lines snipped] > is there any chance you could subscribe this list using an > address whose outgoing mail server does not feel the need to > append 25 lines of legally ineffectual lawyeristic mumbojumbo > to a one-line mail? Or encourage Morgan Stanley to take Commerzbank AG over, as the Morgan Stanley disclaimer manages to say more in only 2 lines. [some lawyers are more efficient than others. I wonder if lawyers can be upgraded] Nicholas Clark
Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?
On Tue, Dec 04, 2007 at 08:25:10PM -0800, Michael G Schwern wrote: > Fergal Daly wrote: > > You have no idea what version of Foo they're using > > Well, you do with version dependency declarations so you control the range. > New versions are, of course, open to breakage but at some point you have to > trust something. I'm also missing why no-one has suggested local $TODO = "Frobulator low on pie" if Foo->VERSION < 3.14; which ought to mean that TODO tests stop being TODO if the user has a new enough version of Foo installed. (ie, you *do* know what version of Foo they're using, at least in the general case) Nicholas Clark
Re: New proposed CPANTS metric: prereq_matches_use
On Mon, Nov 19, 2007 at 03:03:51PM -0800, chromatic wrote: > On Monday 19 November 2007 14:30:51 Jonathan Rockway wrote: > > > I think the real solution, though, is to agree that the perl interpreter > > without all of the core modules installed isn't Perl. (I'm not a big > > fan of core modules, but the concept is especially worthless if you > > can't depend on their existence.) To your own modules? If so, why on earth is anyone upset about this? > Any chance you could convince a few thousand ISPs of that? I'm game. And this is why. Nicholas Clark
Re: prove -j 9
On Mon, Sep 10, 2007 at 11:37:00AM -0700, Eric Wilhelm wrote: > # from demerphq > # on Monday 10 September 2007 10:51 am: > > >> As to the core tests, they trip up like crazy, I think because they > >> are not original in their choice of names for temporary files. > > I solved that in dotReader's tests with some '. $$' action. > > I was playing with the 't/' directory from perl-5.8.8 last night and > having very little luck. For starters, I had to hack t/TEST to print > the filenames, but still came up with 995 files instead of 933, so I > think I did it wrong ;-) In the core use the files as listed in MANIFEST > I currently have no partitioning for what runs in parallel or "groups of > series" (ala `make -j`.) I supposed grouping by directories could be > an easy switch-activated feature. In general though, I'm thinking this > ventures into the realm of "test metadata", so I hesitate to ad-hoc too > much into it. In the core, I found that ensuring that all the test files from the same directory in order worked nicely. I'm sure that some tests can be run in arbitrary order (or literally in parallel) but there are enough of them to ensure that lots of CPUs can get fed. The problems I had were with trying to get t/op/*.t in parallel (which should work) and t/class/*.t (which are slow) Also, I was trying to feed 32 cores, not 2. I think that I documented this somewhere, but once you get to 7 tests in parallel, where you pass the tap stream back to a central parser/reporter, the parser in Test::Harness became the bottleneck. Nicholas Clark
Re: prove -j 9
On Mon, Sep 10, 2007 at 07:39:21PM +0200, demerphq wrote: > On 9/10/07, Eric Wilhelm <[EMAIL PROTECTED]> wrote: > > Note that your test suite may or may not play nicely with that. Have > > you ever run them in parallel before? ;-) Assuming common tempfiles > > and such really rains on the parade. Other than that, we're talking > > 40% less waiting. > > Will this make it easier to parallelize tests in the perl core? > > There the issue isnt so much running the tests in a single module > directory simultaneously but rather running the tests in different > test directories simultaneously. I had a hacked proof of concept for this, but it needed moderately large changes to Test::Harness to make it work. This was around the time that the TAPx::Parser work commenced, so it wasn't going to happen with Test::Harness 2. I still *have* the proof of concept code. The key thing I needed changed was that once you're running 2 or more tests you can't print any progress about *the* test currently running, because your assumption that there is just one is no longer valid. As to the core tests, they trip up like crazy, I think because they are not original in their choice of names for temporary files. Nicholas Clark
Re: Summarizing the pod tests thread
On Tue, Jul 31, 2007 at 04:43:08PM -0400, David Golden wrote: > * CPANTS -- eliminating or penalizing Kwalitee for pod/pod-coverage > tests would seem to be up to Thomas Klausner; I don't think I've seen > him weigh in on this recently. So people need to lobby him or fork > the project. He's somewhere in Outer Mongolia, and won't be back for about 2 weeks: http://use.perl.org/~domm/journal/33801 http://use.perl.org/~domm/journal/33878 He then might be rather busy until the end of YAPC::EU Don't be hasty with the f word. Nicholas Clark
Re: New TAP mailing list
On Mon, Mar 19, 2007 at 02:14:23PM +, Nicholas Clark wrote: > On Mon, Mar 19, 2007 at 09:12:08AM -0500, Andy Lester wrote: > > > > On Mar 19, 2007, at 8:11 AM, Andy Lester wrote: > > > > > > > > > > >Begin forwarded message: > > > > > >>Alright, [EMAIL PROTECTED] should work within 30 minutes or so. > > > > I dunno, I just saw that Nicholas Clark had requested it, and I > > forwarded the email. I don't know the story behind it. > > I didn't request it. The specific question I asked was "how does one go > about doing it?" Ask assumed that it was wanted. Which might be my fault for not making it clear that I was only asking about the process, not making a request. Nicholas Clark
Re: New TAP mailing list
On Mon, Mar 19, 2007 at 09:12:08AM -0500, Andy Lester wrote: > > On Mar 19, 2007, at 8:11 AM, Andy Lester wrote: > > > > > > >Begin forwarded message: > > > >>Alright, [EMAIL PROTECTED] should work within 30 minutes or so. > > I dunno, I just saw that Nicholas Clark had requested it, and I > forwarded the email. I don't know the story behind it. I didn't request it. The specific question I asked was "how does one go about doing it?" Ask assumed that it was wanted. Nicholas Clark
Re: Eliminating STDERR without any disruption.
On Sun, Mar 18, 2007 at 01:37:36AM +, Andy Armstrong wrote: > Are we expecting a YAML reader / writer to be core anytime soon? Not that I'm aware of. Nicholas Clark
Re: TAP::Parser internationalization
On Sun, Mar 18, 2007 at 05:00:32AM -0700, Ovid wrote: > This will likely *not* be in Test::Harness 3.0 unless it turns out to > be easier/more important than I thought, but I think that > internationalization may be important. Right now, all text that it > output is hard-coded in TAP::Parser/TAP::Harness/etc. This has some > negative consequences. > I have three questions: > > 1. How important is this feature? I don't know. I've only worked in English speaking companies. But getting the framework in, even if you only have 2 languages, where one is CAPITALS (or some-such) for testing makes it a lot easier later, as you don't have to go through retrofitting. But on the other hand, doing it now might be over engineering. > 2. Is a YAML phrasebook a sensible approach? Surely that's an implementation detail, behind Locale::Maketext ? > 3. Where's my beer? Last seen here: http://london.pm.org/meetings/locations/antelope.html There will be more beer on 5th April, venue to be confirmed. Possibly with an emergency before then because Simon Cozens is going to Japan. Nicholas Clark
Re: Eliminating STDERR without any disruption.
On Sun, Mar 18, 2007 at 01:06:48AM +, Andy Armstrong wrote: > And in fact now I see how simple the proposal is I realise that even > mentioning how it might be implemented is completely unnecessary. > Sorry folks - it's been a long day :) Well then, you should spend less time working actually writing code, and more time relaxing and being sociable by replying to e-mail on lists. :-) Meetings - the practical alternative to work. Nicholas Clark
Re: [ANNOUNCE] Test::More/Simple/Builder 0.68
Erk. Fails big time on 5.005_03
(Sorry, only just had reason to want to build 5.005_03)
On Tue, Mar 13, 2007 at 05:34:48PM -0700, Michael G Schwern wrote:
> 0.68 Tue Mar 13 17:27:26 PDT 2007
> Bug fixes
> * If your code has a $SIG{__DIE__} handler in some cases functions like
> use_ok(), require_ok(), can_ok() and isa_ok() could trigger that
> handler. [rt.cpan.org 23509]
> - Minor improvement to TB's filehandle detection in the case of overridden
> isa(). [rt.cpan.org 20890]
> - Will now install as a core module in 5.6.2 which ships with Test::More.
> [rt.cpan.org 25163]
>
> New Features
> - Test::Builder->is_fh() provides a way to determine if a thing
> can be used as a filehandle.
>
> Documentation improvements
> - Improved the docs for $Test::Builder::Level showing the encouraged
> use (increment, don't set)
> - Documented the return value of Test::Builder's test methods
> - Split out TB's method documentation to differenciate between test
> methods (ok, is_eq...), methods useful in testing (skip, BAILOUT...)
> and methods useful for building your own tests (maybe_regex...).
>
> Test fixes
> - We required too old a version of Test::Pod::Coverage. Need 1.08 and not
> 1.00. [rt.cpan.org 25351]
Something somewhere in those changes makes it go very wrong, as 0.67 works
a treat. I've not started digging yet, but the first test to clearly go
wrong is:
$ ~/Reference/5.005_03/bin/perl5.00503 -w -Mblib t/Builder.t
Using /export/home/nwc10/Perl/Test-Simple-0.68/blib
ok 6 - current_test() set
And this might give the game away:
$ ~/Reference/5.005_03/bin/perl5.00503 -w -Mblib t/buffer.t
Using /export/home/nwc10/Perl/Test-Simple-0.68/blib
ok 2 - You're ok
ok 4 - You're ok
ok 6 - You're ok
ok 8 - You're ok
ok 10 - You're ok
ok 12 - You're ok
ok 14 - You're ok
ok 16 - You're ok
ok 18 - You're ok
ok 20 - You're ok
given that the output is supposed to be:
$ perl -Mblib t/buffer.t
1..20
ok 1 - I'm ok
ok 2 - You're ok
ok 3 - I'm ok
ok 4 - You're ok
ok 5 - I'm ok
ok 6 - You're ok
ok 7 - I'm ok
ok 8 - You're ok
ok 9 - I'm ok
ok 10 - You're ok
ok 11 - I'm ok
ok 12 - You're ok
ok 13 - I'm ok
ok 14 - You're ok
ok 15 - I'm ok
ok 16 - You're ok
ok 17 - I'm ok
ok 18 - You're ok
ok 19 - I'm ok
ok 20 - You're ok
Does anything spring to mind as to the cause?
Nicholas Clark
Re: Paying for TAP 2.0
On Thu, Mar 08, 2007 at 11:05:31AM -0500, Christopher H. Laco wrote: > Paul Beckingham wrote: > > Because I need to retain the output of all tests, and those files get > > large, but mostly because of the sheer redundancy. > > > > > > > > But how do you know "23 ok" if you were never told that it ran ok? Surely one can post-process a regular TAP file to "sparse" output? And only do so if the TAP file is valid non-sparse output. This seems safer than generating it by default. Nicholas Clark
Re: Test::Simple API incompatibility - cpan-testers FAIL results
On Fri, Feb 16, 2007 at 09:17:00AM +1100, Adam Kennedy wrote: > >This comes from the OMGYOUBROKECPAN! diagnostics formatting change in > >0.60_01. > > Can we use that as the official name for this sort of error? > > Because the other alternative would be "Doing a Schwern" and I I'm sure > you'd prefer to preserve that term for something positive :) Such as herding MakeMaker for years with no-one ever saying thanks. Thanks, Schwern. (although I'm feeling a bit false as I'm not sure how much of that is simply because "I'm glad that it wasn't me who had to do it") Nicholas Clark
Re: Parallelizing TAPx::Harness?
On Mon, Feb 05, 2007 at 03:24:37AM +, Andy Armstrong wrote:
> On 5 Feb 2007, at 01:08, Eric Wilhelm wrote:
> >I was just thinking that since my tests are nicely orthogonal and yet
> >there's very little cpu utilization during testing, that I could
> >probably get them done about 20x faster if they were parallelized.
>
> [snip]
>
> I can't say off the top of my head what problems might arise.
> Obviously the output displayed during test runs would have to change.
> Sounds like a good idea though - I've added it to the TODO list :)
I had an experiment with making Test::Harness run in parallel. The code
exists, but is (intentionally) very hacky - I wanted to see what needed
doing.
>From memory, and I don't know how TAPx::Harness differs, currently when you
run a test under Test::Harness it does roughly the following:
1: Print out the test name with the s
2: Start the test running
3: Suck TAP data from the running test and print out progress
4: When the test completes print out its summary
5: repeat
To run tests in parallel, what I was doing was starting a farm of tests,
and when a test completed, printing out its results and starting another
one, until there were no new tests left to start, at which point wait for
tests to finish. The farm was run via a select loop.
So the logic had to change to roughly (and lets see if I can get the syntax
right):
async {
1: Start a test running
2: Suck TAP data from the test and buffer it
3: When the test completes process it all
4: Print out a summary
5: repeat
}
Important differences were
1: you only print out the entire test results as an atomic line when it has
finished
2: You have no idea which test is going to finish next
3: You can't print running progress on tests
Ideally the same code can run in both modes, so that when you aren't running
tests in parallel (which includes the case of waiting-for-the-last-test)
you go back to printing out the name and processing TAP as it appears.
Something I discovered was that processing TAP is slow. Up to about 7 CPUs,
parallelising tests feeding TAP into the master "thread" was a win. Beyond
that, the process processing the TAP was the bottleneck, and adding more
CPUs (and more parallel tests) didn't help. I would assume that from that
point on it would be better to have each test run by slave harness processes,
which takes the TAP, parses it, and slings (say) Storable objects across
to the master. But right now this isn't an issue as the 32 core Sun isn't
powered up. :-(
For the actual running of tests I tweaked runtests() so that the list of
arguments was treated as follows
1: Plain file names were tests to run in parallel, in any order
2: Array references were groups of tests to run in series
(This let me run tests for each module in ext/ and lib/ in order)
3: There was a special sync object that meant "complete everything up to here"
which was sufficient to give a lot of flexibility in how tests were run
(and discover that the core is very unoriginal in its naming of temporary
files)
Nicholas Clark
Re: Using pip to get testing done better and faster...
On Tue, Jan 09, 2007 at 02:30:56PM +, David Cantrell wrote: > Sadly, I doubt it's really possible. Perl makes far too many Unixy > assumptions. What are these "directories" of which you speak? :-) although I doubt that that one is really the clincher, as Perl has been ported to some somewhat un-Unixy portable devices. IIRC DOS didn't get directories until 1.1 Nicholas Clark
Re: Thoughts about test harness summary
On Fri, Jan 05, 2007 at 11:11:25AM -0800, Ovid wrote: > Pursuant to some discussion with BrowserUK at > http://perlmonks.org/?node_id=593087, I'm looking at this and seeing > some problems. > > -- > > Failed Test Stat Wstat Total Fail List of Failed > - > t/bar.t4 1024134 2 6-8 > t/foo.t1 256101 5 > (2 subtests UNEXPECTEDLY SUCCEEDED). > Failed 2/3 test scripts. 5/33 subtests failed. > Files=3, Tests=33, 0 wallclock secs ( 0.10 cusr + 0.01 csys = 0.11 > CPU) > Failed 2/3 test programs. 5/33 subtests failed. > > -- > > How about this instead? > > -- > > Failed Test | Total | Fail | List of Failed | TODO Passed > +---+--++ > t/bar.t | 13| 4|2, 6-8 |3-4 > +---+--++ > t/foo.t | 10| 1|5 | I'd like Wstat, even if I don't have Stat. I like to know if tests coredumped. I may be in a minority here, but being able to optionally switch to that output is useful. I like the prominence of TODO passed I'm not sure if I like the lines making the table. I guess it's a bit of a bikeshed, but having horizontal lines between each will increase the amount of vertical space needed to convey the same information, which will mean fewer failures will be needed to exceed my screen's height. Nicholas Clark
Re: Sparse Test Output
On Wed, Oct 25, 2006 at 03:53:58PM -0700, Michael G Schwern wrote: > I am totally unconvinced. I would love to see some profiling on this. I > would hope that the act of running 50,000 tests would swamp the cost of > parsing 50,000 tests. I don't have the exact numbers any more, but had experimented with multithreading the core tests (needs some refactoring of Test::Harness to do properly) I had 1 process parsing the TAP streams, with the others running the tests and generating the TAP. Above about 7 processes in parallel the TAP parser process was the bottleneck. This was on a 32 core Sun, so I had plenty of headroom to run more TAP generating threads. Nicholas Clark
Re: Failing test on Windows
On Tue, Oct 24, 2006 at 08:08:45PM -0400, Christopher H. Laco wrote: > With most modules, I agree. But with utility modules like > Module::Pluggable, File::Find::Recursive, etc, not working under taint I'd be surprised if the author of Module::Pluggable wasn't open to patches to fix this. Nicholas Clark
Re: AnnoCPAN Doc Patch Maker
On Sat, Oct 07, 2006 at 12:16:58AM -0400, Ivan Tubert-Brohman wrote:
> On 10/6/06, Ivan Tubert-Brohman <[EMAIL PROTECTED]> wrote:
> >I got something that can send you directly to rt.cpan.org and prefill
> >the textarea with the patch. The problem is that that textarea has
> >hard text wrapping which can cause the patch to be malformed.
> >Therefore, to make life easier for the module maintainer, the patch
> >should be attached as a file. The sad part for the patcher is that the
> >file can't be transmitted easily from one form to the other. The best
> >protocol I could come up with is the following:
> I'm thinking now that a convenient solution for the user would be to
> send the bug report directly by email (from annocpan.org to
> rt.perl.org, with From: being the user's email); that way I can
> include an attachment so the patch won't be messed up. Some drawbacks,
> if I understand RT correctly, are that the report would not be
> associated with an rt.cpan.org user account and the user won't be able
> to set some of the metadata (such as version numbers).
Given that {annocpan,rt}.perl.org both know who the user is, and both are
written in Perl, isn't there some web API available from RT that would let
an authenticated machine file a bug report with unmangled patch?
Nicholas Clark
Re: A short rant on the purpose of the CPAN install chain.
On Fri, Sep 22, 2006 at 09:47:02PM +0100, Adrian Howard wrote: > Belated response. For me good tests suites: > > * are good code (i.e. DRY, well factored, intention revealing, etc.) > * encourage good design (ala TDD) > * fail for bugs, but not refactoring > * only fail for bugs in the thing they're testing Which includes never failing due to random causes. (Such as there being a known 5% chance of a test failing) > * don't depend on the order the tests are run in Which to me means both 1: No .t file relies on others running before them to set up 2: Any .t file can be re-run infinite times without starting to fail (before 1 is achieved, due to some sort of set up data being exhausted) Nicholas Clark
Re: Bad TAP in Perl core?
On Sun, Sep 17, 2006 at 08:39:12AM -0700, Ovid wrote: > So I now have well over 2,000 tests for the parser, but there's a corner case > that's killing me. Test::Harness handles it fine, so I'm thinking that the > TAP specification (and subsequent grammar) is wrong. > Because TAP forbids unescaped hash marks in descriptions and that's how my > parser works, it falls down and goes boom. Unless I hear differently, I'm > going to have to adjust my parser to allow them. Otherwise I can't even > parse the Perl core tests, much less many different test modules on the CPAN. > This is because many people are putting their descriptions after hash marks, > not before. We could just fix the core. Nicholas Clark
Re: Perl::MinimumVersoin (was Re: New TAP Grammar)
On Sat, Sep 16, 2006 at 09:27:21AM -0400, Michael G Schwern wrote: > Seems to me this should be an config option to decide between checking the > point where a feature existed and when it stopped sucking. That sounds like a useful distinction. Hopefully for most features "stopped sucking" won't be vague (due to no clear single release where it became workable) Nicholas Clark
Re: New TAP Grammar
On Thu, Sep 14, 2006 at 09:28:19PM -0400, Michael G Schwern wrote: > Ovid wrote: > > Also, for purposes of backwards compatability, I'm concerned about qr// and > > $code->(). > > When were those first introduced in Perl? I don't want someone using > > really old versions to not be able to use this code. > > qr// was introduced in 5.6. $code->() has been in there as long as I know. > "use warnings" is 5.6. > > To give you an idea, the Test-Simple distribution has been broken on 5.5.3 > for a few releases now (Test::Builder::Tester uses qr//) and I've gotten a > grand total of one bug report about it. I think that it came in with 5.005_00. Certainly, it's in 5.005_03, which is the earliest version I can find precompiled: $ perl5.00503 -le '$a = qr/(.*)/; $] =~ $a; print $1' 5.00503 So whatever the actual cause of the bug leading to the report is, it wouldn't seem to be what the reporter thought it was. Nicholas Clark
Re: post-YAPC::Europe CPANTS news
On Fri, Sep 08, 2006 at 09:15:47PM +1000, Adam Kennedy wrote: > One of the linux distro guys pinged me about Test::Object needing the > very latest (CPAN-only) version of Test::Builder, because it means they > can't package it properly for the distros without upgrading the main > Perl package. Some packaging systems can't handle having the same file > in more than one distro it seems. Said maintainers can work round the deficiencies of their packaging systems by breaking out all the dual life modules from the core Perl package, and making them into other packages on which the core Perl package depends. In my opinion. Clearly this might be a bit tricky if the packager of Test::Object isn't the same person as the packager of core Perl, because then the former has to persuade the latter of the need for such a change. Alternatively said maintainers can work round the deficiencies of their packaging system by having another directory tree in @INC ahead of the core library paths, into which they install newer versions of dual life modules. This way no files need to be overwritten on disk, or dual-owned, yet Perl will still pick up the correct (newer) version. Nicholas Clark
Re: designing a test suite for multiple implementations (tools thread)
On Mon, Aug 14, 2006 at 07:12:06PM -0700, chromatic wrote: > PS - sbk30, please don't send me any more automated followup messages. Fix > your mailing software. I've found that our resident neighbourhood BOFHs have been very helpful at forcibly un-subscribing anyone anti-social enough to be sending automated crap in response to mailing list messages. (Given the full headers of the offensive message. BOFH contact details being in the message headers of any message distributed by the list) Nicholas Clark
Re: designing a test suite for multiple implementations
On Sat, Aug 12, 2006 at 09:44:38PM +0800, Audrey Tang wrote: > That's assuming that the new repo, say, http://svn.perl.org/ > perl6tests/, can > give out commit permissions to parrot and pugs committers, yes. > > But as Jerry's initial motivation was moving Perl 6 tests to a more > official > location, and that http://svn.perl.org/perl6/ is the official repo > for the > Perl 6 design documents, I wonder what's the advantage of hosting the > tests > in a separate repository. Can you elaborate on that? I thought that it might be easier to audit different committer bit policies if the repositories were different, instead of simply one being a subtree of another. That was all. Nicholas Clark
Re: designing a test suite for multiple implementations
On Sat, Aug 12, 2006 at 03:32:47PM +0800, Audrey Tang wrote: > > ?b 2006/8/12 ?W?? 3:01 ???Ajerry gay ?g???G > > >the question is, should it be moved into their own repository, or > >into the repo of > >the "official" perl6 implementation (if such a beast will indeed > >exist,) > > Currently the svn.perl.org repo is the most official-sounding one, by > the domain > name alone. (But I don't understand the motivation for putting the > tests with > "the true implementation" -- I thought the idea is to decouple the > tests with > any implementations.) So I think svn.perl.org is the right choice, > if the admins > are okay with a more relaxed commit bit policy there. There's nothing technical stopping the Perl 6 tests being on svn.perl.org, but in a different svn repository from the current repositories, is there? Setting a svn:external property in the right place on both Parrot and Pugs would mean that both could check out the same testsuite, and both could commit back to it. Nicholas Clark
unit tests or functional tests
This is sort of off-topic because it's more a general question about testing, rather than Perl specific, but the code in question happens to be written in Perl... There is this big hairball of under-tested code. (Nothing new here) So the question is, which to tackle first - unit tests, or functional tests. A colleague's view is that you can't have functional tests until you know that the individual units work, hence start with unit tests. (I believe that the assumption is that when they're mostly complete start on functional tests, but that wasn't stated). This seems the logical approach if you want to refactor things. My view is that because the actual output of the code isn't well specified (sadly nothing that new there either), if we write functional tests to verify that the behaviour we desire is present, then we're actually killing two birds with one stone - we have tests for the spec, and the tests are the spec. (Which isn't perfect as specs go, but it's a heck of a lot better than the current spec). Also, right now we really don't care about the specific output of the the individual units that make up the code - all that matters to the client is the final behaviour. Hence writing unit tests at fine detail for existing code could well be a (relative) waste of effort in that it's quite possible that the units they test are thrown away soon if the implementation is changed. Whereas functional requirements are much less likely to change on a release-by-release basis, so writing them is less likely to generate code that has a short lifetime. And having functional tests is likely to give us better coverage up front, so we're more likely to spot a change that unintentionally breaks behaviour, even if we can't use them to efficiently nail down which change was the culprit. Why am I wrong? Nicholas Clark
Re: planning at the end
On Thu, Jul 20, 2006 at 04:01:00PM -0700, Michael G Schwern wrote: > On 7/20/06, Gabor Szabo <[EMAIL PROTECTED]> wrote: > >If I am not mistaken the problem with no_plan is that the test script > >might exit before actually running all the tests you wanted and Harness > >won't notice it. > > PS In all my years of testing I have never been bitten by this in the > real world which is why I've never bothered to code up a solution. > Anyone else? I've been bitten by library code calling exit() while I've been using no_plan. I can't remember more than that (well, apart from that library wasn't pleasant code to read or use). I would have found some sort of "done testing" call useful to have made the test script fail for this case. Nicholas Clark
Re: Time for a Revolution
On Sat, Jul 15, 2006 at 08:36:54AM +1000, Andrew Savige wrote: > --- Clayton O'Neill wrote: > > I think a core difference between your list and Chromatic's is that > > yours would be part of the standard library in a lot of languages, > > whereas Chromatic seems to be aiming more for things that would be > > part of the language. > > Who's Chromatic? And it wasn't even the start of a sentence. :-) [When doing the perl 6 summaries, Piers reconciled the forces of accuracy and traditional grammar by ensuring by always rephrasing to some sentence order that didn't start with "chromatic"] Nicholas Clark
Re: TAP Namespace Nonproliferation Treaty
On Sat, Jul 08, 2006 at 05:10:50AM -0700, Ovid wrote: > - Original Message > > From: Nik Clayton <[EMAIL PROTECTED]> > > TAPxParser > > > Thought about that, but immediately discarded it. TAPx::OVID::Parser doesn't > say anything about the parser other than authorship and the latter is > verified by glancing at the docs. My jumping on the TAPx::Parser namespace > is almost as bad as my taking TAP::Parser, so I'm just trying to figure out > how to truly distinguish my (currently incomplete) parser from others. I > suspect it will be as arbitrary as your idea, though :) Also naming modules with the original author's CPANID is going to become "fun" if maintainership is transferred, or the original author wants to start a clean second implementation. Nicholas Clark
Re: Proposal Suggestion - Test::Run
On Wed, Jul 05, 2006 at 11:57:42PM -0700, chromatic wrote: > On Wednesday 05 July 2006 23:02, Shlomi Fish n wrote: > > > I don't see using the X11 licence for my software as anti-social. Like I > > said, anyone can easily fork it as a software of a different licence. > > Supposing you actually find a mentee and TPF actually does fund this project > and this code is substantially better than Test::Harness such that it is an > obvious candidate for inclusion in the Perl core. > > It must be under the Perl license. For the core, what he says. Period. Nicholas Clark
Re: TAP::Harness
On Sat, Jul 01, 2006 at 02:36:37PM -0500, Michael G Schwern wrote: > Right now the use cases I have in mind include things such as > parallelized test runs, fancy GUI and HTML outputs (for example, > Provide use cases, what would you want to do with Test::Harness if you > could? What are you doing with Straps? What features do other Use case: Parallel test runs I'd like to be able to specify parallel test runs. Specifically, I'd like to be able to send tests as a set of groups, with instructions on how the tests are to be parallelised. I already have a prototype, hacked from Test::Harness. 1: Send tests as groups 2: Tests within a group run in series (in traditional order) 3: Groups run in parallel 4: have sync points. All groups before a sync point must complete before any groups after a sync point can start. This let me run many of the core tests in parallel, but still allowed the tests that need to run in order to run in order. This level of control let me run the core tests as follows: t/base/*.t one by one (each test is its own group, sync point between each group) t/comp/*.t t/cmd/*.t t/run/*.t t/io/*.t t/op/*.t tests in each directory in sequence, but the 5 directories in parallel wait for all these to finish t/uni/*.t, all tests in ext and lib tests in a the same directory ran in series. All directories could run in parallel With this I can max out a multi CPU (or multi core) box. (Or at least stand a fighting chance, although tests in t/op/*.t in the core need some issues fixing before they can run in parallel with each other. I suspect that their names for temporary files are unimaginative and therefore clash) I've got the code for a select driven event loop that can run all this, if that's helpful. Nicholas Clark
Re: Test::Harness wrangling
On Wed, Jun 28, 2006 at 11:13:02PM -0500, Andy Lester wrote: > Tomorrow, Adam Kennedy and I (and Schwern?) will be banging on > Test::Harness. > > Any bugs that we especially need to work on? They're not so much bugs as structural annoyances that it would be useful to relax to make subclassing Test::Harness more useful. (Well, most accurately, to allow me to write a parallel test runner easily) I hacked Test::Harness and Test::Harness::Straps around to get parallel tests running (version 1), but I made no effort to keep the conventional behaviour working cleanly, hence I'm not sure if sending you that code would be useful. However, based on that, things that it would be nice to refactor out (from memory): The code to do the printing of the test names, the ...s and the numbers could be refactored out into a proper helper class, to allow derivatives more flexibility in what the chose to display. This class would suck in the subroutines associated with &strap_callback as methods. In turn, it would stop the evil storing of callback state in the main T::H object. In turn, the globals $ML $Last_ML_Print can be eliminated and stored as state in this. Nicholas Clark
Re: Test me please: P/PE/PETDANCE/Test-Harness-2.57_06.tar.gz
On Sun, Apr 23, 2006 at 01:02:00AM -0500, Andy Lester wrote:
> I'm approaching the end of this release cycle. I really want to get
> this released.
>
> I've removed the meaningless percentages of tests that have failed.
> If you rely on the output at the end, it's different now.
>
> xoa
> >
> > file: $CPAN/authors/id/P/PE/PETDANCE/Test-Harness-2.57_06.tar.gz
> > size: 69114 bytes
> > md5: 41efc0985146e4f7d678ec7cb9b59047
I have merged this in to blead with change 27940. (Because I want the speedup)
Specifically blead is now identical (Unless I messed up) which means that
this change got removed before I committed:
--- Test-Harness-2.57_06/lib/Test/Harness.pmSun Apr 23 06:53:11 2006
+++ perl/lib/Test/Harness.pmSun Apr 23 21:23:30 2006
@@ -38,7 +38,8 @@
=cut
-$VERSION = '2.57_06';
+$VERSION = "2.57_06";
+$VERSION = eval $VERSION;
# Backwards compatibility for exportable variable names.
*verbose = *Verbose;
And this change did get committed:
//depot/perl/lib/Test/Harness/t/version.t#2 -
/home/nick/p4perl/perl/lib/Test/Harness/t/version.t
--- /tmp/tmp.27775.0Sun Apr 23 21:25:55 2006
+++ /home/nick/p4perl/perl/lib/Test/Harness/t/version.t Sun Apr 23 21:23:28 2006
@@ -19,5 +19,5 @@ BEGIN {
}
my $ver = $ENV{HARNESS_VERSION} or die "HARNESS_VERSION not set";
-like( $ver, qr/^2.\d\d(_?\d\d)?$/, "Version is proper format" );
+ok( $ver =~ /^2.\d\d(_\d\d)?$/, "Version is proper format" );
is( $ver, $Test::Harness::VERSION );
Note that I removed the ? from the _ inside the regexp, which was added
in http://public.activestate.com/cgi-bin/perlbrowse?patch=27925
I felt it better to get the code perfectly in sync than to preserve minor
recent (possible) fixes and have to keep track of the fork.
Nicholas Clark
Re: TODO tests
On Tue, Apr 18, 2006 at 01:21:37PM -0500, Steve Peters wrote: > One of my unwritten TODOs is to go through the current Perlbug queue and > write test cases for all the currently testable problems. My hope is > that unexpected fixes would be caught much sooner in these cases. I've > made a bit of a start on this, and given tuits (or proper financial > motivation) I'm hoping to finish it sometime this year. Of course, The is an application form for motivation: http://www.perlfoundation.org/gc/grants/proposals.html Beware. Motivation is a double edged sword. It means everyone knows that you're going to do stuff. Nicholas Clark
TODO tests
Last time I checked the core has "6 TESTS UNEXPECTEDLY SUCCEEDED" What's the expected number of unexpected successes? Can it be made to be zero, even though we're testing the test modules? If so, I think that that would be useful, as it would mean that any (real) TODO test that unexpectedly started passing would be noticed. I bring this up because we seem to have inadvertently fixed really old regexp bugs that we didn't have a test case for, but I realise that right now adding TODO tests wouldn't actually have been *that* useful - if a TODO passes we don't notice. It would be good if we were in a position to notice. I'm not sure how much work that would be. Nicholas Clark
Re: Module requirements (was: Module::Build and installing in non-standardlocations)
On Tue, Apr 04, 2006 at 11:27:07AM +0200, Sébastien Aperghis-Tramoni wrote: > I don't think that the problem of "core is too big" is a matter of disk > size, but more a matter of number of modules. P5Porters time is a scarce > ressource, and they already lack the time to do all the work they'd > like to do just on the interpreter. Making core modules dual-life is > a way to handle these to someone else who has spare time and who doesn't > need to have deep XS or Perl guts knowledge. At least I think that's > the reason, otherwise why was I accepted as the maintainer of two > such modules (XSLoader and Sys::Syslog)? Well, I think it's that reason too. :-) Yes, to me, "size" is maintainance liability, not disk space or bandwidth. Putting things in core is a pain. Keeping them there is a pain. I remember the "fun" of getting Storable sufficiently portable that it could go into the core. Trying to work around strange issues thrown up by certain AIX compilers in certain configurations... Nicholas Clark
Re: New kwalitee metric - eg/ directory
On Tue, Mar 14, 2006 at 12:48:52PM -0500, David Golden wrote: > Adam Kennedy wrote: > >For all those component distributions I consider it a failure if it is > >so complex that you need something more than just three or four lines > >from the SYNOPSIS. > > Maybe there should be a Kwalitee metric for the length of the synopsis? I was thinking this too. Some synopses aren't. Nicholas Clark
Re: Kwalitee in your dependencies (was CPAN Upload: etc etc)
On Fri, Feb 03, 2006 at 11:30:13AM +0100, Tels wrote: > Problaby just because the last guy running RISC OS has died 4 years ago. > SCNR :-) Well, the list is *slightly* more active than that: http://www.nntp.perl.org/group/perl.riscos Nicholas Clark
Re: Dependency trees was: CPAN Upload: D/DO/DOMM/Module-CPANTS-Analyse-0.5.tar.gz
On Sat, Jan 28, 2006 at 09:08:48PM +1100, Adam Kennedy wrote: > Graph::Easy installation failing here with YAML 0.50 (newer versions of > YAML seem to be uninstallable at the moment due to Class::Spiffy + > Spiffy + Test::Base install failures... > > Any suggestions? You're getting install failures even with the newest versions of Class::Spiffy etc? I had failures with YAML not liking an existing installed Spiffy, but upgrading everything seemed to resolve that. If that's not it, then I don't know, but Ingy is often around on IRC. Nicholas Clark
Re: Kwalitee in your dependencies (was CPAN Upload: etc etc)
On Fri, Jan 27, 2006 at 11:50:49PM -0800, chromatic wrote:
> Let me save you the trouble of writing it to find the biggest problem right
> now: fairly broken automated testing systems that can't even *run* the
> Build.PL file *or* the compatibility Makefile.PL yet send FAIL reports
> anyway.
I believe that the following won't be a problem with PITA, but the general
smoke system really $expletive annoys me for sending FAIL reports when it
tries to build one of my XS modules on a system without a C compiler.
My opinion is that this is a BUG IN THE SMOKE SYSTEM. The default for perl
is source distribution, so to build perl it's inferred you need a C compiler.
MakeMaker and Module build already automatically handle converting any .xs or
.c files that I ship into installable modules. When creating my package, I
don't need to do anything - it *already just works*. It's already supported.
So my opinion is that it's not necessary for me to manually add to my metadata
saying "I need a C compiler". If my module ships with .c or .xs files, it's
BLOODY OBVIOUS.
So the smoke system should verify whether $Config::Config{cc} exists and can
run, and if it can't, N/A any package that isn't pure perl.
Yes, I really like valid smoke reports, pass or fail.
But FALSE NEGATIVES ARE PROCESS ERRORS.
Nicholas Clark
Re: Flexible testing
On Wed, Dec 21, 2005 at 01:34:39PM -0800, Joe McMahon wrote: > print < 1..1 > not ok 1 ... failed once > not ok 1 ... failed twice > ok 1 ... worked > EOS > Obviously, you can wrap up the actual test in a retry loop/function, > but this doesn't match up with the simplicity of Test::More and related > testing methods. It seems like the only way to address this is to > subclass Test::Builder (or write a new class) that buffers up the test > output and only outputs it after tests are "committed" (i.e., "I've run > this N times and am sticking with the final result"). > > Or am I stretching TAP too far? Thoughts? To me the cleanest way to do this ought to be to subclass Test::Builder to give the behaviour that it only prints the last result line for a given test number when it sees the first result line for the next test number appears. That would mean that the body of your test script would still call all the regular ok/is/like etc functions without changing them, and your test script still outputs regular TAP where each numbered test reports exactly once. Nicholas Clark
Re: ENV problems with testing
On Mon, Sep 19, 2005 at 09:42:52PM -0400, James E Keenan wrote: > (Secondary reason for tests needing 5.6: as you, Schwern, pointed out > several weeks back, doing without qr// is just too painful!) 5.005 introduced qr// $ ~/Reference/5.005_04/bin/perl5.00504-32 -le '$r = qr/(p...)/; $^X =~ $r; print $1' perl Nicholas Clark
Re: Testing module madness
On Sun, Sep 11, 2005 at 12:35:43PM -0500, Andy Lester wrote: > >Usually, Test::* modules are only used for the test phase. > > I really don't understand the idea of "only used for the test phase", > as if the tests don't matter, or if there are levels of failure. > Either they install OK on the target system, and you can use them > with confidence, and they've done their job, or you're going to > ignore the tests completely and then who needs 'em? > > It's like if I'm installing a washing machine, and I don't have a > level. I can say "Ah, I only need it for the installation, and it > looks pretty level, so I don't need the level", or I can say "I'm not > using this appliance until I've proven to myself that the machine is > level and won't cause me any problems in the future because of an > imbalance." This is a good analogy. It's correct. But the assumptions behind it cover only one case. It's as if the requirements for the washing machine say: To install and use this machine you will need: * a power supply * a water supply * drainage * a level which is valid if you're both the installer and the user. But if someone else helps you install the machine, then you don't actually need the level, if they bring theirs and use it for the install. I think that the build_requires/test_requires distinction *is* important, if it can be made, as it eases the lives of anyone wishing to package up modules, build them from source in one place, and then distribute their packages to many other machines, be they OS vendors or sysadmins. The tests are run and pass on the build machine, prior to packaging. But the automatic dependency system doesn't need to make installation of this module depend on installing Test::* onto the production machine. (for the general case) But it's only important if it's easy to make. And I'd much prefer time and effort to go into writing better modules, better tests, and better tools, than generating heat. Nicholas Clark
Re: kwalitee: drop Acme?
On Fri, Sep 09, 2005 at 10:37:01PM +1000, Adam Kennedy wrote: > Or make is_prereq SO easy to game that it's a nonissue? Why should a > module depended upon by another author be ranked any higher than one > that isn't. For CPANTS I see no reason. For search results quite the opposite. I'd really like if if the default way people got search results back for CPAN modules at least attempted to order at some level based on citations. (ie number of pre-requisites) For example, I believe that that way the 4 or so decent template modules would fall out immediately from the 50 others on CPAN. And that this is worth doing because the commonly used templating modules are almost certainly what most people who need to search would find more than good enough. Rather than letting them fight through bogosorted results and getting swamped by choice. But it seems that there is something amiss with the roundness of my stock of tuits. (or in other words there are things I find I care more about and deal with first) Nicholas Clark
Re: kwalitee: drop Acme?
On Thu, Sep 08, 2005 at 11:20:33PM +0200, Philippe 'BooK' Bruhat wrote: > Le jeudi 08 septembre 2005 à 10:45, Ovid écrivait: > Take a less random example: > > Acme::MetaSyntactic You've been learning this new definition of "random" from cog, haven't you? :-) [eg "random person to wear fishnet"] > > We should at least throw the poor module author's a bone and leave > > Acme:: out of this. > > Hey! that would decrease *my* score! ;-) I've no idea what it would do to my score. Nicholas Clark
Re: OSCON testing tutorial?
On Wed, Jul 20, 2005 at 02:52:40PM -0500, Andy Lester wrote: > On Wed, Jul 20, 2005 at 08:49:11PM +0100, Nicholas Clark ([EMAIL PROTECTED]) > wrote: > > Because Andy would be far too polite to push the book he and Richard Foley > > wrote: http://www.apress.com/book/bookDisplay.html?bID=399 > > That's because I didn't write it. I'm more a uber-tech-editor and > helped shape some of the prose, but none of it is really my writing. > That's why it's "with Andy Lester." > > Apress asked me to sign books at the book signing at OSCON and I > declined. It's really Richard's book. Well, "Andy far too polite", so at least I was 50% correct. :-) Nicholas Clark
Re: OSCON testing tutorial?
On Wed, Jul 20, 2005 at 02:46:43PM -0500, Andy Lester wrote: > On Wed, Jul 20, 2005 at 11:57:36AM -0700, Michael G Schwern ([EMAIL > PROTECTED]) wrote: > > Am I imagining things or was there supposed to be a testing tutorial at > > OSCON > > with Andy Lester, chromatic and the gang? Or am I thinking of YAPC? > > No, it's not a tutorial. It's 90 minutes of random testing stuff. > Think 16 testing lightning talks. A big grab bag of testing excitement. > It's me, Bill Odom, Ian Langworth and chromatic, the latter two pushing > their new "Perl Testing: A Developer's Notebook". Because Andy would be far too polite to push the book he and Richard Foley wrote: http://www.apress.com/book/bookDisplay.html?bID=399 Nicholas Clark
Re: False Positives from Automated Testing at testers.cpan.org
On Tue, Jul 19, 2005 at 09:37:39PM -0700, Michael G Schwern wrote: > On Tue, Jul 19, 2005 at 10:49:12PM -0400, James E Keenan wrote: > > Am I correct in this inference and this judgment? Or is there something > > about the automated testing that I don't understand? > > If I understand correctly, the issue here is you failed to list a > dependency yet the tests failed to catch it because that module was already > installed. Its a false positive, but not testers.cpan.org's fault. It > would have to attempt to automatically figure out what modules your code > needs and check that you list them as dependencies. This is fraut with > peril. When I rule the world (and therefore have requisitioned everyone's tuits) there will be smoke testing that starts from a clean install each time. However, right now I don't have time to try to write this. Sorry. Nicholas Clark
Re: like, cmp_ok and undef
On Fri, Jul 01, 2005 at 07:26:16PM -0700, Michael G Schwern wrote: > It seems pretty clear to me that most warnings should not be supressed. > But what about uninit warnings, specificly? Is there any utility to > something like: > > like( $foo, qr/.../ ); > > and: > > cmp_ok( $foo, 'eq', $bar ); > > warning about $foo being undefined? Especially since the test diagnostics > will let you know about what was undef. I'm not sure. For the specific example of cmp_ok with 'eq', then because the programmer has chosen that over is(), it seems fair that he/she made an explicit choice that $foo being undef was an acceptable match for $bar being an empty string. (Not so sure about the other 13 comparison operators) I think I'd prefer like to fail (distinct from warning) if $foo is undef, given that the expected regexp may well intentionally match an empty string. Nicholas Clark
Re: [RFC] Test::CPANTS
On Mon, Mar 14, 2005 at 04:05:36PM -0500, Christopher H. Laco wrote: > I have an itch. It just came to me while surfing PerlMonks and CPAN. > I noticed the new Test::Strict module which keeps me honest by making > sure I always 'use strict'. I'll be adding that to my modules this > evening. [I wish is did use warnings too]. Acme::use::strict::with::pride does warnings too :-) http://search.cpan.org/dist/Acme-use-strict-with-pride/pride.pm I'm not suggesting that you use it in production code though. Partly because it is an Acme module, but mainly because it stops Exporter working. Then again, OO purists might consider that a feature. Nicholas Clark
Re: CPAN modules coverage
On Mon, Mar 07, 2005 at 07:59:40PM +0100, Paul Johnson wrote: > To do it properly it would need to be on a machine somewhere which would > accept uploaded coverage databases from anyone who wanted to submit one. > > I discussed cover.perl.org or something with Andy and Robrt on irc a > while back, but I've not made progress on the sw side. Because your free time is already filled up with working on Devel::Cover itself? This sounds like the sort of project that anyone who felt particularly keen about could try experimenting with at home, to see if they can get it to work. Having a demonstrable experiment is a very good first step to convincing others to help you with it, and having something that is mostly finished is a good way to get someone else to host a fully working version. Actions speak louder than words. Nicholas Clark
Re: Test comments
On Mon, Feb 14, 2005 at 10:04:44AM -0800, Ovid wrote: > Is this not correct? Where is the TAP protocol documented? http://search.cpan.org/~petdance/Test-Harness-2.46/lib/Test/Harness/TAP.pod (Any Test-Harness distribution 2.46 or later, IIRC) Nicholas Clark
Re: Anomalous Difference in Output between HTML Files Created by
On Mon, Jan 31, 2005 at 10:12:16AM +0100, Paul Johnson wrote: > I suppose that's the price you pay for TIMTOWTDI. > > [ Is that a Python programmer I hear giggling in the background? ] Does Python have any equivalent tool to Devel::Cover? Nicholas Clark
Re: Test::Builder versus Unicode
On Wed, Dec 22, 2004 at 11:41:56AM -0800, Ovid wrote: > --- Nicholas Clark <[EMAIL PROTECTED]> wrote: > > > On Wed, Dec 22, 2004 at 10:26:02AM -0800, David Wheeler wrote: > > > 1. Perl gets smarter about duping file handles, so that the dupes > > get > > > the same i/o layer settings as the handles they dupe. > > > > Changing this going forwards doesn't change any of the installed > > perls out there in the wild. > > So far, given that this problem has only surfaced in relation to > Unicode, I can't say I'm overly concerned about fixing it on versions > of Perl where Unicode is already known to be broken. Of course, as Which you're sort of implying is all versions up to and including 5.8.6 :-) (Well, I can misread it as that. I don't think that you really are implying this). Personally I'd be quite happy using anything 5.8.3 or later with Unicode. The later the better, as more bugs have got fixed. But I feel that Unicode isn't more broken than any other existing part of perl by 5.8.1. And they are out there, and they aren't going away rapidly. Nicholas Clark
Re: Test::Builder versus Unicode
On Wed, Dec 22, 2004 at 10:26:02AM -0800, David Wheeler wrote: > 1. Perl gets smarter about duping file handles, so that the dupes get > the same i/o layer settings as the handles they dupe. Changing this going forwards doesn't change any of the installed perls out there in the wild. So whatever happens in the future, if we want things to work on existing installations, we need to work round the problem in some way. (I don't currently have time to look into or think about whether it's a bug. Or how to fix it if it is. Or who could fix it.) Nicholas Clark
