Re: TAP - Test::More - fork
Schwern, On Thu, Nov 10, 2011 at 5:57 PM, Michael G Schwern wrote: > On 2011.11.10 4:59 PM, Buddy Burden wrote: >> Does that do anything? I didn't think prove respected the shebang >> line. Anyway, I thought the -w to prove would be effectively doing >> that all along. > > Perl respects the *options* on the #! line, prove only adds to them. I did not know this ... just to be super-clear, obviously I know that if I have script.pl and it starts with #! /usr/bin/perl -w and I make it executable and run it directly, I get perl -w. But you're saying that even if I type: perl script.pl I _still_ get perl -w? That, I was not aware of. > Adding -w to the #! line on your tests is a very good idea to avoid the > gotcha of differences between a test being run with `prove` (does not set > -w), ... Well, yes, prove doesn't _normally_ set -w, but if you run prove -w, it does, doesn't it? -- Buddy
Re: Problem with running lots of tests (I think)
Guys, Okay, just to follow-up in case anyone cared what the resolution on this one was, changing the loop full of ok()s to one giant pass() or fail() per loop fixed _everything_. Plus it runs a lot faster now. I know I've seen test suites that do thousands and thousands of tests, but they must be doing it some other way that I'm not privvy to or somethin'. Anyway, thanx everyone for the help; I've finally gotten a non-developer release of Data::Random out tonight. -- Buddy
Smoker setup to compare old vs new versions?
I'd like a smoker setup which can do two things: 1) Run smokes for all the Test:: modules. 2) Compare the results between two different installed versions of Test::More. This will allow me to smoke Test::Builder 1.5 against CPAN, see what it's broken and try to fix it. I've never used the CPAN smoker software before and I'm hoping somebody has a recipe? Thanks. -- package Outer::Space; use Test::More tests => 9;
Re: Discuss change of namespace Test::Builder2 -> TB2?
On Mon, Nov 14, 2011 at 12:13:45PM -0800, Michael G Schwern wrote: > Besides, "tee bee two" rolls off the mouth nicely and TB:: is a bit too short. Also I'd expect TB::* to be something to do with tuberculosis. -- David Cantrell | Cake Smuggler Extraordinaire Vegetarian: n: a person who, due to poor lifestyle choices, is more likely to get arse cancer than a normal person
Re: Dual life t/test.pl?
This was suggested previously in 2008 http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2008-06/msg00912.html On Mon, Nov 14, 2011 at 04:59:57PM -0800, Michael G Schwern wrote: > Having a parallel and featureful testing system is very useful. I use it to > test Test::More (in the Test-Builder1.5 branch). Others might find it useful > to do the same. Other than people writing test modules, who would find it useful? > t/test.pl is very important to the Perl core tests, but it is largely > undocumented and untested. Going through the p5p process to patch it makes > working on it difficult and open to bike shedding. If it were dual-lived it > could be worked on as its own thing with its own commits and tests. test.pl is a script, not a package. Our machinery for dealing with dual life things is really only set up to deal with modules. So doing this is not going to just drop in to what we have. > By having a dedicated feed of issues and commits, careful attention could be > paid to what code and features are going in. Close attention and discussion > can happen about whether the code is simple enough and the features apt for > the needs of the core. Surely the needs of the core *are* better determined by the core developers, not some third party group? Also, having upstream be anywhere other than the core actively gets in the way of t/test.pl actually being useful to core developers (or at least to me, as a core developer who has added things to it). Even having it being upstream blead gets in the way of it being useful. Right now, as all users of the core's t/test.pl are in the core, it's possible to a: add things to it without stopping to think "is this a decent, supportable API that can last forever", because if the API turns out to be wrong, it's entirely possible to have a "flag day" commit that changes it and all its users b: remove things from it, because nothing is using them any more c: not worry about what versions of perl it supports, or working round problems in them (Although to be fair, if a particular piece of perl syntax had been buggy before, it's possibly something fragile that should be avoided in case it breaks again for a different reason) Nothing in the above stops anyone taking a copy of test.pl and using it. Doing this also doesn't add a layer of process into working on the core. > If folks are cool with that, I can set up a Github repository for it. We'll > worry about whether CPAN release is appropriate later. I'm distinctly uncool about doing it that way. I'm unconvinced about doing it with blead upstream. This feels like a lot of work for little benefit, with the work outweighing the benefit. We did start work on trying to extract runperl() as Test::Builder module. There's a branch from blead where I attempted to convert module tests to use it and found which features were necessary for them: http://perl5.git.perl.org/perl.git/shortlog/refs/heads/smoke-me/runperl Beware that IPC::Cmd can't capture output on Win32 with core-only modules, and IPC::Run can't reliably capture output on Win32, thanks to Microsoft's socketpair implementation following advice from Klortho: #11912 Evidently it's important to you to get the wrong answer as quickly as possible. (The default for closing a socket on Win32 is to drop all unsent buffered data on the floor. No, I'm not kidding. Note To assure that all data is sent and received on a connection, an application should call shutdown before calling closesocket (see Graceful shutdown, linger options, and socket closure for more information). Also note, an FD_CLOSE network event is not posted after closesocket is called. http://msdn.microsoft.com/en-us/library/ms737582%28v=vs.85%29.aspx I've also found http://msdn.microsoft.com/en-us/library/ms738547%28v=vs.85%29.aspx Whereas the BSD socket() implementation works in such a way that a socket FD can be substituted for a file FD, pipe FD or tty FD, and the program continues to work just as it did before, without needing any special case code. Doesn't even need recompiling.) Nicholas Clark
Re: Discuss change of namespace Test::Builder2 -> TB2?
On Mon, Nov 14, 2011 at 4:11 AM, Michael G Schwern wrote: > I have written "Test::Builder2" more times than I can count. > > Wait, I can count them. It occurs at least 1280 times in the distribution. > That's 18,000 keystrokes. 19,000 if you count shift. It gets a little > tiresome. That's just the current count, who knows how many times I've > retyped it. > > In the interest of saving my hands, and the hands of test module authors, I'm > considering changing the namespace from Test::Builder2 to just TB2. Like LWP > and DBI. > > Test::Builder2 has a lot of parts, as you can see here: > https://metacpan.org/release/MSCHWERN/Test-Simple-2.00_07/ > > Here's some examples. > > Before After > -- - > Test::Builder2 TB2 > Test::Builder2::History TB2::History. > Test::Builder2::Event::SetPlan > TB2::Event::SetPlan > Test::Builder2::Formatter::TAP::v13 TB2::Formatter::TAP::v13 > > Test::Builder, Test::More and Test::Simple remain unchanged. > > This both saves wear on joints and it establishes a clear "keep outta my > namespace" namespace for TB2 to expand into. 3rd party extensions can go into > TB2X:: or just Test:: like normal. > > On the down side, it doesn't contain the word "Test". I'm not too worried as > this is the guts of testing, and by the time you need it you should already > have picked up enough Perl to find it. > > It also busts compatibility with the existing TB2 alphas... but that's what > alphas are for. I've very well broken compatibility nine other ways this > month anyway. > > Oh, and somebody has to do it. > > Thoughts? I'm not seeing the point really. By this logic we can reduce all frameworks on CPAN to some three letter acronym. To be honest I don't think Test::Builder is used directly often enough to justify that. Leon
Re: Discuss change of namespace Test::Builder2 -> TB2?
On 11/15/11 8:40 AM, Leon Timmermans wrote: I'm not seeing the point really. By this logic we can reduce all frameworks on CPAN to some three letter acronym. To be honest I don't think Test::Builder is used directly often enough to justify that. I'm against abbreviation; it makes things harder to read. Your text editor's completion mechanism should take care of having to typing out long names.
Re: Smoker setup to compare old vs new versions?
On Tue, Nov 15, 2011 at 8:28 AM, Michael G Schwern wrote: > I've never used the CPAN smoker software before and I'm hoping somebody has a > recipe? Regression testing is less well developed. As far as I know, people have rolled their own. I have some poorly documented tools I used to use to regression smoke Module::Build: https://github.com/dagolden/cpan-testers-tools Steffen has another regression smoker (possibly with some parts adapted from mine) that he's been using to do Perl regression tests against CPAN: https://github.com/tsee/cpan_perl_branch_smoke The general approach is this: * Smoke configuration 1 and save reports to a directory * Smoke configuration 2 and save reports to a different directory * Compare directories and output HTML to browse differences Hope that helps point you in the right direction. -- David
Pod-Perldoc needs tests. You can help.
I've taken over Pod-Perldoc and was surprised to find that it has virtually no tests. When I started, it had several calls to pass() and a checked that three modules loaded. You can help change that. There are many interesting test challenges here. For instance, Pod::Perldoc::ToMan, perhaps the most used of all the formatters, shells out to pod2man and then to nroff. That's three things that need some encoding love. And, shelling out from a Perl program to a Perl program that is a call to a module makes it hard to test: the big red flag that Your Doing It Wrong. Many of the issues in RT deal with pagers and how they act on various systems. I have no idea how to test that. Maybe you do. There's a *::ToTk module. Good luck testing that. However, there are some things we can do to fail with better error messages. There are various niggling pod things that perldoc has to handle correctly, too, and I imagine testing those would benefit modules such as Pod::Checker. -- brian d foy
Re: Dual life t/test.pl?
On Nov 15, 2011, at 6:43 AM, Jerry D. Hedden wrote: >> Other than people writing test modules, who would find it useful? > > FWIW, threads and threads::shared use test.pl in their test > suites. This seems to be historical and related to the fact > that older versions of Test::More didn't work will with > threads. However, changing to use Test::More may be > undesirable because it means testing threads using a module > that needs to support threads testing. The DBI uses it, too, for performance testing, I believe. http://svn.perl.org/viewvc/modules/dbi/trunk/test.pl?view=markup Best, David
Re: Dual life t/test.pl?
On Tue, Nov 15, 2011 at 09:34, Nicholas Clark wrote: > This was suggested previously in 2008 > > http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2008-06/msg00912.html > > On Mon, Nov 14, 2011 at 04:59:57PM -0800, Michael G Schwern wrote: >> Having a parallel and featureful testing system is very useful. I use it to >> test Test::More (in the Test-Builder1.5 branch). Others might find it useful >> to do the same. > > Other than people writing test modules, who would find it useful? FWIW, threads and threads::shared use test.pl in their test suites. This seems to be historical and related to the fact that older versions of Test::More didn't work will with threads. However, changing to use Test::More may be undesirable because it means testing threads using a module that needs to support threads testing.
Re: TAP - Test::More - fork
On 2011.11.15 1:01 AM, Buddy Burden wrote: > I did not know this ... just to be super-clear, obviously I know that > if I have script.pl and it starts with > > #! /usr/bin/perl -w > > and I make it executable and run it directly, I get perl -w. But > you're saying that even if I type: > > perl script.pl > > I _still_ get perl -w? That, I was not aware of. Yep. Now, want to have your mind blown? $ cat ~/tmp/test #!/usr/bin/python def hello(): print "Hello world!" return hello(); $ python ~/tmp/test Hello world! $ perl ~/tmp/test Hello world! >> Adding -w to the #! line on your tests is a very good idea to avoid the >> gotcha of differences between a test being run with `prove` (does not set >> -w), ... > > Well, yes, prove doesn't _normally_ set -w, but if you run prove -w, > it does, doesn't it? Yes. If you don't set it in #! you, or somebody else, will forget to pass -w to prove and then you'll spent a bunch of time scratching your head wondering why the test is running differently from in "make test". -- Just call me 'Moron Sugar'. http://www.somethingpositive.net/sp05182002.shtml
Re: Problem with running lots of tests (I think)
On 2011.11.15 1:14 AM, Buddy Burden wrote: > Okay, just to follow-up in case anyone cared what the resolution on > this one was, changing the loop full of ok()s to one giant pass() or > fail() per loop fixed _everything_. Plus it runs a lot faster now. I > know I've seen test suites that do thousands and thousands of tests, > but they must be doing it some other way that I'm not privvy to or > somethin'. There is a 3rd option. The perl core maintains a separate implementation of Test::More in t/test.pl. You can drop that into your project and use it for your enormous tests. It does not record test results, that I know, and should not consume memory. I use it to test Test::More and am considering spinning it out into its own project. -- 94. Crucifixes do not ward off officers, and I should not test that. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: Discuss change of namespace Test::Builder2 -> TB2?
On 2011.11.15 6:40 AM, Leon Timmermans wrote: > I'm not seeing the point really. By this logic we can reduce all > frameworks on CPAN to some three letter acronym. To be honest I don't > think Test::Builder is used directly often enough to justify that. Test::Builder was just one monolithic module. It isn't any more. There's lots of pieces now. People who develop Test modules that do interesting things will be typing it more often. People who don't won't even know anything has changed, so they're not really a concern. Elliot Shank uttered: > I'm against abbreviation; it makes things harder to read. Your > text editor's completion mechanism should take care of having to > typing out long names. When you make them up on the spot and use them twice, abbreviations are bad. When you repeat it 1300 times in the project... folks will figure it out. While I agree good editor skills are good, and "I have a crappy editor" isn't an excuse, getting an editor to autocomplete Perl class names is non-trivial. It gets into TAG files and having the editor be aware of the whole project and parsing Perl correctly and guh. I don't know how to do it. I don't expect contributors and users to know or have a copy of Perl Hacks. -- 101. I am not allowed to mount a bayonet on a crew-served weapon. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: Dual life t/test.pl?
First thing to clarify, I don't really envision this being turned into a CPAN module. That would be directly against its purpose. The primary purpose here is to separate and focus development and provide a point for module authors to copy it into their project. It probably will *not* be released to CPAN and it probably will *not* be API stable. The needs of the core outweigh the needs of the few people that use it. On 2011.11.15 6:34 AM, Nicholas Clark wrote: > Other than people writing test modules, who would find it useful? Test::More is about to become a lot more complicated internally, so anyone who does things which Test::More might be using. In addition, .t files which run a bazillion tests would find it useful both for speed and memory usage. I just recommended Buddy Burden use it for some huge Data::Random tests. Abigail could use it for her enormous Regexp::Common test suites. But that's not really what it's about. The library upon which 372 core tests rely is undocumented and untested. Moving it into its own repo and tracker allows it to be tested and stable without having to go through the p5p memorial bike shed. It's a lot easier to get people to work on a single file Perl library than it is to get them to patch the Perl core. In its own project, I can hand a newbie a URL like this and say "pick one of these and do it!" https://github.com/schwern/test-more/issues?labels=Easy >> t/test.pl is very important to the Perl core tests, but it is largely >> undocumented and untested. Going through the p5p process to patch it makes >> working on it difficult and open to bike shedding. If it were dual-lived it >> could be worked on as its own thing with its own commits and tests. > > test.pl is a script, not a package. Our machinery for dealing with dual > life things is really only set up to deal with modules. So doing this is not > going to just drop in to what we have. It will exactly drop in to what we have. Pulling it into its own repository doesn't mean it has to be turned into a package and modularized and all that. It would probably become lib/test.pl and you just copy that into t/test.pl. >> By having a dedicated feed of issues and commits, careful attention could be >> paid to what code and features are going in. Close attention and discussion >> can happen about whether the code is simple enough and the features apt for >> the needs of the core. > > Surely the needs of the core *are* better determined by the core developers, > not some third party group? The groups would obviously overlap and we wouldn't be stingy with the commit bits. > Also, having upstream be anywhere other than the core actively gets in the > way of t/test.pl actually being useful to core developers (or at least to > me, as a core developer who has added things to it). We deal with dual life files all the time, the same procedure applies. If you do it a lot (and you have this year) get a commit bit. Make the changes in a sterile environment with tests. Copy it into blead. Or do it in blead and paste the commit ID into an issue on the project to get someone else to pick it up. To put this in perspective, t/test.pl was edited just 28 times this year. Nearly all of them were by you in March. > Even having it being upstream blead gets in the way of it being useful. > Right now, as all users of the core's t/test.pl are in the core, it's > possible to > > a: add things to it without stopping to think "is this a decent, supportable >API that can last forever", because if the API turns out to be wrong, >it's entirely possible to have a "flag day" commit that changes it and >all its users > b: remove things from it, because nothing is using them any more > c: not worry about what versions of perl it supports, or working round >problems in them >(Although to be fair, if a particular piece of perl syntax had been > buggy before, it's possibly something fragile that should be avoided in > case it breaks again for a different reason) > > Nothing in the above stops anyone taking a copy of test.pl and using it. > Doing this also doesn't add a layer of process into working on the core. I'm ok with all that. The goal is not API stability. Moving it to its own repo does not prevent any of a, b or c. >> If folks are cool with that, I can set up a Github repository for it. We'll >> worry about whether CPAN release is appropriate later. > > I'm distinctly uncool about doing it that way. > I'm unconvinced about doing it with blead upstream. I was thinking blead would be downstream. -- Being faith-based doesn't trump reality. -- Bruce Sterling
Task: Write a TAP v12 formatter for Test::Builder1.5
I have an important task for getting Test::Builder 1.5 stable. Test::Builder 1.5 outputs TAP version 13. There are minor formatting changes including a TAP version header and changes to the ending commentary. A lot of tests look at this output and so they break. Rather than make everybody change their tests (and have to upgrade their Test::More dependency), I'd rather provide a legacy TAP formatter that outputs like Test::Builder currently does. Then folks can switch it on in their tests, and Test::Builder::Tester will use it by default. That's the task. Write a TAP version 12 formatter. https://github.com/schwern/test-more/issues/215 https://github.com/schwern/test-more/wiki/Preferred-workflow It's a mildly difficult task, mostly because the v13 formatter does things which really should be done in a superclass, but it's way easier than trying to do it in Test::Builder 0.x. If anyone would like to take it on, please say so in the ticket and we'll try to get things off the ground. Thanks. -- 164. There is no such thing as a were-virgin. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
