Re: [Module::Build] [RFC] author tests
Tyler MacDonald writes: Chris Dolan [EMAIL PROTECTED] wrote: [lots of author test examples, including:] * versionsync.t - Checks that the $VERSION is the same in all bin/* and *.pm files. This test is pointless after release, since it's already been tested before release * pod-coverage.t - Checks POD completeness. This test is pointless after release, since it's already been tested before release ... since there's absolutely no value in these types of tests for anybody else except the module author, there's no real point in having a convention, just stick 'em anywhere that the make/buildfiles will ignore them. That premise is untrue. Some module authors sync the version numbers of all files (like Chris apparently does); some prefer only to bump the version number in a file if that particular file has changed. If I'm sending a patch to a module author I sometimes try to make it be a complete patch for the distro -- including the code, the docs, updating the change-log, and so on. And hopefully I'd run the tests before submitting the patch. So the inclusion of versionsync.t would clue me in that Chris wanted all the version numbers changing, and I could do that in the patch, thereby reducing the work Chris has to do to accept my patch and increasing the chance of me getting the feature I wanted included. Or perhaps as a user of a module I think the docs could do with a little improvement. Before submitting a patch it'd be useful to run pod-coverage.t to check that I haven't missed anything or made matters worse. Remember that this is software with the source available -- so there isn't a strict author v user distinction: any user can suddenly elect to modify the code and become an author (even if only for modifying a private version). Smylers
Re: [Module::Build] [RFC] author tests
Adam Kennedy [EMAIL PROTECTED] wrote: Write this up. Then exhaustively test it on every single Perl platform (50ish?) and every Perl version back to 5.004, including a random collection similarly weird combinations (5.004 VMS, that 5.6.0 from RedHat 7, 5.6.1 on Windows 95). I let testers.cpan.org and ppm.activestate.com take care of that. Then make sure the code is so clean and complete you'll never need add another lines of code or tweak the docs. That's difficult, which is why nearly every core perl module has a newer version available on CPAN... Then lets think about adding a new compulsory near-core dependency. Ok, so Chris and I were getting a bit too excited about the idea. :) A new module doesn't need to be added to the core, so long as there is a way that we can reliably detect when a person wishes to build and test any given perl package for an objectively unselfish purpose such as 1:prepackaging, 2:automated testing, or 3:releasing. All three are viral so it's best to make sure they do no harm, while still maintaining some level of convenience for the end user. There's already AUTOMATED_TESTING for 2 and make disttest for #3 (just keep a file around in your repo that's not in your MANIFEST and test for it's presence.) Any good way to detect #1? And now that I think about it, I'm not so convinced about that whole concenience for the end user nonsense. If they're mucking about installing perl modules from the CPAN packages by themself, they're probably developers that need some extra time to sit there and think about what sort of madness they're getting themselves into anyway. - Tyler
Re: [Module::Build] [RFC] author tests
On Thu, Feb 02, 2006 at 02:56:09AM -0800, Tyler MacDonald wrote: A new module doesn't need to be added to the core, so long as there is a way that we can reliably detect when a person wishes to build and test any given perl package for an objectively unselfish purpose such as 1:prepackaging, 2:automated testing, or 3:releasing. All three are viral so it's best to make sure they do no harm, while still maintaining some level of convenience for the end user. There's already AUTOMATED_TESTING for 2 and make disttest for #3 (just keep a file around in your repo that's not in your MANIFEST and test for it's presence.) Any good way to detect #1? An environment variable. PERL_TEST_EXHAUSTIVE? And make/Build disttest can set it too. In the perl core, the few tests that are normally not run are requested by a -torture switch. But I think despite this precedent exhaustive sounds better. But please, lets not have new ta/ or whatever directories or .ta files; this is a problem that is well taken care of with skip_all. If somebody really wants, they could go the Test::Exhaustive route, which would automatically do the skip_all if the env vars weren't set. But putting a few stock lines at the start of your .t file isn't really all that big a deal.
Re: [Module::Build] [RFC] author tests
Hi, Tyler MacDonald wrote: And now that I think about it, I'm not so convinced about that whole concenience for the end user nonsense. If they're mucking about installing perl modules from the CPAN packages by themself, they're probably developers that need some extra time to sit there and think about what sort of madness they're getting themselves into anyway. I strongly disagree! CPAN is the preferred source for any Perl modules and the associated tools like CPAN.pm or CPANPLUS are the preferred method of installing them. Why? Because since you are not going to see all modules in .deb or whatever packages and not in the most recent versions. Using both the system's package system and CPAN's for module installation is going to bite you on most systems some day. Also, if you write applications that use CPAN modules, you might a) not have the license to include the modules or b) not want to be responsible for maintaining the bundled versions or c) not want to inflict 5 gazillion installed versions of module Foo on every user and d) not want to create different distributions for different platforms. Thus, user's need to be able to install Perl modules from CPAN. Of course, that's just my understanding and my opinion. I have had to maintain such a cross-platform application that needed to bundle a lot of modules. It was hell. We're extraordinarily lucky to have such a good toolchain and such a great repository of modules. Steffen
Re: [Module::Build] [RFC] author tests
On Feb 1, 2006, at 10:35 PM, Tyler MacDonald wrote: Chris Dolan [EMAIL PROTECTED] wrote: There is a class of tests that module authors perform that end users are not expected to run. For example code coverage tests, spelling tests, coding style tests, etc. These tests are either prohibitively expensive or complicated or unpredictable for end users to run. I call these private tests or author tests. I really like this idea. But as you pointed out, it's not just authors that need to worry about running these tests, it's packagers (ppm/deb/etc), automated testers (cpants/testers.cpan.org/etc), and hackers. I'd suggest we call these exhaustive tests. No, I disagree. I'm specifically talking about author tests, NOT packager tests. Things like Test::Spelling are pointless and difficult for packagers to execute because Test::Spelling relies on an external aspell or ispell program *and* performs differently in the presence of an author's custom dictionary (mine has Dolan; does yours?) These specifically are not exhaustive tests but spit-and-polish tests. To make this less abstract, let me list the specific author tests that I employ for most of my CPAN modules, along with an explanation of why it wouldn't work for a packager * copyright.t - Ensures that there is a Copyright .([localtime]- [5]+1900) somewhere in every .pm file. Will break 11 months from now. * distribution.t - Relies on Test::Distribution, which is not in my prereq list * perlcritic.t - Runs Test::Perl::Critic on all .pm files. Will fail without my specific $HOME/.perlcriticrc and will fail with future, more exhaustive versions of P::C * spelling.t - Runs Test::Spelling. Will fail without my custom dictionary * versionsync.t - Checks that the $VERSION is the same in all bin/* and *.pm files. This test is pointless after release, since it's already been tested before release * pod.t - Checks POD validity. This test is pointless after release, since it's already been tested before release * pod-coverage.t - Checks POD completeness. This test is pointless after release, since it's already been tested before release and one I have not yet employed: * coverage.t - Ensures that Devel::Cover totals are higher than some threshold Chris -- Chris Dolan, Software Developer, Clotho Advanced Media Inc. 608-294-7900, fax 294-7025, 1435 E Main St, Madison WI 53703 vCard: http://www.chrisdolan.net/ChrisDolan.vcf Clotho Advanced Media, Inc. - Creators of MediaLandscape Software (http://www.media-landscape.com/) and partners in the revolutionary Croquet project (http://www.opencroquet.org/)
Re: [Module::Build] [RFC] author tests
Steffen Mueller [EMAIL PROTECTED] wrote: And now that I think about it, I'm not so convinced about that whole concenience for the end user nonsense. If they're mucking about installing perl modules from the CPAN packages by themself, they're probably developers that need some extra time to sit there and think about what sort of madness they're getting themselves into anyway. I strongly disagree! CPAN is the preferred source for any Perl modules and the associated tools like CPAN.pm or CPANPLUS are the preferred method of installing them. Why? Because since you are not going to see all modules in .deb or whatever packages and not in the most recent versions. Using both the system's package system and CPAN's for module installation is going to bite you on most systems some day. Steffen, I more meant the madness they are getting themselves into by programming in perl in the first place. :-) - Tyler
Re: [Module::Build] [RFC] author tests
* Chris Dolan [EMAIL PROTECTED] [2006-02-02 16:55]: On Feb 1, 2006, at 10:35 PM, Tyler MacDonald wrote: I really like this idea. But as you pointed out, it's not just authors that need to worry about running these tests, it's packagers (ppm/deb/etc), automated testers (cpants/testers.cpan.org/etc), and hackers. No, I disagree. I'm specifically talking about author tests, NOT packager tests. Things like Test::Spelling are pointless and difficult for packagers to execute because Test::Spelling relies on an external aspell or ispell program *and* performs differently in the presence of an author's custom dictionary (mine has Dolan; does yours?) These specifically are not exhaustive tests but spit-and-polish tests. I was just gonna say. It’s pointless for anyone but the author to check POD or test coverage. Only under the assumption that the author was negligent and shipped a distribution without running the POD tests does it make any sense for a packager to run them. And then it still doesn’t make sense for *every* packager to run them. Similarly for Devel::Cover – what’s the packager to do, write more tests to include with the platform-specific package? That makes no sense. These are tests that need to pass once on the author’s system *before* release. If a release is cut without them passing, it’s pointless of others to re-run them. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: [Module::Build] [RFC] author tests
A. Pagaltzis wrote: * Chris Dolan [EMAIL PROTECTED] [2006-02-02 16:55]: On Feb 1, 2006, at 10:35 PM, Tyler MacDonald wrote: I really like this idea. But as you pointed out, it's not just authors that need to worry about running these tests, it's packagers (ppm/deb/etc), automated testers (cpants/testers.cpan.org/etc), and hackers. No, I disagree. I'm specifically talking about author tests, NOT packager tests. Things like Test::Spelling are pointless and difficult for packagers to execute because Test::Spelling relies on an external aspell or ispell program *and* performs differently in the presence of an author's custom dictionary (mine has Dolan; does yours?) These specifically are not exhaustive tests but spit-and-polish tests. I was just gonna say. It’s pointless for anyone but the author to check POD or test coverage. Only under the assumption that the author was negligent and shipped a distribution without running the POD tests does it make any sense for a packager to run them. And then it still doesn’t make sense for *every* packager to run them. Similarly for Devel::Cover – what’s the packager to do, Normally I'd agree, but that's not 100% set in stone. I work on Linux/Win32 and run the usual pod syntax/coverage tests on those platforms. I also have FreeBSD servers for which package could/would be created. On more than one occasion, I've had pod2html/man (troff) errors under FreeBSD that were only found by running the author tests there, even though the pod syntax/coverage was perfectly fine on two other platforms. It's these sorts of problems where the packager running those tests are quite beneficial...Especially on platforms where the packagers/dists may be adding patches to the core dist. -=Chris signature.asc Description: OpenPGP digital signature
Re: [Module::Build] [RFC] author tests
Chris Dolan [EMAIL PROTECTED] wrote: * copyright.t - Ensures that there is a Copyright .([localtime]- [5]+1900) somewhere in every .pm file. Will break 11 months from now. * distribution.t - Relies on Test::Distribution, which is not in my prereq list * perlcritic.t - Runs Test::Perl::Critic on all .pm files. Will fail without my specific $HOME/.perlcriticrc and will fail with future, more exhaustive versions of P::C * spelling.t - Runs Test::Spelling. Will fail without my custom dictionary * versionsync.t - Checks that the $VERSION is the same in all bin/* and *.pm files. This test is pointless after release, since it's already been tested before release * pod.t - Checks POD validity. This test is pointless after release, since it's already been tested before release * pod-coverage.t - Checks POD completeness. This test is pointless after release, since it's already been tested before release and one I have not yet employed: * coverage.t - Ensures that Devel::Cover totals are higher than some threshold Wow, you really *are* exhaustive. How do you find the time to write any code? ;-) Now that I understand exactly what you mean by author tests, here's what I think: Whatever convention you're using, if these tests are only going to work on your system, then they definately shouldn't be in t. And since there's absolutely no value in these types of tests for anybody else except the module author, there's no real point in having a convention, just stick 'em anywhere that the make/buildfiles will ignore them. - Tyler
Re: [Module::Build] [RFC] author tests
A. Pagaltzis [EMAIL PROTECTED] wrote: I was just gonna say. It???s pointless for anyone but the author to check POD or test coverage. I agree about the POD coverage. But if I got a different level of code coverage on somebody else's system than my own? I'd be very interested in finding out why. I was actually considering playing with YACSmoke and Devel::Cover to make a site that shows the code coverage metrics for all of CPAN, but then I realised that I'm getting far too obsessed with this testing stuff and I need to focus on writing real code if I ever want to finish an application. :-) - Tyler
Re: [Module::Build] [RFC] author tests
On Thursday 02 February 2006 10:04, Tyler MacDonald wrote: A. Pagaltzis [EMAIL PROTECTED] wrote: I was just gonna say. It???s pointless for anyone but the author to check POD or test coverage. I agree about the POD coverage. But if I got a different level of code coverage on somebody else's system than my own? I'd be very interested in finding out why. Perhaps this is something PITA could report. -- c
Re: [Module::Build] [RFC] author tests
On Thursday 02 February 2006 02:56, Tyler MacDonald wrote: And now that I think about it, I'm not so convinced about that whole concenience for the end user nonsense. If they're mucking about installing perl modules from the CPAN packages by themself, they're probably developers that need some extra time to sit there and think about what sort of madness they're getting themselves into anyway. Ahh, but what about all of those developers using Perl 5.004 who just *have* to be on the bleeding edge of new code? (Yeah, I don't understand that argument either.) -- c
Re: [Module::Build] [RFC] author tests
* Christopher H. Laco [EMAIL PROTECTED] [2006-02-02 18:50]: On more than one occasion, I've had pod2html/man (troff) errors under FreeBSD that were only found by running the author tests there, even though the pod syntax/coverage was perfectly fine on two other platforms. Are you sure that’s not instead an argument from fixing the corresponding pod2foo converter to behave on that platform? Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: [Module::Build] [RFC] author tests
* copyright.t - Ensures that there is a Copyright .([localtime]-[5]+1900) somewhere in every .pm file. Will break 11 months from now. * distribution.t - Relies on Test::Distribution, which is not in my prereq list snipped ones that need your personal files, you are certainly right there. I wouldn't even include them at all. Put them in the MAKEFILE.SKIP. * versionsync.t - Checks that the $VERSION is the same in all bin/* and *.pm files. This test is pointless after release, since it's already been tested before release This one you are wrong on. It's important to have this enabled in the distribution. The reason is the following. Imagine you add a new module, or remove a module. For various reasons (often involving things like Class::Autouse recursive loading or Module::Pluggable or similar things) you can in some cases accidentally load a module from an old version of the dist installed somewhere else in @INC. versionsync.t (or various other implementations of it) is not only testing that the versions within the dist are synced, but that in loading your module you aren't ACCIDENTALLY loading something else you shouldn't be. I've had it happen 2 or three times in real life now on things like PPI (which had a huge number of classes which moved a number of times) and the Email::Send problem that only got fixed a few days ado. * pod.t - Checks POD validity. This test is pointless after release, since it's already been tested before release * pod-coverage.t - Checks POD completeness. This test is pointless after release, since it's already been tested before release and one I have not yet employed: * coverage.t - Ensures that Devel::Cover totals are higher than some threshold There is some value in running these in some contexts. If you look at the way the make/Build system works, you can see it's designed to encourage free software and distributed development. Otherwise why would you have a Makefile for everyone that installs a module that is also capable of doing a new release. Now personally, with the exception of pod.t I don't even include the files in my distribution. Those sorts of tests are done by my release tool. But I'm starting to think that might be wrong... TEST_AUTHOR or TEST_RELEASE seems like a good idea to me. It doesn't add anything to the CPAN release infrastructure, and it's easy to implement. That said, you'd have to manually turn it on when you were releasing, and then turn if off when installing other people's modules... :/ But I don't know that I like disttest autodetection. I quite like being able to run the additional tests manually if needed, and not be limited to only during the disttest process. Adam K
Re: [Module::Build] [RFC] author tests
chromatic wrote: On Thursday 02 February 2006 10:04, Tyler MacDonald wrote: A. Pagaltzis [EMAIL PROTECTED] wrote: I was just gonna say. It???s pointless for anyone but the author to check POD or test coverage. I agree about the POD coverage. But if I got a different level of code coverage on somebody else's system than my own? I'd be very interested in finding out why. Perhaps this is something PITA could report. -- c Just as a datapoint on this topic, the PITA request objects (as of 5 minutes ago) now support the ability to explicitly set environment variables you want set when running the tests, on top of the default-but-overridable ones like AUTOMATED_TESTING (yes, you can turn it off if you really want to). So any additional tests you want to allow to be enabled by an environment variable, we can help you out with the on other systems bit. Also, if anyone has a good knowledge of Xen, and they let me know? Adam K
Re: [Module::Build] [RFC] author tests
On Thursday 02 February 2006 17:45, Adam Kennedy wrote: Just as a datapoint on this topic, the PITA request objects (as of 5 minutes ago) now support the ability to explicitly set environment variables you want set when running the tests, on top of the default-but-overridable ones like AUTOMATED_TESTING (yes, you can turn it off if you really want to). So any additional tests you want to allow to be enabled by an environment variable, we can help you out with the on other systems bit. Fantastic! I hope to release some Module::Build-based code in the next *mumble* to show one way of segregating tests without adding boilerplate to their headers. -- c
Re: [Module::Build] [RFC] author tests
* Adam Kennedy [EMAIL PROTECTED] [2006-02-03 02:45]: But I don't know that I like disttest autodetection. I quite like being able to run the additional tests manually if needed, and not be limited to only during the disttest process. Maybe do it the other way around? Define INSTALL_AUTOMATED and INSTALL_INTERACTIVE environment variables that get set during installation, so author tests run by default and can be skipped when installing. That does require touching more infrastructure, though. But it should solve the problem long-term, because by doing it that way, excluding tests not meant for end-user installation would not require assinging names up front for every phase in the lifecycle of a module and skipping each of them explicitly. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: [Module::Build] [RFC] author tests
On Thu, Feb 02, 2006 at 10:01:48AM -0800, Tyler MacDonald wrote: I strongly feel that authors should keep everything necessary for their distribution public; either in the CPAN distribution itself, or via a permanent publicly available version control system. Who's to say you won't lose interest in maintaining the module or in perl altogether at some point? Or move to Antarctica and be unable to maintain it? Or have your home/business or wherever your files are kept destroyed in a hurricane or other natural disaster? And, of course, we all will die someday. Some suddenly. I do agree that many (all?) of these tests are irrelevant to someone packaging your distribution. Chris Dolan [EMAIL PROTECTED] wrote: * copyright.t - Ensures that there is a Copyright .([localtime]- [5]+1900) somewhere in every .pm file. Will break 11 months from now. * distribution.t - Relies on Test::Distribution, which is not in my prereq list * perlcritic.t - Runs Test::Perl::Critic on all .pm files. Will fail without my specific $HOME/.perlcriticrc Test::Perl::Critic allows you to configure Perl::Critic by passing the path to a Fperlcriticrc file in the Cuse pragma. For example: use Test::Perl::Critic (-profile = 't/perlcriticrc'); all_critic_ok(); Probably you'd like this to keep in sync with any changes to your .perlcriticrc, which may require some changes to your Makefile.PL/ Build.PL. and will fail with future, more exhaustive versions of P::C It would be nice if there was some way to indicate which version of P::C was expected to pass, and TODO any newly looked for problems. * spelling.t - Runs Test::Spelling. Will fail without my custom dictionary There's Test::Spelling::add_stopwords and =for stopwords. These should be used as much as possible instead of adding to your custom dictionary. But at least your custom dictionary is recreatable (because all the needed words are included in your pod :) should someone else pick up maintenance of your distribution. * versionsync.t - Checks that the $VERSION is the same in all bin/* and *.pm files. This test is pointless after release, since it's already been tested before release * pod.t - Checks POD validity. This test is pointless after release, since it's already been tested before release * pod-coverage.t - Checks POD completeness. This test is pointless after release, since it's already been tested before release and one I have not yet employed: * coverage.t - Ensures that Devel::Cover totals are higher than some threshold Wow, you really *are* exhaustive. How do you find the time to write any code? ;-) Now that I understand exactly what you mean by author tests, here's what I think: Whatever convention you're using, if these tests are only going to work on your system, then they definately shouldn't be in t. And since there's absolutely no value in these types of tests for anybody else except the module author, there's no real point in having a convention, just stick 'em anywhere that the make/buildfiles will ignore them. I disagree; presumably anyone running disttest would want these tests run, so they belong in t and named .t, with an appropriate skip_all at the top. Does anyone have a problem with disttest setting PERL_TEST_EXHAUSTIVE? Or a suggestion for a better name? Chris, how are you currently set up to run these tests only when preparing a release?
Re: [Module::Build] [RFC] author tests
On Feb 2, 2006, at 9:19 PM, Yitzchak Scott-Thoennes wrote: Chris, how are you currently set up to run these tests only when preparing a release? I make no such distinction. Instead, I see these tests as part of my day-to-day development and run all of them with every ./Build test. Presently, I simply include them all in my MANIFEST.SKIP so they are not included in my uploaded package. That's why I call these author tests and not release tests or exhaustive tests. Chris -- Chris Dolan, Software Developer, Clotho Advanced Media Inc. 608-294-7900, fax 294-7025, 1435 E Main St, Madison WI 53703 vCard: http://www.chrisdolan.net/ChrisDolan.vcf Clotho Advanced Media, Inc. - Creators of MediaLandscape Software (http://www.media-landscape.com/) and partners in the revolutionary Croquet project (http://www.opencroquet.org/)
Re: [Module::Build] [RFC] author tests
That doesn't allow for what happens if both are missing. I find the following a good general rule. There are tests that test how the code actually works, that could convievably be different on different platforms. These are always on. There are tests for issues that are almost certainly (but not provably) going to pass everywhere, but add extra dependencies (and thus increase risks relating to failures, perl version deps, and platform compatibility). I include POD tests, because your pod might have issues with older POD parsers found in some places. They get disabled BY _by_default_ unless the end user happens to have the module anyway, but they don't hold up the show for someone that doesn't. And they get run in automated testing, sometimes (really depends if the automated testing has testing the end-user case or testing the module itself as it's primary focus) There are tests that are ONLY going to need to be run by the author before they can release. They get disabled _by_default_ unless the person running the tests asks them to be run. Note the by defaults. In the absense of any information to the contrary, everything but the primary functionality testing should always be off. This keep dependencies and thus fallout to a minimum, and lets people do what they need to. As for naming of things, I'd consider the tests for the author to be release testing, and thus perhaps the name TEST_RELEASE for the environment variable to enable for release testing? It also has the nice property of being an imperative command when you say it. Testing system, test release! Adam K A. Pagaltzis wrote: * Adam Kennedy [EMAIL PROTECTED] [2006-02-03 02:45]: But I don't know that I like disttest autodetection. I quite like being able to run the additional tests manually if needed, and not be limited to only during the disttest process. Maybe do it the other way around? Define INSTALL_AUTOMATED and INSTALL_INTERACTIVE environment variables that get set during installation, so author tests run by default and can be skipped when installing. That does require touching more infrastructure, though. But it should solve the problem long-term, because by doing it that way, excluding tests not meant for end-user installation would not require assinging names up front for every phase in the lifecycle of a module and skipping each of them explicitly. Regards,
Re: [Module::Build] [RFC] author tests
Chris Dolan [EMAIL PROTECTED] wrote: There is a class of tests that module authors perform that end users are not expected to run. For example code coverage tests, spelling tests, coding style tests, etc. These tests are either prohibitively expensive or complicated or unpredictable for end users to run. I call these private tests or author tests. I really like this idea. But as you pointed out, it's not just authors that need to worry about running these tests, it's packagers (ppm/deb/etc), automated testers (cpants/testers.cpan.org/etc), and hackers. I'd suggest we call these exhaustive tests. Spelling tests, code coverage tests, testing interaction with CPAN.pm, etc. are things that I think should be done by automated testers, and by people packaging your module for distribution inside another larger system (eg, PPMs or debian packages). So maybe it would make sense if there were certain tests that only run if AUTOMATED_TESTING is set, and/or if the disttest action is being run. That said, it would be easy to whip together a Test:: module that only allows these tests to run under these conditions, especially if Module::Build, say, set a DIST_TEST environment variable when the disttest action was being run. I already check for AUTOMATED_TESTING to manipulate my test environments to be more exhaustive. Both Module::Build, and perhaps future versions of ExtUtils::MakeMaker (if there will ever be a future version) could depend on this Test:: module (whether or not they use it themselves) so that it's already on everybody's system. It'd be nice and small so I don't see anybody complaining too loudly. Maybe we could call it Test::Exhaustive? It's mandate being to SKIP_ALL of a test if you havent somehow (AUTOMATED_TESTING, disttest, etc) made it clear that you wish to exhaustively test this module? Roughly, I propose that we adopt a standard file extension for author tests, like t/*.ta, and add an ACTION_authortest to M::B that runs both t/*.t and t/*.ta (which ones first??) If they were named differently (i don't think they have to be), I'd want to see them run in lexical order like the current tests, so it'd go 01foo.t, 02bar.ta, 03baz.t. Cheers, Tyler
Re: [Module::Build] [RFC] author tests
The trouble is, EVERYONE wants to add just one more little dependency (me included *cough*Params::Util*cough*). I'll make you a deal. Write this up. Then exhaustively test it on every single Perl platform (50ish?) and every Perl version back to 5.004, including a random collection similarly weird combinations (5.004 VMS, that 5.6.0 from RedHat 7, 5.6.1 on Windows 95). Then make sure the code is so clean and complete you'll never need add another lines of code or tweak the docs. Then lets think about adding a new compulsory near-core dependency. Adam K Tyler MacDonald wrote: Chris Dolan [EMAIL PROTECTED] wrote: There is a class of tests that module authors perform that end users are not expected to run. For example code coverage tests, spelling tests, coding style tests, etc. These tests are either prohibitively expensive or complicated or unpredictable for end users to run. I call these private tests or author tests. I really like this idea. But as you pointed out, it's not just authors that need to worry about running these tests, it's packagers (ppm/deb/etc), automated testers (cpants/testers.cpan.org/etc), and hackers. I'd suggest we call these exhaustive tests. Spelling tests, code coverage tests, testing interaction with CPAN.pm, etc. are things that I think should be done by automated testers, and by people packaging your module for distribution inside another larger system (eg, PPMs or debian packages). So maybe it would make sense if there were certain tests that only run if AUTOMATED_TESTING is set, and/or if the disttest action is being run. That said, it would be easy to whip together a Test:: module that only allows these tests to run under these conditions, especially if Module::Build, say, set a DIST_TEST environment variable when the disttest action was being run. I already check for AUTOMATED_TESTING to manipulate my test environments to be more exhaustive. Both Module::Build, and perhaps future versions of ExtUtils::MakeMaker (if there will ever be a future version) could depend on this Test:: module (whether or not they use it themselves) so that it's already on everybody's system. It'd be nice and small so I don't see anybody complaining too loudly. Maybe we could call it Test::Exhaustive? It's mandate being to SKIP_ALL of a test if you havent somehow (AUTOMATED_TESTING, disttest, etc) made it clear that you wish to exhaustively test this module? Roughly, I propose that we adopt a standard file extension for author tests, like t/*.ta, and add an ACTION_authortest to M::B that runs both t/*.t and t/*.ta (which ones first??) If they were named differently (i don't think they have to be), I'd want to see them run in lexical order like the current tests, so it'd go 01foo.t, 02bar.ta, 03baz.t. Cheers, Tyler
