Re: page - wiki - wiki - wiki
Ok, perl-qa.yi.org redirects there now and it looks like everything's working... Michael G Schwern [EMAIL PROTECTED] wrote: Andy Lester wrote: On Aug 1, 2007, at 5:09 PM, Michael G Schwern wrote: http://qa.perl.org/ [2] links to: Ask can fix that. Ask? Oh wait, qa.perl.org is mine. What do you want I should link to instead? http://perl-qa.hexten.net/wiki please. --
Re: planning at the end
Michael Peters [EMAIL PROTECTED] wrote: use Test::More tests = 'defer'; and then plan past_tests = $n; What benefit would that give? plan() is nice because it provides protection against you test script exiting prematurely. stating the obvious The exact same benefit as doing a 'plan' at the beginning, except this would work for scripts that don't know how many tests they are going to run in advance. If the calculated result used in the 'plan' at the end does not match the number of tests actually run, then you know you've got a problem with your test code. /stating the obvious I think that'd be a nice feature to have. - Tyler
Re: planning at the end
Michael Peters [EMAIL PROTECTED] wrote:
If the calculated result used in the
'plan' at the end does not match the number of tests actually run, then you
know you've got a problem with your test code.
/stating the obvious
So this gives you protection against not being able to count?
Exactly. :-) Take a look at, for instance, IPC::Run's test cases...
an array of sub{} blocks, some of which get skipped over on certain OS'es.
It'd be really easy to accidentally put to ok()'s in one sub{} block and
screw up the count.
- Tyler
Re: Volunteer wanted: We need a new wiki.
Michael G Schwern [EMAIL PROTECTED] wrote: These TAP extension proposals and designs for parsers and questions for details about the TAP grammar... they should all go into the Wiki. But the Perl QA Wiki sucks. Its slow. Its spammed. UseModWiki sucks. And I don't have the time to maintain it. We need a new wiki. schwern.org is too anemic to run anything serious (64 megs of RAM, woo!). We need a volunteer with server space to setup and maintain the Perl QA Wiki. I'd prefer MediaWiki (that which Wikipedia uses). It seems the best known, best of breed with the least fuss. Please contact me if you would like to do this. I'd be happy to. yi.org is already running a few mediawikis, as well as cpants.perl.org. Let me know and I'll get it set up. Cheers, Tyler
Re: Volunteer wanted: We need a new wiki.
Michael G Schwern [EMAIL PROTECTED] wrote: On 7/5/06, Tyler MacDonald [EMAIL PROTECTED] wrote: I'd be happy to. yi.org is already running a few mediawikis, as well as cpants.perl.org. Let me know and I'll get it set up. You win! Set it up. Let us know when its online. We'll worry about getting a proper domain/uri for it later. Done! http://qa.yi.org/ - Tyler
Re: Volunteer wanted: We need a new wiki.
Michael G Schwern [EMAIL PROTECTED] wrote: You win! Set it up. Let us know when its online. We'll worry about getting a proper domain/uri for it later. Done! http://qa.yi.org/ Thanks! Could you switch that to perl-qa.yi.org? Just to make it clear this isn't yi.org's QA department. yi.org's QA department... I almost snorted me tea out of my nose when I read that. ;-) OK, I've set perl-qa.yi.org to point there. I figured once it was ready for production you'd point a more official perly URL at it. (If/when you want to do that, please let me know so I can update apache etc...) Cheers, Tyler
Re: The new wiki
Michael G Schwern [EMAIL PROTECTED] wrote: Thanks to Tyler MacDonald and yi.org we now have a brand spanking new wiki! http://perl-qa.yi.org/ is its location, we'll worry about getting more official domains later. Since I set up my own server for (then nx, now yi).org 8 years ago and started doing the dynamic DNS thing, my hope was to be able to give something back to the hackers in the open source community that have already given me more than a lifetime's worth of value. Perl/CPAN in particular has made my life both more enjoyable and profitable. I'm glad that the server is fulfilling it's use; thanks to you guys for doing what you're doing! :) - Tyler
Re: Non-Perl TAP implementations
Ovid [EMAIL PROTECTED] wrote: Tracking the results in an object is a better choice than scraping from a print buffer. One of the frustrating issues with Perl's testing tools is the limited flexibility we have due to reading the output from STDOUT. I like that aspect about TAP... it's simple, humans and machines can read it easily, and there's no reason why it has to be langauge-specific. I think the only thing that could make it better would be if TAP output was parseable by YAML. :-) we should at least publish an EBNF grammar for the output. Definately! - Tyler
Re: Module requirements (was: Module::Build and installing in non-standardlocations)
Sébastien Aperghis-Tramoni [EMAIL PROTECTED] wrote: OTOH, who still runs pre-5.8.x code deserves what they get. There are horrible bugs in older Perls, and I don't know why people still insist using insecure, buggy and feature-lacking code like 5.6. or even gasp 5.004. Just think Unicode support, hash randomization, memory leaks. FWIW I've heard at least one valid argument for sticking with 5.6, and that actually is unicode support. :-) Apparently, in 5.6 regular expressions don't have full unicode support. But in 5.8 the regexp unicode stuff is expensive. And to make matters worse, if anything in any package that you import uses unicode, the new, slower unicode regular expression algorithm is applied to *everything* and there's no easy way to turn that off. This is just a nasty rumour I'm sure, but I've heard it multiple times from multiple people. I like to stay on the cutting edge and if my regexps are too slow, I'll find some other way to do it. :-) Cheers, Tyler
Apache::Test - cpan.testers weirdness
I just got some cpan testers reports on a new module, CGI::JSONRPC. 1 pass, 2 failures. One of the failures seems to be my fault, but the other one seems really odd: http://www.nntp.perl.org/group/perl.cpan.testers/298838 The odd part is here: [ERROR] [Wed Mar 15 10:06:14 2006] MAKE TEST failed: No such file or directory PERL_DL_NONLAZY=1 /usr/local/perl-5.8.5/bin/perl -MExtUtils::Command::MM -e test_harness(0, 'blib/lib', 'blib/arch') t/*.t t/Apache2-JSONRPC[ error] STDIN is not attached to tty, skip interactive config [ error] Skipping the test suite execution, while returning success status # No tests run! dubious Test returned status 255 (wstat 65280, 0xff00) DIED. FAILED tests 1-6 Failed 6/6 tests, 0.00% okay In the same breath it looks like the Apache::Test suite both says that it's going to return success, and returns a failure code. The really odd thing is that this only happens in Apache2-JSONRPC.t: http://search.cpan.org/src/CRAKRJACK/CGI-JSONRPC-0.01/t/Apache2-JSONRPC.t While CGI-JSONRPC.t skips its tests as well, and returns a correct result code: http://search.cpan.org/src/CRAKRJACK/CGI-JSONRPC-0.01/t/CGI-JSONRPC.t Does anybody know what's happening here? Thanks, Tyler
Re: Best Practice for testing compilation of scripts
Geoffrey Young [EMAIL PROTECTED] wrote: I was suggesting the functionality be added to Test::More as compile_ok(), rather than runperl() in some separate CPAN module, as it seems to closely parallel use_ok() for modules and would be rather useful on a larger scale. I agree, a well written version of that would be awesome in Test::More. Cheers, Tyler
Re: New kwalitee metric - eg/ directory
Steve Peters [EMAIL PROTECTED] wrote: /eg scripts are a nice hands-on way of finding out how a module works in real life. No distribution should be without one! Unless, of course, it has an examples/ directory, which would cause the kwalitee test to fail. ;) I do think its a good idea, especially with large, all-encompassing type modules to provide some examples, but testing for (in effect, regulating) the name of the directory that will be difficult to do. I'm not convinced this is a good idea, but if it were implemented, I don't think a regexp to catch all of them would be that complicated... Well so far the only ones I've seen are eg, examples, and from that renegade GD::Graph, samples. If we were to actually use this as a metric, I'd suggest negating it if there is a bin directory, since applications may not need examples... Cheers, Tyler
Re: Activestate and Scalar-List-Utils
David Golden [EMAIL PROTECTED] wrote: So back at the beginning of February, there was some email traffic about how ActiveState's automated PPM build system was using an outdated version of Scalar-List-Utils, which was causing a cascading prerequisite failure for many distributions. Has anyone heard any updates on this? Does anyone have an inside contact at ActiveState that can shed some more light on the subject? The problem's bigger than that and has to do with ActiveState's QA process, dual-life packages, and PPM. From what I gather a few solutions have been proposed and the perl hackers at ActiveState are looking into it, but they're a bit distracted right now by all the extra work that comes from severing themselves from Sophos; moving offices, splitting up the IT infrastructure, etc. After that, un-borking modules like Scalar::List::Utils and Spiffy are a top priority. This is all just gossip; I work at Sophos and occasionally run into an Activator on the patio, and I also have several modules who fail under PPM but succeed elsewhere. Here's a weird one: http://ppm.activestate.com/BuildStatus/5.8-windows/windows-5.8/DBIx-Transaction-0.007.txt That build seems to hang on the first ExtUtils::MakeMaker::prompt(), even though ActiveStates cpanrun sets AUTOMATED_TESTING and PERL_MM_USE_DEFAULT. Cheers, Tyler
Re: Trends in Code Quality
Matisse Enzer [EMAIL PROTECTED] wrote: Jeff I think that is a good theory - I mean, it is a testable theory. I hope it is true, but I am not sure. I suggest you interview a few IT managers - come up with a list of 6 questions and ask them to answer in email - I can introduce you to a couple outside of the company you are at now, and I bet other people can to - getting answers from a dozen different IT managers would add useful information. And consider the different environments - small company, large company, fast-moving, slow-moving, etc. Ooooh, a testable theory! When you do this, can you post the results as Test::Harness output so we can calculate the Kwalitee of the theory? Something like ok 1 - IT Managers like code quality not ok 2 - Programmers are lazy etc... Thanks, Tyler
Re: Best practice for version control of locally installed CPAN modules
Hello Matisse, I like these two ideas: Matisse Enzer [EMAIL PROTECTED] wrote: 2) Put the CPAN .tar.gz files in a local CPAN repository and use CPAN::Site to install - that way we *only* get the versions in our local CPAN repository and dependencies are managed by the module Makefile.PL / Build.PL scripts. 3) Put the .tar.gz files in our source-code control system, and checkout and build each one during a release process. #3 seems like the lowest maintenence. Maintaining on a .tar.gz in CVS seems easier than integrating diffs of the newest version whenever you want to upgrade. #2 has it's benefits too - you could even mirror all of CPAN, and just maintain a script with install commands to install the versions you want; install KWILLIAMS/Module-Build-0.27_04.tar.gz; etc. That makes both upgrading easy and makes your build process independant from the availablity of your favourite CPAN mirror. Cheers, Tyler
META.yml feature for autotesters?
I've been thinking about automated testing again. I know this is a bad habit and I should stop it and just get on with my work, but here's where I'm at: Sometimes it's beneficial for an automated tester to install additional packages (in software I'm releasing, Test::CPANpm and sqlite are perfect examples). It would be good to have a standard way to tell a smoke tester what these packages are. How does everybody feel about making this a defined feature in the META.yml spec? Something like: optional_features: - automated_testing: description: Automated testing of all of this package's features requires: DBD::SQLite2: 0 If an optional_feature field for packages useful for automated testing was standardized, functionality could be added to CPAN::YACSmoke and PITA to take advantage of it, and then there would be an easy way for an author to define prerequisites for doing an exhaustive smoke test of their distribution. What do y'all think? Thanks, Tyler
Re: META.yml feature for autotesters?
Adam Kennedy [EMAIL PROTECTED] wrote: So while we are on the subject of META.yml, I think the dynamic_config approach is horrible, because it defaults to an efficient error case and relies on the author to fix the error, rather than defaulting to the inefficient correct case, and giving the option to make it faster. Where would be the best place to get this flag changed (or for a discussion on doing so) in the spec? The module-build project, maintained by Ken Williams, is currently hosting the META.yml spec; http://module-build.sourceforge.net/ I guess META.yml's reccomends feature accomplishes most of what I'm after (if you can't install the module don't worry about it, but if you can, it'll enhance things), but what I was looking for was a place to dump stuff like that that doesn't affect the usability of the package, just the testability. build_recommends was suggested on module-build-general, but it appears to have disappeared from the Module::Build code and isn't on the META.yml spec. - Tyler
Re: Request for Comments: Package testing results analysis, result codes
Adam Kennedy [EMAIL PROTECTED] wrote: Firstly is that it might turn an otherwise normal result into something else, with no clear rule. It makes a judgement call that some level of testing is good or bad, which isn't really the place of an installer to call. The reason Kwalitee has metrics like this is that it's not important in the scheme of things, it's only looking for indicators which may well be wrong (hence the name Kwalitee). The Kwalitee of a module does not prevent it being installed. What makes 79 skips different from 80 skips? You need some clear distinction between the two states, not just something arbitrary (be it 50 or 80 or something else). I was speaking in percentages, not actual numbers, but I do see your point: part of the problem is that while we can send a message to the screen about why a test was skipped, there's no convention for making a *computer* understand why. If there were flags or tags that were standard convention (eg; need_config, missing_module, etc) then this would be a lot easier. Now 100% skips, THAT could potentially be interesting, or maybe TODOs. But then I don't necesarily know why it would be worthy of a different result code. Is there metadata stored apart from these result codes? If so it might be useful to just store the statistics on skips. Assuming this database was searchable, it'd be easy to identify modules that could be tested better by, say, sorting by % of tests skipped and then looking at the top curplits' log output manually. That actually brings up another point: Currently testers.cpan.org only provides verbose log output for failures. In cases like this, (or cases where you're driving yourself mad trying to understand why platform X always passes whereas platform Y which doesn't seem too different always fails), it'd be handy for the module author to have access to full logs of the test output for both passes *and* fails. The second issue is that it has the potential to override the package author. If the author felt that the tests weren't critical, and could be skipped quite happily, who are we to make a call that he is wrong and that it's a probably, or anything other than 100% ok. I'd still like such a thing to be visible in some way. Of course you're going to happily skip tests that require a database if you don't have DBI_DSN set. (I am toying with the idea of marshalling a whitebox for the purpose of doing CPAN testers specifically for database and mod_perl-based packages... that'll probably be a project in itself, hacking YACsmoke to only pick up those packages, add -mysql or whatever to the uname that it posts with, etc. :) Cheers, Tyler
Re: Request for Comments: Package testing results analysis, result codes
Adam Kennedy [EMAIL PROTECTED] wrote: I'd still like such a thing to be visible in some way. Of course you're going to happily skip tests that require a database if you don't have DBI_DSN set. Not necesarily... it all depends on how important it is to you. I see some potential cases where you'd rather abort the install if you can't do proper testing. Fair enough. So both uses cases exist and it sounds like you're gearing PITA to be able to handle them in some fashion. I'm looking forward to seeing it. :) Cheers, Tyler
Re: Request for Comments: Package testing results analysis, result codes
Adam, I have one more edgey case I'd like to see on this list: 13. Tests exist, but fail to be executed. There is tests, but the tests themselves aren't failing. It's the build-process that is failing. 14. Tests run, and some/all tests fail. The normal FAIL case due to test failures. 15. Tests run, but hang. Covers non-skippable interactive question. Covers infinite attempts to connect network sockets. Covers other such cases, if detectable. Tests run, but 50% (or maybe 80%?) are skipped. From what I've seen, the most common cause of this is that the package is untestable with the current build configuration. Eg; you needed to specify a webserver or database or something to get these tests to run. Apache::Test provides some hooks for autotesters to configure themselves to test packages using it, IMHO setting DBI_DSN etc should be enough for packages that test against a database. I've been thinking a lot about how to properly autotest database-driven packages lately. So far the best I've got is: - If a package depends on a particular DBD:: driver, and you have the matching database server available, specify a DBI_DSN that uses that driver. - If a package depends on DBI, but no particular driver, try testing it with every driver you can use, starting with SQLite since it does not need a live database server. In the case where a package supports multiple, but not all, database driver backends, they would probably depend on DBI, but reccomend each DBD:: driver in the META.yml, which would be covered by the first bullet. Cheers, Tyler
Re: Binary distributions
Offer Kaye [EMAIL PROTECTED] wrote: Just an example, IO::All [1] version 0.33 has been available since Dec 17, 2004. It passed testing many times, at least according to its testers page [2]. My default 5.8.7 ActivePerl distribution lists IO::All version 0.17 . According to http://ppm.activestate.com/BuildStatus/5.8-windows/windows-5.8/IO-All-0.33.txt it is not being built because Spiffy failed it's tests. According to Spiffy's log, *it* did not build becuase Scalar-List-Utils failed to build... Which is the same module that both Adam Kennedy and David Golden are complaining about. (I wonder how many modules are being held back over this one bug?) I'm not sure which of the reasons you listed are valid, as a user I have no real way of knowing... that's why I thought an automated Windows-PPM sub-domain on CPAN which would be updated automatically for every new package would be great. What you are saying, that ActiveState already have such a system, is news to me, especially in view of examples such as the above... I would rather use CPAN, but the PPM works fine for me, only it's not as updated as CPAN. I think it's a combination of too much to do, too little time and the QA philosophy of ActiveState. On ppm.activestate.com, if the unit tests fail, you don't get a PPM; I guess this protects ActivePerl users from installing modules that won't work properly on their install. ppm.cpan.org would be a good idea and probably not too hard to implement. What would it's mandate be? Provide every package out there, even if unit tests fail, so long as it acutally built? Who is going to provide the icky sticky windows box with VC++-compiled versions of all the libraries people expect to be able to use with perl (libgd, libmysqlclient, apache2, etc)? I certainly don't want to lay my hands on any windows box. ;-) I'd like to clarify something - I don't really expect Gozer to handle this problem... or a problem with any other module... there are so many, how can a single person keep up? It would be better if the system were automated... and worked :) I think if the module is close to the metal and a lot of other modules depend on it, it would be worth ActiveState's time to investigate the problem and either fix the build system or fix the module. It's a big payoff if fixing one module means that all of a sudden another 30 or so automagically build. I would rather not use 2 seperate package installation schemes... either CPAN or PPM (or something else), not both. I think those two schemes are exactly what we need for perl. It's analogous to building and installing a source package, or just installing the latest RPM/.deb/whatever. - Tyler
Re: Binary distributions
Offer Kaye [EMAIL PROTECTED] wrote: Phillipe Chaisson aka Gozer (one of the mod_perl authors) is responsible for the ActiveState PPM repositories now, Hi Gozer, nice to meet you. Gratz on ActiveState's move to a new company, good luck :) It's a Good Thing for all companies involved, but as a developer for Sophos, I'll certainly miss having them upstairs. (Not to mention the @activestate.com email addy ;-) - Tyler
Re: IPC::Run or something else?
Tamas Dober [EMAIL PROTECTED] wrote:
Hello,
I'm a beginner, please forgive me b/c of the simple questions.
I'd like to test that that a batch file (starting a Java app) gives me the
expected output or not.
I really like IPC::Run, but it is a bit of a heavy package to make
users install just for your test cases. If your module is already using it
for something else, go for it! Otherwise i'd suggest just opening a pipe if
you don't need bidirectional communication;
open(my $fh, '-|', @ls);
my $out = join('', $fh);
...
Cheers,
Tyler
use warnings;
use IPC::Run 'run';
my $out;
my @ls = ( '\/perl\/Feb\/packager\/bin\/package.bat' ) ;
run [EMAIL PROTECTED], \undef, \$out or die bat returned $? ;
like($out, qr/Usage/, 'Usage message' );
is( $out =~ /config/, 'this is like that');
--
The outcome:
C:\perl\Feb\packager\binprove -v Run2.t
Run21..2
Usage:
-config[Configuration XML URL]
-request [Request XML URL]
--help
not ok 1 - Usage message
# Failed test 'Usage message'
# in Run2.t at line 8.
# '
#
#
# '
# doesn't match '(?-xism:Usage)'
not ok 2
# Failed test in Run2.t at line 9.
# got: ''
# expected: 'this is like that'
# Looks like you failed 2 tests of 2.
It seems I couldn't redirect the output.
Maybe IPC::Run is not a good choice?
If I try it under LINUX (using package.sh instead of package.bat) I'm having
the same issue.
Could you please help me what direction I should go, what module should I use?
Thank you
Tamas
-
Yahoo! Mail - Helps protect you from nasty viruses.
Re: Binary distributions
Offer Kaye [EMAIL PROTECTED] wrote: Why not start off by providing ppm.cpan.org (as the OP suggested for linux distors), or something similar? There are many modules that I want to use where the PPM version provided by ActiveState or some other repository is badly of out date.. ActiveState always serves the last available version where all tests passed on a given platform. It attempts to build and test every package on CPAN at least once a week. If something isn't available, it means the tests failed, which could mean: 1) Tests are failing because it's windows 2) Tests are failing because they're failing everywhere else as well 3) Tests are failing because of ActiveState's build system For 3), that could mean because the build system itself is screwed or they don't have some library available. Phillipe Chaisson aka Gozer (one of the mod_perl authors) is responsible for the ActiveState PPM repositories now, and the he's told me that if there's an issue with the build system that causes some particular module's build to fail when it shouldn't, he'll fix it right away if it can be identified. http://ppm.activestate.com/ contains full logs of every single package build success and failure, so if a package isn't available, have a look there and see why/how it failed and let him know. Cheers, Tyler
Re: [Module::Build] [RFC] author tests
Adam Kennedy [EMAIL PROTECTED] wrote: Write this up. Then exhaustively test it on every single Perl platform (50ish?) and every Perl version back to 5.004, including a random collection similarly weird combinations (5.004 VMS, that 5.6.0 from RedHat 7, 5.6.1 on Windows 95). I let testers.cpan.org and ppm.activestate.com take care of that. Then make sure the code is so clean and complete you'll never need add another lines of code or tweak the docs. That's difficult, which is why nearly every core perl module has a newer version available on CPAN... Then lets think about adding a new compulsory near-core dependency. Ok, so Chris and I were getting a bit too excited about the idea. :) A new module doesn't need to be added to the core, so long as there is a way that we can reliably detect when a person wishes to build and test any given perl package for an objectively unselfish purpose such as 1:prepackaging, 2:automated testing, or 3:releasing. All three are viral so it's best to make sure they do no harm, while still maintaining some level of convenience for the end user. There's already AUTOMATED_TESTING for 2 and make disttest for #3 (just keep a file around in your repo that's not in your MANIFEST and test for it's presence.) Any good way to detect #1? And now that I think about it, I'm not so convinced about that whole concenience for the end user nonsense. If they're mucking about installing perl modules from the CPAN packages by themself, they're probably developers that need some extra time to sit there and think about what sort of madness they're getting themselves into anyway. - Tyler
Re: [Module::Build] [RFC] author tests
Steffen Mueller [EMAIL PROTECTED] wrote: And now that I think about it, I'm not so convinced about that whole concenience for the end user nonsense. If they're mucking about installing perl modules from the CPAN packages by themself, they're probably developers that need some extra time to sit there and think about what sort of madness they're getting themselves into anyway. I strongly disagree! CPAN is the preferred source for any Perl modules and the associated tools like CPAN.pm or CPANPLUS are the preferred method of installing them. Why? Because since you are not going to see all modules in .deb or whatever packages and not in the most recent versions. Using both the system's package system and CPAN's for module installation is going to bite you on most systems some day. Steffen, I more meant the madness they are getting themselves into by programming in perl in the first place. :-) - Tyler
Re: [Module::Build] [RFC] author tests
Chris Dolan [EMAIL PROTECTED] wrote: * copyright.t - Ensures that there is a Copyright .([localtime]- [5]+1900) somewhere in every .pm file. Will break 11 months from now. * distribution.t - Relies on Test::Distribution, which is not in my prereq list * perlcritic.t - Runs Test::Perl::Critic on all .pm files. Will fail without my specific $HOME/.perlcriticrc and will fail with future, more exhaustive versions of P::C * spelling.t - Runs Test::Spelling. Will fail without my custom dictionary * versionsync.t - Checks that the $VERSION is the same in all bin/* and *.pm files. This test is pointless after release, since it's already been tested before release * pod.t - Checks POD validity. This test is pointless after release, since it's already been tested before release * pod-coverage.t - Checks POD completeness. This test is pointless after release, since it's already been tested before release and one I have not yet employed: * coverage.t - Ensures that Devel::Cover totals are higher than some threshold Wow, you really *are* exhaustive. How do you find the time to write any code? ;-) Now that I understand exactly what you mean by author tests, here's what I think: Whatever convention you're using, if these tests are only going to work on your system, then they definately shouldn't be in t. And since there's absolutely no value in these types of tests for anybody else except the module author, there's no real point in having a convention, just stick 'em anywhere that the make/buildfiles will ignore them. - Tyler
Re: [Module::Build] [RFC] author tests
A. Pagaltzis [EMAIL PROTECTED] wrote: I was just gonna say. It???s pointless for anyone but the author to check POD or test coverage. I agree about the POD coverage. But if I got a different level of code coverage on somebody else's system than my own? I'd be very interested in finding out why. I was actually considering playing with YACSmoke and Devel::Cover to make a site that shows the code coverage metrics for all of CPAN, but then I realised that I'm getting far too obsessed with this testing stuff and I need to focus on writing real code if I ever want to finish an application. :-) - Tyler
Re: [Module::Build] [RFC] author tests
Chris Dolan [EMAIL PROTECTED] wrote: There is a class of tests that module authors perform that end users are not expected to run. For example code coverage tests, spelling tests, coding style tests, etc. These tests are either prohibitively expensive or complicated or unpredictable for end users to run. I call these private tests or author tests. I really like this idea. But as you pointed out, it's not just authors that need to worry about running these tests, it's packagers (ppm/deb/etc), automated testers (cpants/testers.cpan.org/etc), and hackers. I'd suggest we call these exhaustive tests. Spelling tests, code coverage tests, testing interaction with CPAN.pm, etc. are things that I think should be done by automated testers, and by people packaging your module for distribution inside another larger system (eg, PPMs or debian packages). So maybe it would make sense if there were certain tests that only run if AUTOMATED_TESTING is set, and/or if the disttest action is being run. That said, it would be easy to whip together a Test:: module that only allows these tests to run under these conditions, especially if Module::Build, say, set a DIST_TEST environment variable when the disttest action was being run. I already check for AUTOMATED_TESTING to manipulate my test environments to be more exhaustive. Both Module::Build, and perhaps future versions of ExtUtils::MakeMaker (if there will ever be a future version) could depend on this Test:: module (whether or not they use it themselves) so that it's already on everybody's system. It'd be nice and small so I don't see anybody complaining too loudly. Maybe we could call it Test::Exhaustive? It's mandate being to SKIP_ALL of a test if you havent somehow (AUTOMATED_TESTING, disttest, etc) made it clear that you wish to exhaustively test this module? Roughly, I propose that we adopt a standard file extension for author tests, like t/*.ta, and add an ACTION_authortest to M::B that runs both t/*.t and t/*.ta (which ones first??) If they were named differently (i don't think they have to be), I'd want to see them run in lexical order like the current tests, so it'd go 01foo.t, 02bar.ta, 03baz.t. Cheers, Tyler
Re: testers.cpan.org out of sync with search.cpan.org?
Leon Brocard [EMAIL PROTECTED] wrote: Usually this clears up in about a day, but in some cases it's been 3 or 4 days now and search.cpan.org is telling me that tests have run, but testers.cpan.org doesn't seem to know anything about them. Sorry, I'll prod testers.cpan.org again. Give it a while to sync up. Awesome, thanks! Is the code that drives testers.cpan.org open source? I'd really like to learn more about it. Cheers, Tyler
bug with Test::Exception? or imacat's autotest box?
Take a look at this output: http://www.nntp.perl.org/group/perl.cpan.testers/285112 It looks like this particular system is not noticing that Test::Exception requires Sub::Uplevel, then gets confused thinking it was *my* module that needed Sub::Uplevel. What's even more concerning is the presence of line noise right after the make test FAILED... Any idea what can be going on here? Thanks, Tyler
Re: bug with Test::Exception? or imacat's autotest box?
Fergal Daly [EMAIL PROTECTED] wrote: I have a fail against a module for exactly the same reason. I initially blamed Module::Build but they convinced me it was Imacat's setup. Apparently the output looks like an old version of something or other. http://rt.cpan.org/NoAuth/Bug.html?id=15034 has details. Imacat didn't respond to my email at the time, That sucks. :-( If imacat's box has gone AWOL, is there anything cpan testers can do to flag it as such? At least until his/her attention is grabbed and the problem is addressed? Thanks, Tyler
Re: Kwalitee in your dependencies (was CPAN Upload: etc etc)
OK, speaking of Kwalitee, I saw cpants for the first time today. And saw that it claims to update every sunday, but there hasn't been an update since december 5th. I also saw this interesting .pm file that appeared to have an anonymous hash of every tarball in CPAN in it, all on one line. That nearly crashed my browser. :-) What's up with CPANTS? Is it a defunct project? - Tyler
Re: bug with Test::Exception? or imacat's autotest box?
imacat [EMAIL PROTECTED] wrote: That sucks. :-( If imacat's box has gone AWOL, is there anything cpan testers can do to flag it as such? At least until his/her attention is grabbed and the problem is addressed? I'll look into it and return ASAP. It's Chinese New Year here, so give me a break and allow some delay, OK? You could write to me again if I do not return for some certain period. I wouldn't have suggested that unless fergal had indicated that it was a long-standing issue. Now that I know you're aware of the problem and looking into it, I thank you and will patiently await your reply. :) Happy year of the dog! - Tyler
Re: bug with Test::Exception? or imacat's autotest box?
Greg Matheson [EMAIL PROTECTED] wrote: On Tue, 31 Jan 2006, Tyler MacDonald wrote: Take a look at this output: http://www.nntp.perl.org/group/perl.cpan.testers/285112 It looks like this particular system is not noticing that Test::Exception requires Sub::Uplevel, then gets confused thinking it was *my* module that needed Sub::Uplevel. What's even more concerning is the presence of line noise right after the make test FAILED... Any idea what can be going on here? It's a NLS error message. In Big 5, it says. 'This file or directory does not exist.' Ahh, in my firefox it looked like a bunch of copyright symbols and squiggly lines. Guess I've got a few fonts to install. :) The weird thing is, on most chinese language webpages, I get little squares with the 2-byte hex code for the appropriate character written inside... - Tyler
Re: Binary distributions
Andreas J. Koenig [EMAIL PROTECTED] wrote: FWIW, we're using dh-make-perl to create debian packages from CPAN modules. Andreas, I've used this tool a few times when a CPAN module wasn't already available in unstable/main, but I havent looked into it too closely. I'm curious, does it do anything about .so's that XS modules need, or build vs. runtime dependancies in environments that support it (Module::Build, etc)? Thanks, Tyler
Re: Binary distributions
Barbie [EMAIL PROTECTED] wrote: I'm one of the maintainers/developers for CPAN::YACSmoke. I was intrigued by your post about adding a Packager plugin to it. However, I'm unclear as to what purpose it would serve. CPAN::YACSmoke is purely about reporting on test results. CPANPLUS does the actual work. Barbie, From what I gather, CPANPLUS is a linear package building system, whereas YACsmoke is a wrapper around that that tries to build as many packages as is humanly (er, computerly) possible on a system, with the side effect of sending it's results through Test::Reporter. YACsmoke also can maintain a database of what it has built and what it hasn't, allowing a YACsmoke system to eventually exhaustively build every single package on CPAN without building the same one twice. Is this correct? What would the plugins for .deb, .rpm and .ppm do? Currently we just pass the path to CPANPLUS and let CPANPLUS unwrap, make and test the distribution. We then just interrogate the results. These plugins would execute *after* YACsmoke/CPANPLUS has done it's work, but before it's cleaned up after itself, and generate the actual .deb/.rpm/.ppm files. Perhaps this is something you meant for CPANPLUS to handle, in which case I'm sure that's something Jos would be interested in, as he has previously looked at some kind of binary support. Well, maybe the actual low-level package generation could be kicked down a level, but the whole reason I thought of using YACsmoke for this was because of it's grab everything and try to build it nature. I plan to find time soon to complete the tests required for the next release of CPAN::YACSmoke, so if what you want does relate to CPAN::YACSmoke I can start taking a look and see what needs to be done to implement it. Sounds good :) Thanks, Tyler
Re: Binary distributions
Gabor Szabo [EMAIL PROTECTED] wrote: You simple cannot guess what libraries/compiler/system/kernel the user has installed, unless you know the distribution and version *and* require that the user never updates anything. I think I agree. That's what I would like to see solved. If I stick to the standard apt-get (or whatever) of my distribution I would like to be able to get all the CPAN modules by saying # apt-get install Module::Name Well in debianland it's actually apt-get libmodule-name-perl :) The solution to this sounds straightforward, although probably not simple to implement. Your build box is running the distribution+arch you want to release the modules for. After building the packages, ldd any binaries found within. This will give you a list of library filenames. In debian's case, /var/lib/dpkg/info/*.list contains a list of files supplied by each package on the system. Use that to match dependancies up. If a package requires something on the system that it *doesn't* link against (say, a certain executable), that problem won't be solved, but I think that case is a bit rarer. - Tyler
testers.cpan.org out of sync with search.cpan.org?
On several of my modules, the search.cpan.org page shows test passes/failures, whereas when I follow the link to see the actual list, there are no tests reported. eg: http://search.cpan.org/~CRAKRJACK/Class-Driver-0.004/ Usually this clears up in about a day, but in some cases it's been 3 or 4 days now and search.cpan.org is telling me that tests have run, but testers.cpan.org doesn't seem to know anything about them. Is this a common problem? Is it fixable? Is there somewhere else I can go to find out about the results of my modules being tested, short of subscribing to the mailing list where *every* module tested gets posted? Thanks, Tyler
Re: Dependency trees was: CPAN Upload: D/DO/DOMM/Module-CPANTS-Analyse-0.5.tar.gz
Chris Dolan [EMAIL PROTECTED] wrote: * for windows only * only includes Foo-Bar, but not it's dependecies It will auto-install dependencies just like CPAN, I believe. And, yes, it's currently Windows-only. Didn't you offer bonus points for Windows?? Um, no it isn't! http://ppm.activestate.com/ PPMs are built for a variety of platforms, windows, OSX, and various unixes. You can download ActivePerl for these platforms here: http://www.activestate.com/Products/Download/Download.plex?id=ActivePerl - Tyler
Re: Dependency trees was: CPAN Upload: D/DO/DOMM/Module-CPANTS-Analyse-0.5.tar.gz
Jeffrey Thalhammer [EMAIL PROTECTED] wrote: Randy Kobes distributes Win32 PPMs for some of the modules that ActiveState doesn't provide. It is not entirely automated, so the latest code isn't always available. But Randy is very helpful if there's anything you want to see. http://theoryx5.uwinnipeg.ca/ppms/ What is actually happening on ppm.activestate.com, is that only modules that pass all their unit tests are packaged for the general public. IMHO this is a great idea, since then people who are ppm installing stuff from activestate repoes can be reasonably(*) confident that the package will work on their system. Part of the problem is that a lot of modules out there are fully functional even when a few of their tests fail due to assumptions about the environment they are being tested in. Another part is that the ActiveState perl package build process (cpanrun) doesn't behave exactly the same way as CPAN::YACSmoke. So, a lot of packages that build successfully for CPAN testers don't for ppm.activestate.com (and sometimes, the opposite is true). Gozer (the new ActiveState CPAN-PPM guru) is working hard to get as many of these failing modules working properly ASAP. In the past few weeks he's implemented the PERL_MM_USE_DEFAULT and AUTOMATED_TESTING environment variables in cpanrun. He's also working on getting as many non-perl dependancies as possible (eg; libgd, libpg, etc) installed on his build boxes so that XS-based packages that depend on these things can be built. A lot of packages still dont get released through activeperl, but the situation is getting better. There's still always going to be packages that fail unit tests, that people want anyways; I think keeping those packages out of a quality assured repo is a neccessary sacrifice to maintain integrity. Maybe it would be a good idea for there to be an official unstable PPM repo where packages that built, but failed unit tests, get placed -- then somebody who wants to be on the bleeding edge can add that repo to their list to get at the packages, and maybe even lend a hand in figuring out why the tests are failing. Cheers, Tyler
Re: Binary distributions
Gabor Szabo [EMAIL PROTECTED] wrote: I have just moved to Ubuntu and thought I will try to rely on apt-get to install my Perl modules. Quckly I hit a wall and could not install some of the basic modules. I did not have the time to investigate and check if I made a mistake or if there is a .deb repository with the latest CPAN modules for Ubuntu. I reverted to use CPAN.pm. BTW here is an article on how to build Debian packages of Perl modules: http://www.debian-administration.org/articles/78 Anyway I think instead of trying to setup our own binary distribution we might want to make sure there are up to date repositories of Perl modules for the major distributions (and I am not talking only about Linux distributions here). It can be done by helping the people who already maintain some of these distributions or by setting up repositories such as debian.cpan.org, fedora.cpan.org, etc... That is such an incredibly good idea. I've got plenty of bandwidth to burn and I'm willing to set up debian.cpan.org. I think the most obvious way to automate this would be to take advantage of the whole perl package / dependancy / build / test process that the YACsmoke module already offers us. Maybe CPAN::YACSmoke::Plugin::Packager, with children ::Deb, ::RPM, ::PPM, etc. These modules could just stick their built packages into an outgoing directory (or maybe multiple, noarch, i386, etc); some distros would be able to just nab those and their metainfo and roll a repo out of it, maybe some we'll have to write tools to do that for as well. - Tyler
Re: Test Script Best-Practices
Jeffrey Thalhammer [EMAIL PROTECTED] wrote: * Should a test script have a shebang? What should it be? Any flags on that? It's not at all neccessary, but IMHO it is good form; it's a surefire way for anything else (HTTP server, IDEs, etc) to figure out that you're actually a perl script and do the right thing (run it, syntax highlight it, whatever). For test scripts and other executables, I use: #!/usr/bin/perl For .pm files, I just put #!perl * When run outside of 'make test', should the test script force modules to load from the distro's lib or blib directory by default? Or should it just load from the user's existing @INC (whatever it may be). The convention in running tests is to use the 'prove' command; prove t/01_class.t That should take care of blib for you. * Should a test script make any assumptions about the CWD? It's expected that a test script will be executed from the package's root directory, where the Makefile.PL etc reside. Is it fair/safe for the test script to change directories? Yes. Typically, each test script runs as it's own process, so if it changes directories, it only affects that one test file and not the others. It would probably be nice of the test script to only change to directories inside it's packages root, as opposed to snooping around the harddrive of whomever is doing the testing. * What's the best practice for testing-only libraries? Where do they go and how do you load them? Is there a naming convention that most people follow (e.g. t::Foo::Bar). I don't know. If there is that could be far more convienent than what I do. I usually use My::Package::Test::blah. - Tyler
how to detect that we're running under CPAN::Testers?
Hello, Is there any way to tell if my package is being tested automatically under CPAN::Testers? Here's the situation: I have a DBI extension (DBIx::Transaction) whose unit tests currently depend on a valid DSN being available. SQLite makes a good testbed for mock databases, so if DBD::SQLite2 is available, my module defaults to testing against a temporary database using that. Thus, DBIx-Transaction-0.04 depends on DBD::SQLite2 *unless* the DBI_DSN environment variable is set to a non-SQLite2 driver. If DBI_DSN is set to an empty string, the tests do not run at all. Some people think that in a perfect world, end users should not be made to install DBD::SQLite2 by default. I'm leaning towards that as well; less bloat is good. However, when these packages are being auto-tested as part of the perl QA effort (and possibly for packaging for use in ActivePerl on ppm.activestate.com as well), I'd like my Makefile.PL to notice this and depend on DBD::SQLite2 after all, so that the tests will be run and we have an assurance that the package works. Any thoughts/ideas? Thanks, Tyler
Re: how to detect that we're running under CPAN::Testers?
Sébastien Aperghis-Tramoni [EMAIL PROTECTED] wrote: For CPAN smokers based on CPAN::YACSmoke, the answer is: test the presence of the AUTOMATED_TESTING environment variable. See also the following page for more details: http://search.cpan.org/dist/CPAN-YACSmoke/lib/CPAN/YACSmoke/FAQ.pod Awesome, that's what I was after. :) Are there any other automated testing environments that post their results to testers.cpan.org I should worry about? Thanks, Tyler
