Re: Automated testing
On Sun, 30 Nov 2008, Bill Moseley wrote: On Sun, Nov 30, 2008 at 09:22:02AM +0200, Gabor Szabo wrote: I tired SmokeRunner::Multi with partial success. I got as far as installing it and didn't really get how to use it. But, by then I had a running shell script that does a svn export, builds what needs to be built, and runs tests. The script only runs when it detects a new revision in svn, and will set a flag on errors so that it can also report when all tests are passing again. Heh, that's _exactly_ what SmokeRunner::Multi will do. The only difference is that it makes it easy to add new repos, for example if you have multiple projects or branches to test, and it tries to test the ones most in need of testing first (longest time between last test run and most recent change). -dave /* http://VegGuide.org http://blog.urth.org Your guide to all that's veg House Absolute(ly Pointless) */
Re: Automated testing
On 29 Nov 2008, at 00:18, Bill Moseley wrote: Is there something that's designed for this job that I'm missing? You may want to investigate Smolder for gathering the test results. And parrot uses buildbot for continuous integration. There's also a scrappy script that Test::Harness uses - you can find it here: https://svn.hexten.net/tapx/trunk/smoke/ smoke.pl is a cron job - and there are some sample config files in there too. -- Andy Armstrong, Hexten
Re: Automated testing
On Sun, Nov 30, 2008 at 09:22:02AM +0200, Gabor Szabo wrote: I tired SmokeRunner::Multi with partial success. I got as far as installing it and didn't really get how to use it. But, by then I had a running shell script that does a svn export, builds what needs to be built, and runs tests. The script only runs when it detects a new revision in svn, and will set a flag on errors so that it can also report when all tests are passing again. make test returns an error if any tests fail. That makes it easy in the shell script to know when to report problems (including sending stderr output). One of the test writers used diag() to add comments in the tests so all those end up in any error report as they are always written to stderr, but it's not a huge problem. One thing I noticed is that Makefile.PL still exists *without* an error even if required modules are missing. Maybe I should just add a test that reads Makefile.PL and checks the required modules. Thanks for the suggestions. -- Bill Moseley [EMAIL PROTECTED] Sent from my iMutt
Re: Automated testing
On Sun, Nov 30, 2008 at 8:20 PM, Bill Moseley [EMAIL PROTECTED] wrote: One thing I noticed is that Makefile.PL still exists *without* an error even if required modules are missing. That's probably because Makefile.PL is conceived of as a thingy to make a makefile. A makefile just lists dependencies between targets. So if you buy into that design, the proper time to fail is later (likely during the 'test' phase) when the dependencies are needed. The Makefile.PL has properly done its job, it's created a proper makefile, so it doesn't exit with a failure code, and you have to wait until later (or parse the Makefile.PL output, or find the Magic Comment in the makefile and check the prereqs yourself) to find whether there are missing prereqs. As you point out, that's not very convenient; I think that's one illustration of why the design isn't an apt one. If you're also working with Module::Build distributions, note that Build.PL doesn't exit with a failure code in this situation either, but it does have at least 2 mechanisms for letting callers explicitly check dependencies: either look at the _build/prereqs file, or call the prereq_failures() method. -Ken
Re: Automated testing
On Sat, Nov 29, 2008 at 6:08 AM, Andy Lester [EMAIL PROTECTED] wrote: On Nov 28, 2008, at 9:55 PM, David Golden wrote: You might want to look at Smolder: http://www.slideshare.net/mpeters/smolder-introduction Smolder doesn't run tests. It only reports on the results. Look at http://search.cpan.org/~drolsky/SmokeRunner-Multi-0.14/ I was not the OP but I was glad to see this post as I just wanted to setup smoke testing on the Padre SVN repository. I tired SmokeRunner::Multi with partial success. http://www.perlmonks.org/?node_id=726817 In short it seems that currently it does not create correct TAP archives so I could not use it yet. I already sent a message to Dave, I hope he'll have time to check what might have gone wrong. For now I think buildbot http://buildbot.net/ is a better solution. One might integrate buildbot - that runs the full cycle - with Smolder that can hold the TAP result and display them in a nice way. Eventually I'd like to see a full integration of something that will do the build/test cycle and store all the data, both the output from the build commands and the TAP results. BTW PostgreSQL also has a very nice home-brew system. Gabor
Automated testing
I'm setting up a test server that is suppose to do check-outs (well exports) from svn for our projects, run the Makefile.PL and then make test and if there's any failures do something with the output. It's not a very difficult task, but I assume there's probably some module that does this already (something related to CPAN.pm's functionality, perhaps). But, my searching today is not finding anything (in Test::, CPAN::, TAP, etc.). Is there something that's designed for this job that I'm missing? Thanks, -- Bill Moseley [EMAIL PROTECTED] Sent from my iMutt
Re: Automated testing
You might want to look at Smolder: http://www.slideshare.net/mpeters/smolder-introduction -- David On Fri, Nov 28, 2008 at 7:18 PM, Bill Moseley [EMAIL PROTECTED] wrote: I'm setting up a test server that is suppose to do check-outs (well exports) from svn for our projects, run the Makefile.PL and then make test and if there's any failures do something with the output. It's not a very difficult task, but I assume there's probably some module that does this already (something related to CPAN.pm's functionality, perhaps). But, my searching today is not finding anything (in Test::, CPAN::, TAP, etc.). Is there something that's designed for this job that I'm missing? Thanks, -- Bill Moseley [EMAIL PROTECTED] Sent from my iMutt
