Re: Test::Aggregate - Speed up your test suites
On 31/12/2007, Sam Vilain [EMAIL PROTECTED] wrote: Ovid wrote: If you have slow test suites, you might want to give it a spin and see if it helps. Essentially, it concatenates tests together and runs them in one process. Thus, you load Perl only once and load all modules only once. Yuck. Why not just load Perl once and fork for the execution of each test script. You can pre-load modules before you fork. Fork not being all that portable or implemented equivalently on all platforms might be an issue. Yves -- perl -Mre=debug -e /just|another|perl|hacker/
Re: Test::Aggregate - Speed up your test suites
--- Sam Vilain [EMAIL PROTECTED] wrote: Ovid wrote: If you have slow test suites, you might want to give it a spin and see if it helps. Essentially, it concatenates tests together and runs them in one process. Thus, you load Perl only once and load all modules only once. Yuck. Why not just load Perl once and fork for the execution of each test script. You can pre-load modules before you fork. Forking is also more likely to be used for parallelization. Often code requires sweeping changes before it can be run in parallel. So this means we're reduced to running the code sequentially and forking doesn't offer a huge advantage and can mask hidden state assumptions like when naughty code is munging globals such as filehandles or built-in globals. Also, since forking is only emulated on Windows, it's not reliable (I've had it crash and burn more than once). I prefer to avoid writing modules that are limited to specific platforms. (I'm not saying forking is a bad solution, just a different one). Finally, Test::Aggregate is designed to have tests run with minimal changes. For many tests, just move them to the aggregate directory. No worries about which modules to preload or anything like that. Finally, if you think my code is such a bad idea, I'm sure folks would welcome alternatives. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/
Re: Test::Aggregate - Speed up your test suites
--- Eric Wilhelm [EMAIL PROTECTED] wrote:
Now, assuming that both ways carry the same caveats, I too would
prefer
the latter because it not only eliminates load time, but also allows
concurrency and provides isolation via the boundary between child
processes.
I may be a bit of a heretic, but after years of using Test::Class, I
rather like the fact that I don't have the isolation between tests
because a lot of code makes incorrect assumptions about all sorts of
things. For example, here's a couple of time bombs waiting to happen:
sub slurp {
open FH, $file or die $!;
do { $/ = undef; FH }
}
Without the process boundary, I'm more likely to catch this in tests.
I know most folks would disagree with me on this, but I find this a far
more common problem than folks think.
Hmm, could one slurp the __DATA__/__END__ block into a scalar and
open
DATA, '', \$scalar it on behalf of the subprocess?
PPI?
The goal of Test::Aggregate was to improve performance :)
Kidding aside, PPI could be an optional feature to get around some
issues, but only with aggressive caching and the open question is
whether or not speeding up tests is a widespread enough problem to
warrant the effort.
I don't see anything in Test::Aggregate where which modules to
preload
is specified. Presumably, it lets perl's compile-time handle that.
Correct. As I've stated before, the goal of this is to make it as
transparent as possible. Move tests from t/ to your 'aggregate'
directory.
I also wonder about debug-ability where the `prove aggtests/foo.t` is
potentially a completely different landscape than the full run. If
the
full run is 10 minutes and the landscape-specific failure is at the
end
that seems a bit costly.
You can dump the file that it's running and debug that, including
commenting out slow bits. This is documented.
Cheers,
Ovid
--
Buy the book - http://www.oreilly.com/catalog/perlhks/
Perl and CGI - http://users.easystreet.com/ovid/cgi_course/
Personal blog - http://publius-ovidius.livejournal.com/
Tech blog - http://use.perl.org/~Ovid/journal/
Re: Test::Aggregate - Speed up your test suites
On 31 Dec 2007, at 02:24, Sam Vilain wrote: Ovid wrote: If you have slow test suites, you might want to give it a spin and see if it helps. Essentially, it concatenates tests together and runs them in one process. Thus, you load Perl only once and load all modules only once. Yuck. Why not just load Perl once and fork for the execution of each test script. You can pre-load modules before you fork. Why not replace this thing you have done which works and is useful with the thing I think it should be but haven't implemented Hey, Sam - instead of doing whatever you're doing right now come and clean my bathroom. That's what I'd prefer you to do. -- Andy Armstrong, Hexten
Re: Test::Aggregate - Speed up your test suites
Taken at face value, absent intonation and other extralinguistic context, Sam’s “yuck” was way over the line. I have to say I am no less taken aback at the reactions though. * Andy Lester [EMAIL PROTECTED] [2007-12-31 05:40]: I think this wins the Least Helpful Response To Someone Who Has Provided A New Module For Everyone To Use. Bravo! * Andy Armstrong [EMAIL PROTECTED] [2007-12-31 14:00]: Hey, Sam - instead of doing whatever you're doing right now come and clean my bathroom. That's what I'd prefer you to do. What should it profit a man, if he should win a flame war, yet lose his cool? Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Test::Aggregate - Speed up your test suites
On Dec 31, 2007 6:49 AM, Andy Armstrong [EMAIL PROTECTED] wrote: Why not replace this thing you have done which works and is useful with the thing I think it should be but haven't implemented Hey, Sam - instead of doing whatever you're doing right now come and clean my bathroom. That's what I'd prefer you to do. OK -- this has degenerated and that's uncalled for. (a) Sam didn't say what what in the quotes above. That was someone's (bad) paraphrase. (b) Other than the initial yuck, Sam asked a very reasonable question about the design decisions behind the module. Posting about a module in public invites public criticism of the module -- particularly the technical details and often motivational criticism (when modules duplicate existing ones). If someone doesn't like it, just release modules to CPAN and don't post about it afterwards. Let's not confuse design commentary with personal attacks. David
Re: Test::Aggregate - Speed up your test suites
On 31 Dec 2007, at 14:04, David Golden wrote: On Dec 31, 2007 6:49 AM, Andy Armstrong [EMAIL PROTECTED] wrote: Why not replace this thing you have done which works and is useful with the thing I think it should be but haven't implemented Hey, Sam - instead of doing whatever you're doing right now come and clean my bathroom. That's what I'd prefer you to do. OK -- this has degenerated and that's uncalled for. Aye. My post was an attempt at a light-hearted parody of how Sam's post might be interpreted. But I see I forgot a smiley or any other indication that I wasn't frothing with rage as I wrote it. I'll abandon my aspirations as a diplomat and get back to coding :) -- Andy Armstrong, Hexten
Re: Test::Aggregate - Speed up your test suites
On 31 Dec 2007, at 11:15, Ovid wrote: --- Eric Wilhelm [EMAIL PROTECTED] wrote: Now, assuming that both ways carry the same caveats, I too would prefer the latter because it not only eliminates load time, but also allows concurrency and provides isolation via the boundary between child processes. I may be a bit of a heretic, but after years of using Test::Class, I rather like the fact that I don't have the isolation between tests because a lot of code makes incorrect assumptions about all sorts of things. For example, here's a couple of time bombs waiting to happen: [snip] That's been my experience too. I've caught many nice bugs that would have been missed by completely-clean slate tests. Of course there have also been many nice bugs caught by clean-slate tests that would have been missed in a persistent environment ;-) I don't think it's a particularly clear cut issue. On the fork front two issues come to mind. 1) Windows fork :-/ 2) With previous test parallelisation hacks that I've put together the biggest problem I come across is test suites that assume that only one test script is running at a time. My hunch, from having had to mess with several large messy commercial test suites in the past, is that something like T::A is more likely to work than a forking/ parallel solution for many suites. Not that a nice parallelisation system wouldn't be welcome too - especially one that can distribute tests over multiple machines. if I only had some spare tuits :-) Cheers, Adrian
Re: Test::Aggregate - Speed up your test suites
# from Ovid
# on Monday 31 December 2007 03:15:
For example, here's a couple of time bombs waiting to happen:
sub slurp {
open FH, $file or die $!;
do { $/ = undef; FH }
}
Without the process boundary, I'm more likely to catch this in tests.
I know most folks would disagree with me on this, but I find this a
far more common problem than folks think.
Do you happen to have another example? This one looks to me like poorly
written code in the test (or are you citing this as code in the
product?) Terribly hackish code in the tests is not such a huge deal,
so I'm reluctant to see being forced to address it as a benefit of
T::A.
Either way, it is glaringly bad code.
a. any call to slurp() doesn't pass a filename -- screams of evil
b. 2-arg form of open -- banned
c. non-lexical filehandles -- banned
That is, I think there are other ways to catch such fish. I suspect
that adding peculiarities of the test environment is a not-so-good
detection mechanism.
Perhaps you're trying to address the code makes global state
assumptions issue? Well, I think that might be borrowing trouble.
Consider e.g. a singleton which is customizable upon creation
(by-design it assumes that only one set of parameters applies from
creation onward.) How could one test variations on that singleton's
parameters with T::A?
I think a preload-then-fork approach more closely replicates the
production environment.
--Eric
--
Cult: A small, unpopular religion.
Religion: A large, popular cult.
-- Unknown
---
http://scratchcomputing.com
---
Re: Test::Aggregate - Speed up your test suites
# from Adrian Howard # on Monday 31 December 2007 11:18: That's been my experience too. I've caught many nice bugs that would have been missed by completely-clean slate tests. Are they bugs in the tests or actual bugs? 1) Windows fork :-/ Big deal? Are the issues relevant to preloading or only concurrency? If the latter, then just turn off the concurrency feature. 2) With previous test parallelisation hacks that I've put together the biggest problem I come across is test suites that assume that only one test script is running at a time. My hunch, from having had to mess with several large messy commercial test suites in the past, is that something like T::A is more likely to work than a forking/ parallel solution for many suites. Again, preloading or concurrency? Concurrency. If preloading-only via fork gives a working test suite and loads faster than aggregation, it is a win independent of whether your tests play nice with concurrency. Not that a nice parallelisation system wouldn't be welcome too - especially one that can distribute tests over multiple machines. Bit of a different bag of beasts there. Though I suspect it is doable in much the same code as a forking preloader (e.g. reading results on a pipe and etc.) For instance, assume a master and slave preloaders where the master does the SGI::FAM bits and restarts the slaves whenever the files have changed (or something like that.) Which test to run maps from per-process to per-node quite nicely and independently of the startup code. --Eric -- Moving pianos is dangerous. Moving pianos are dangerous. Buffalo buffalo buffalo buffalo buffalo buffalo buffalo. --- http://scratchcomputing.com ---
Re: buildbot - an experiment
On Sat, Dec 29, 2007 at 05:51:50PM -0500, James E Keenan wrote: How might this be used to perform smoke-testing for a project like Parrot, where we want to test on many combinations of operating system, platform and C compiler? If anyone can give me an idiots' guide to how to grab the most recent source tree, build it, and test it, then I can test it on the same boxes as I do CPAN testing, plus maybe a couple of others. -- David Cantrell | Cake Smuggler Extraordinaire PLEASE NOTE: This message was meant to offend everyone equally, regardless of race, creed, sexual orientation, politics, choice of beer, operating system, mode of transport, or their editor.
