So far we've been quite happy with Test::Aggregate here at work. It's
one of a number of strategies we've used to improve our test suite
performance, something we desperately need (7.1 minutes down to 3.6
minutes for aggregated tests).
That being said, the process boundary issue that Eric Wilhelm was
concerned about is a real one. When the software isn't expecting
global changes, we have found several real bugs. When the software
*tests* don't take this into consideration, we have found several
spurious bugs:
use Test::Most 'no_plan';
use Some::Module;
*Some::Module::foo = sub { ... }; # oops
$ENV{SOME_CRITICAL_VALUE} = 7; # oops
Simply localizing those changes, since they're only for the test, helps
out quite a bit, but not everything is as easy to localize as that
(unloading a singleton's namespace would be useful, for example).
Does anyone have any generic suggestions on how to handle this problem?
I'm thinking that pre- and post- handlers can help, but I really don't
want to go too far towards eliminated process boundaries because
eventually I'd be trying to emulate fork in a cross-platform way.
Introducing too much overhead and complications leads to more bugs and
eliminates the benefits of aggregated tests. Also, since eliminating
process boundaries has revealed a number of bugs, I want to retain that
benefit.
My initial thought is something like this:
use Test::Aggregate;
my $tests = Test::Aggregate->new( {
dirs => 'aggtests/',
set_filenames => 1,
pre => { # keys are regex matches
'*/customers/*' => \&clean_boundaries,
'/acceptance.t' => \&clean_boundaries,
}
post => {
'*/order/failure.t' => \&restore_static_data,
}
} );
sub clean_boundaries { ... }
sub restore_static_data { ... }
$tests->run;
The primary consideration here is to ensure that tests can still be
moved in and out of aggregated tests without alteration.
Thoughts?
Cheers,
Ovid
--
Buy the book - http://www.oreilly.com/catalog/perlhks/
Perl and CGI - http://users.easystreet.com/ovid/cgi_course/
Personal blog - http://publius-ovidius.livejournal.com/
Tech blog - http://use.perl.org/~Ovid/journal/
