On 13 Apr 2009, at 12:46, aslak hellesoy wrote:
A couple of years ago I was on a project that had fallen into the
trap of too many integration tests (exactly the horror scenario that
J.B. Rainsberger describes: http://www.jbrains.ca/permalink/239).
The whole suite had hundreds of slow watir tests and took several
hours to run. If there was a failure, it was usually just in a
coupel of them.
We ended up improving this a lot with a homegrown distributed RSpec
runner (based on Drb) and employing techniques from pairwise testing
(http://www.pairwise.org/).
At the time I also explored a third technique that we never put in
use: Heuristics.
If we could establish relationships between arbitrary files and
failing tests over time, then we would be able to calulate the
probablilty that a certain commit would break certain tests. We
could then choose to run the tests that had a high probablilty of
breakage, and exclude the others. A neural network could potentially
be used to implement this. The input neurons would be files in the
codebase (on if it's changed, off if not) and the output neurons
would be the tests to run.
So if someone develops a better AutoTest with a plugin architecture,
and that doesn't have to run as a long/lived process, then I'd be
very interested in writing the neural network part - possibly backed
by FANN (http://leenissen.dk/fann/)
It's so crazy it has to be tried!
Aslak
Aslak as usual you have stepped it up another seventy levels!
I'm really keen to write a generic test runner that works on both
RSpec and Cucumber examples alike (and could extend for other testing
libraries). Allowing for pluggable strategies for test selection as
Stephen has described is a terrific idea.
Aslak do you have any ideas in the pipeline for building a fancier
test runner for Cucumber? I know there was some brain-storming on the
list a while back about a 'thick client'.
Matt Wynne
http://beta.songkick.com
http://blog.mattwynne.net
_______________________________________________
rspec-users mailing list
rspec-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/rspec-users
