On Mon, Jun 19, 2006 at 10:26:08PM +1000, Adelle Hartley wrote:
> > Does anyone here have any expereince with mutation testing,
> > (http://benno.id.au/blog/2006/06/18/mutants), or have any
> > opinions on it?
>
> Intriguing, but I have doubts about it being a sensible path.
>
> Surely, for any interpreted language, it ought to be possible to build a
> version of the interpreter that records which parts of the byte code were or
> weren't executed?
I don't think mutation testing is intended for examining code coverage
(because you're right, you should be able to do coverage analysis for at
least interpreted languages), it's meant to examine the effects of small
changes in your production code on the validity of your test suite. I'll
try and give a bit of a picture of what I think it does, let's see if I can
make this comprehensible...
Let's say that you want your program to go down a certain path (not code
path, exactly, think of it like a lane in the road):
|
|
|
Before you have any unit testing, the path that your program could be taking
(when it's modified, for example) is essentially unbounded (the angles are
the boundaries you've defined for your program):
> | <
> | <
> | <
When you put a decent test suite around your software, you bring those
boundaries closer, so there's less "wiggle" room for your software before
the test suite tells you that it's going down the garden path:
> | <
> | <
> | <
What mutation testing does, then, is the job of automatically wiggling your
software around within it's boundaries. If you have a good enough test
suite, *any* wiggling that the mutations produce will be picked up as a
failed test case. This is good, according to the mutation testing doctorine
-- your test suite is essentially packed hard-up against the production
code, and it can't go *anywhere* you don't want it to without your test
suite telling you about it. Otherwise, the mutation testing tool has
identified some extra test cases that you need to write.
> Mutation seems kind of hit or miss.
What it seems to me to be is a return to the fallacy of "complete testing".
Remember the days when people were advocating testing every single input
value and checking it returned the correct output value? No wonder testing
was a complete joke. To my way of thinking, mutation testing against any
reasonable codebase is going to produce so many "you need a test for this"
results that you're just going to ignore it, just like everyone ignored test
gurus who said you're going to have to test every possible input, because in
the real world it's just not feasible.
I've always thought of unit testing as like a poker machine -- each test is
a pull of the handle. If the test initially fails (showing you something
that needs to be fixed), then that's a win. Similarly, if you write a test
for new functionality and it unexpectedly passes, then you've won as well --
your program already does what your test wanted it to do, so no new
production code needed.
Writing tests for working code, that already does it's job satisfactorily
and for which the test suite is reasonably complete already, is just going
to be a diminishing returns thing. If the mutation testing tool says "you
need to write 100 new tests", how many of those 100 tests that it identifies
are ever actually going to "pay out", in the poker machine analogy?
Maybe the MT tools are clever enough to tell you which tests are most
worthwhile -- but I doubt it. The mutations I've heard about (though
probably not all of the ones that are used) are fairly simplistic and don't
seem to be the sort of thing that's likely to identify real show-stopping
bugs.
- Matt
_______________________________________________
coders mailing list
coders@slug.org.au
http://lists.slug.org.au/listinfo/coders
