On Dec 10, 2009, at 11:57, Alan Blackwell wrote:


My own experience, based on recent investigation with a grad
student to compile more evidence for his claims regarding
complexity metrics (in this case Harrison's entropy-based measure
rather than McCabe) was that the closer we looked at the measure
itself, the more flawed it seemed to be - both on fundamental
theoretical grounds, and also on the evidence presented for its
empirical validity.

I think there is some significant work waiting to be done in this
area. However the purposes that these metrics get put to are also
questionable, so doing the work to construct a sufficiently
controlled and validated metric would quite possibly result in it
being abused further.


Is your other list just having a flame war,
or do you think there is someone who would like to make a serious
investigation? If the latter, I'm sure that there are plenty of
PPIG folk who would encourage it, and provide methodological
advice.

No, not a flame war, just a short interchange. Unfortunately most are software engineers and so not well versed in this type of experimental study design, more about predicting bugs.

I can think of a million confounds (e.g. programmer's familiarity with coding style/syntax) but hey, that's what good experimental design is all about :)

Alan

Over on the libresoft mailing list we're having a conversation about
interpretation of complexity metrics (e.g. McCabe cyclometic
complexity etc).  The studies that we know on this metric demonstrate
that the metric is useful to predict bugs, but I often hear the
further interpretation that complexity actually causes more bugs (or
inhibits their fixes) because the code is harder to understand.

That interpretation seems to need stronger validation than the
correlational studies.  I thought this forum might know of some
studies that approach this.  For example has anyone tried to measure
the impact of (e.g.) higher cyclometric complexity on the speed of
fixing a bug in code?  I'm thinking of some comparison of effort to
fix the "same bug" with different coding styles (one with high
cyclometric complexity, one with lots of function calls, something
else)?

An alternative explanation of the correlation might be that complexity
metrics measure the difficulty of work (ie difficulty of the work is
driving both the complexity and the bugs, at the same time).

Thanks,
James
http://james.howison.name


--
Alan Blackwell
Reader in Interdisciplinary Design, University of Cambridge
Further details from www.cl.cam.ac.uk/~afb21/



Reply via email to