To my thinking the graphs in figure 1 show no effect of experience.
That means that interesting as the experiment was, it may not bear
on the question of whether there is empirical evidence that the
'imperative' paradigm is most intuitive.
It showed some correlation between developer performance and relative
occurrences of a construct in source code. A correlation between
source code occurrences and developer experience is assumed. This
is a reasonable assumption to make, but of course the size of this
correlation is unknown.
The graph on the left of figure 1 _hinted_ that the population
might be heterogeneous (one group answering about 100 problems
and the other group about 175). Extracting the numbers from
the graph, here's a histogram:
No histogram :-(
It was extremely puzzling that "%" scored so badly compared
with "*" and "/". Could there be interference from % formats?
There are a variety of possible interferences, eg * is also a
unary operator. At some point an operator may occur so rarely
that subject answers are effectively random. The small subject
size may have contributed to a large variance in this case.
That study asked some questions that needed to be asked, but
I think it has to be considered as a pilot study, and the question
of the possible heterogeneity of the population means that I
don't regard it as having answered them yet.
I agree. I would like to rerun this experiment several more
times. My access to experienced subjects is limited, and I have
been running different experiments at each ACCU conference.
The high error rate 33% raises another question.
This question was addressed by the ACCU 07 experiment.
Unfortunately this had a very low turnout (it was up against
a talk by Microsoft on the latest release of Visual Studio).
Thanks for providing your problems. I meant to give them to
some 3rd year students this year, but forgot. (I asked them to
read patents instead. Produced a classful of rabid patent-haters.)
On several occasions I have worked towards running this experiment
using students. The feedback I have had from members of staff
is that many students are likely to be unfamiliar with the bitwise
operators. I would expect student answers to be close to random.
I have investigated creating an online version of this experiment.
The (free) packages I have looked at do not have the required
Webexp, www.webexp.info/, came closest (I have not tried the release
that appeared a few weeks ago).
The following experiment showed that developers used variable
names to make precedence decisions:
I hadn't seen this one. I've just printed part 1 off and will read
it. Looks interesting. The link "Part 2 _download_ pdf" gives me
a page not found error.
Fixed. The accu07a.pdf you downloaded should have been named
Derek M. Jones tel: +44 (0) 1252 520 667
Knowledge Software Ltd mailto:de...@knosof.co.uk
Source code analysis http://www.knosof.co.uk