Richard,
Thanks for the detailed comments.
2. The experiment made use of deception: subjects were led
I don't think deception was used. The original experiment performed
in at the 2004 ACCU conference, www.knosof.co.uk/cbook/accu04.html,
required a filler task and one involving deciding which arm of an
if statement would be executed was used. I realized that it might
be possible to obtain useful information from the results of the
filler task.
The identifiers used in the 2006 experiment were chosen to try and
answer some of the questions that the 2004 experiment threw up.
A different filler task was used because nothing of interest came
out of the if statement filler. The precedence results were so
interesting that I wrote them up first and gave them more prominence.
to believe that the experiment was about remembering the
values of variables from one side of the paper to the other.
"The answers" that "subjects [gave when] asked to ... list
any strategies they used" "all related to remembering information
about the assignment statements. There was no mention of
the parenthesis problem." This suggests that the subjects
may have regarded the actual experiment as an unimportant
distractor and not TRIED to perform it accurately.
The possibility that subjects did not try to accurately answer the
precedence questions is called out as a threat to the validity of
the results in the write up.
3. C's bitwise operations are unusually, not to say unwisely, placed.
Page 8 of the PDF tells us that one subject commented on being
unfamiliar with << . We are given no reason to believe that that
subject was alone. At one time, 53% of developers were said to
An even bigger threat that I did not spot until after writing up
was that there was no "I would look this up" box for subjects to tick.
It is possible that by forcing subjects to make a choice they made
more mistakes than they would otherwise have made. This is an
important issue and I am annoyed I missed it. I plan to rerun
the experiment and allow subjects to give this response.
use Visual Basic; perhaps the VB operators would have been, or
would be, a better should for such an experiment?
The ACCU conference is the Association of C and C++ users, and it
also has a Java stream. So I think it is reasonable to assume that
subjects were very familiar with these languages.
4. I have some fairly serious statistical doubts about the analysis.
Most of them are technical, but one isn't: the results are based
on just 17 subjects. The fact that one subject out of the original
18 didn't understand the problem makes one wonder about the sample.
I would be very interested in hearing them. Readers of this list might
not be so keen, so perhaps you would send them to me direct.
--
Derek M. Jones tel: +44 (0) 1252 520 667
Knowledge Software Ltd mailto:[EMAIL PROTECTED]
Applications Standards Conformance Testing http://www.knosof.co.uk
----------------------------------------------------------------------
PPIG Discuss List (discuss@ppig.org)
Discuss admin: http://limitlessmail.net/mailman/listinfo/discuss
Announce admin: http://limitlessmail.net/mailman/listinfo/announce
PPIG Discuss archive: http://www.mail-archive.com/discuss%40ppig.org/
