Hi Richard

Here are some detailed remarks about technique that you just might find useful. If it's grandmother and egg time, please accept my apologies.

1) Given your hypothesis, I don't think you need to use huge programs. If you get a significant result from something N lines long, you can expect a larger effect from something 10N lines long, unless there's something unusual happening.

2) There's a difference between reading for detail or verbatim recall versus reading for gist, well documented in the psycholinguistics literature. Françoise Détienne showed that the difference also applies in program comprehension, but I forget the details of her studies. You might want to use more than one measure, one for gist (like telling a story about the program) and one for detailed comprehension. Judith Good has done excellent work on telling stories about programs. My own work has mainly been on reading for detail.

3) If you want to look at detail, rather than gist, you're probably going to be looking at times or errors as the outcome measure. Time scores are quite nice to use, because they're much finer grain than number of errors. But it's essential that the participants know whether you want them to be incredibly accurate or incredibly fast - there's a trade-off - one option is present them a scenario, such as safety-critical software, and explain that you want them to be accurate even at the cost of slow performance. Or the other way round, if preferred.

Using reading for gist, e.g. telling stores about the program, will require more work from the experimenter in assessing each reply.

4) When reading for detail, there's a difference between reasoning 'upstream' and reasoning 'downstream'. For an ordinary imperative structure, 'downstream' means in the direction of program flow, which is the easy way to reason. I did a study on answering questions about conditional structures, with professional programmers, using varieties of syntax for the conditional structure. Given something like

        if P then

                if Q then A

                else B
        else if R then ...

etc, the downstream direction is : Given the truth-values for P, Q, R ..., what does this program do? The upstream direction is: The program did action B, what must have been the truth-values?

What I found was very little difference (using time-to-answer) for the downstream direction, but large and very significant differences for upstream.

I suggest therefore that you try to include both directions in your tests.

5) A technique you might consider for detail comprehension is a version of the cloze procedure (fill in the blank). I did an unpublished study using forced-choice from about 4 alternatives. Iirc the participants were told what the program was intended to do, and had to find one of the alternatives that gave the correct result - so for instance, if the program was to find the mean of N numbers, you could have a blank in the summation, and offer different operators; or you have a blank in the divisor, and offer N, or N+1, or something else. (My test programs were a good deal larger, of course.) You have to be careful that the answer can't be found on purely syntactic reasoning. This might be quite an attractive method for you, assuming you want to use something like time scores, because correct reasoning could be quite complex and therefore even small differences in readability might show up.

5) If you use time scores, you'll probably get skewed distributions (but not nearly as skewed as error scores usually seem to be, at least in my studies). Anova is said to be reasonably robust for small amounts of skew, as long as it's always in the same direction, but a technique I have frequently used is to supplement anova on the raw scores with anova on transformed scores - if the skew is positive (usual case), you can use a log transform, for example. If you get the same pattern of significances, you're home and dry. If you don't, er, it depends.

6) There are alternatives to complete Latin squares. Incomplete squares confound the controlled variables; if you can accept the confounding, you can save experimental effort. You'd need to look at a book on experimental design for more info.

7) Don't forget to talk to the participants afterwards and ask what they thought, how they did the tasks, and so on. You may find some surprises - I certainly have done.

Can't think of anything else just now but open to questions. Good luck and make sure the results are made public!

Thomas Green

On 16 Jul 2010, at 07:42, Richard O'Keefe wrote:

On Jul 16, 2010, at 2:20 AM, Alan Blackwell wrote:

Hi Richard,

A couple of clarifications - your primary experimental
manipulation is 'style', is that right? Could you explain what
this means?

I thought I already said explicitly that it was about
whether to use
- runtogetherlowercasewords (like BSD 'getprogname')
- well_separated_words (as recommended in AQ&S and Meyer OOSC3)
- baStudlyCaps (asInJava)

The variants of each program are generated from a master.
Keywords are in bold and comments in italics, so that it isn't
any harder to find the keywords in any style than in any other,
and the indentation is identical.  The _only_ difference is
whether it's leftcapacity or left_capacity or leftCapacity
(or whatever the word is, that's an actual example).

And do you have a hypothesis, regarding the impact of 'style' on
your measures? (and the expected interaction of age/experience
with style).

Null hypothesis: no significant difference.
Preferred alternative: well_separated beats baStudly beats
The class is actually pretty homogenous this year, do I
don't expect age or experience effects to be detectable,
although I expect that in the wider world they exist and matter.

This is NOT intended to be the definitive experiment.
Its primary excuse for existence is to give the students
some experience of such experiments, with a research question
that I'd genuinely like to know the answer to, so it's "real"
in some sense.

With luck, we will soon have the Psychology of Programming book
available online, after which David Gilmore's chapter on
'Methodological Issues in the Study of Programming' will provide
useful guidance along these lines.

Excellent news.  You couldn't put that chapter up first?

The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302).

73 Huntington Rd, York YO31 8RL

Reply via email to