Hi Richard,

I guess the main thing that jumps out at me is the possibility of the students 
being "primed" by reading the first program and this affecting their 
comprehension of the second and third program. An obvious control would be to 
have the students read the programs in random order. 

I am not sure how to test comprehension because it will be difficult to 
disentangle individual student skill from the how readable the programs are. It 
will depend on a number of factors like comprehension of the concepts involved. 
So for instance if your programs are OO, then the students who do not grasp OO 
will have a harder time regardless of the readability of the program. Perhaps 
testing them on the concepts in the test program prior to the experiment would 
help control this.

From: Richard O'Keefe <o...@cs.otago.ac.nz>
To: Ppig-Discuss-List <Ppig-Discuss-List@open.ac.uk>
Sent: Thu, July 15, 2010 1:40:18 AM
Subject: How to do a small readability experiment?

I want to run a small readability experiment this semester
as a class exercise.  I've done a lot of web searching looking
for guidelines, but not found anything really helpful.
My idea is
    - 3 programs, about 2 pages each
    - 3 styles
    - Latin square
    - record reading times + time to find mistakes +
      confidence in answers + comprehension questions +
      demographics (age, experience)
    - students should criticise the experiment
Looking at it dispassionately, the program size isn't really
enough to evaluate the question I'm interested in, but it's
about as much as I can ask people to do for an assignment.
The point is to give the students experience with what a
(toy) software engineering experiment is like.

Anyone have any ideas on what good comprehension questions
look like?

The programming language is basically an imperative pseudocode
to avoid the "if this is Java then it ought to be laid out/
spelled/structured/ like this" effect.

-- The Open University is incorporated by Royal Charter (RC 000391), an exempt 
charity in England & Wales and a charity registered in Scotland (SC 038302).

Reply via email to