I guess the main thing that jumps out at me is the possibility of the students
being "primed" by reading the first program and this affecting their
comprehension of the second and third program. An obvious control would be to
have the students read the programs in random order.
I am not sure how to test comprehension because it will be difficult to
disentangle individual student skill from the how readable the programs are. It
will depend on a number of factors like comprehension of the concepts involved.
So for instance if your programs are OO, then the students who do not grasp OO
will have a harder time regardless of the readability of the program. Perhaps
testing them on the concepts in the test program prior to the experiment would
help control this.
From: Richard O'Keefe <o...@cs.otago.ac.nz>
To: Ppig-Discuss-List <Ppig-Discuss-List@open.ac.uk>
Sent: Thu, July 15, 2010 1:40:18 AM
Subject: How to do a small readability experiment?
I want to run a small readability experiment this semester
as a class exercise. I've done a lot of web searching looking
for guidelines, but not found anything really helpful.
My idea is
- 3 programs, about 2 pages each
- 3 styles
- Latin square
- record reading times + time to find mistakes +
confidence in answers + comprehension questions +
demographics (age, experience)
- students should criticise the experiment
Looking at it dispassionately, the program size isn't really
enough to evaluate the question I'm interested in, but it's
about as much as I can ask people to do for an assignment.
The point is to give the students experience with what a
(toy) software engineering experiment is like.
Anyone have any ideas on what good comprehension questions
The programming language is basically an imperative pseudocode
to avoid the "if this is Java then it ought to be laid out/
spelled/structured/ like this" effect.
-- The Open University is incorporated by Royal Charter (RC 000391), an exempt
charity in England & Wales and a charity registered in Scotland (SC 038302).