Hi Richard, I guess the main thing that jumps out at me is the possibility of the students being "primed" by reading the first program and this affecting their comprehension of the second and third program. An obvious control would be to have the students read the programs in random order.
I am not sure how to test comprehension because it will be difficult to disentangle individual student skill from the how readable the programs are. It will depend on a number of factors like comprehension of the concepts involved. So for instance if your programs are OO, then the students who do not grasp OO will have a harder time regardless of the readability of the program. Perhaps testing them on the concepts in the test program prior to the experiment would help control this. ________________________________ From: Richard O'Keefe <o...@cs.otago.ac.nz> To: Ppig-Discuss-List <Ppig-Discuss-List@open.ac.uk> Sent: Thu, July 15, 2010 1:40:18 AM Subject: How to do a small readability experiment? I want to run a small readability experiment this semester as a class exercise. I've done a lot of web searching looking for guidelines, but not found anything really helpful. My idea is - 3 programs, about 2 pages each - 3 styles - Latin square - record reading times + time to find mistakes + confidence in answers + comprehension questions + demographics (age, experience) - students should criticise the experiment Looking at it dispassionately, the program size isn't really enough to evaluate the question I'm interested in, but it's about as much as I can ask people to do for an assignment. The point is to give the students experience with what a (toy) software engineering experiment is like. Anyone have any ideas on what good comprehension questions look like? The programming language is basically an imperative pseudocode to avoid the "if this is Java then it ought to be laid out/ spelled/structured/ like this" effect. -- The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302).