The thing about mechanical turk is that most Turkers are gaming it (so it's perfect for game theory), so you have to design your tasks so that it can't be gamed. The Turker wants to minimize the work for the money (but if you pay more, they still do the minimal work but get more money). So judgements of "which is better" (that are on the order of a sentence or a paragraph) work well. But even those need some sort of quality control (questions that let you judge whether people are even reading the task or just selecting one answer -- you can refuse to pay people who can't answer those questions right). Multiple choice quizes work with that caveat. You might be able to do "find the bug" with simple, short code snippets.
So the real question is, can you design your study so that you can get at your research questions with these sorts of tasks? (I've been looking at Mechanical Turk, but not for programming tasks. I have no idea how many programmers there are on the list, but my bet is that if you wanted general programming skill, not knowledge of a particular language, you could find enough). For someone who has used Mechanical Turk for research purposes, you might look at Ed Chi's work. Robin On Wed, Dec 23, 2009 at 6:31 PM, Derek M Jones <[email protected]> wrote: > All, > > Has anybody on this list used Amazon's Mechanical Turk > aws.amazon.com/mturk/ > to run psychology of programming experiments? > > I have no idea how many programmers might be members of this > service. The list of tasks does not look that technical. > > An interesting blog by somebody who has been following this > service: > behind-the-enemy-lines.blogspot.com/ > > -- > Derek M. Jones tel: +44 (0) 1252 520 667 > Knowledge Software Ltd mailto:[email protected] > Source code analysis http://www.knosof.co.uk >
