The thing about mechanical turk is that most Turkers are gaming it (so it's
perfect for game theory), so you have to design your tasks so that it can't
be gamed.  The Turker wants to minimize the work for the money (but if you
pay more, they still  do the minimal work but get more money).  So
judgements of "which is better" (that are on the order of a sentence or a
paragraph) work well. But even those need some sort of quality control
(questions that let you judge whether people are even reading the task or
just selecting one answer -- you can refuse to pay people who can't answer
those questions right).   Multiple choice quizes work with that caveat.  You
might be able to do "find the bug" with simple, short code snippets.

So the real question is, can you design your study so that you can get at
your research questions with these sorts of tasks?

(I've been looking at Mechanical Turk, but not for programming tasks.  I
have no idea how many programmers there are on the list, but my bet is that
if you wanted general programming skill, not knowledge of a particular
language, you could find enough).

For someone who has used Mechanical Turk for research purposes, you might
look at Ed Chi's work.


On Wed, Dec 23, 2009 at 6:31 PM, Derek M Jones <> wrote:

> All,
> Has anybody on this list used Amazon's Mechanical Turk
> to run psychology of programming experiments?
> I have no idea how many programmers might be members of this
> service.  The list of tasks does not look that technical.
> An interesting blog by somebody who has been following this
> service:
> --
> Derek M. Jones                         tel: +44 (0) 1252 520 667
> Knowledge Software Ltd       
> Source code analysis         

Reply via email to