Robin,
be gamed. The Turker wants to minimize the work for the money (but if you
pay more, they still do the minimal work but get more money). So
My experience of running experiments with software developers is
that they generally tend to try and minimize the amount of work
they have to do, even though I they know that it is a 30 minute
experiment (or xx minutes). I think it is part of developer make-up
to try and do things the easy way and in many ways this is
desirable behavior; who wants to hire a developer who tries to
do things the hard way?
For instance an experiment I ran this year asked developers to
remember write some code that involved them using either if or
switch (I was looking to duplicate the findings in figure 2 of
www.knosof.co.uk/cbook/accu09a.pdf) and remembering some unrelated
information.
It looks like they used a fixed strategy for selecting if/switch
(all but one always used one or the other), concentrating their
effort on remembering the unrelated information.
judgements of "which is better" (that are on the order of a sentence or a
paragraph) work well. But even those need some sort of quality control
(questions that let you judge whether people are even reading the task or
just selecting one answer -- you can refuse to pay people who can't answer
those questions right). Multiple choice quizes work with that caveat. You
might be able to do "find the bug" with simple, short code snippets.
I am wondering whether non-programmers would attempt the problems and
just guess the answers. The monetary rewards are so small that they
are obviously not the primary motivation, or are we dealing with
third-world people here?
So the real question is, can you design your study so that you can get at
your research questions with these sorts of tasks?
My interest is in the cognitive issues that involve less than 10 seconds
of time, so the Mechanical Turk looks like it might be applicable.
For someone who has used Mechanical Turk for research purposes, you might
look at Ed Chi's work.
Thanks. Do you mean this guy?
http://www2.parc.com/istl/groups/uir/people/ed/ed.htm
Robin
On Wed, Dec 23, 2009 at 6:31 PM, Derek M Jones <de...@knosof.co.uk> wrote:
All,
Has anybody on this list used Amazon's Mechanical Turk
aws.amazon.com/mturk/
to run psychology of programming experiments?
I have no idea how many programmers might be members of this
service. The list of tasks does not look that technical.
An interesting blog by somebody who has been following this
service:
behind-the-enemy-lines.blogspot.com/
--
Derek M. Jones tel: +44 (0) 1252 520 667
Knowledge Software Ltd mailto:de...@knosof.co.uk
Source code analysis http://www.knosof.co.uk
--
Derek M. Jones tel: +44 (0) 1252 520 667
Knowledge Software Ltd mailto:de...@knosof.co.uk
Source code analysis http://www.knosof.co.uk
