I'm listening, but have little to add. I just point back to Marian's list --
those are the issues, Stefano. Gather what data you can and hope that your two
groups are roughly equivalent on the variables you're hoping to control for.
A bigger issue (again, reflecting back to Marian's list) is the design of the
task. I'm guessing that this is Mini-Logo vs. Scratch? Or MiniLogo vs.
BYOB-Scratch vs. Scratch? In any case, there's going to likely be an
interaction between the task and the environment -- some tasks are impossible
in Scratch that the others can do, so a task that all three can do is likely to
NOT take advantage of whatever makes MiniLogo and BYOB-Scratch unique. It's
hard to do these kinds of tests. Even after this experiment, some think-aloud
protocols with the environments that you most want to compare might give you
the most insightful results about the real impact of the tool.
Stefano, computer scientists do this, too. In particular, HCI designers
regularly deal with these kinds of issues all the time. I'm a professor in a
School of Interactive Computing in a College of Computing -- my colleagues and
I do work with human subjects regularly. I think it's terrific for Programming
Language designers to care about user testing and experiments with human
participants! But consider that the run of participations you're about to make
could just be a pilot. Make your mistakes now, and the next iteration will be
publishable and head-turning.
On Mar 19, 2011, at 6:50 AM, Thomas Green wrote:
> On 19 Mar 2011, at 09:55, Stefano Federici wrote:
>> what I claim is the easiest programming environment ever designed so far).
> Er, yes. You might need to restrict what you mean by 'programming' ..... I
> regard using spreadsheets as programming. But Scratch is very good at its
> job, to be sure.
> I take it then that you're trying to out-do Scratch.
>> Does this sound reasonable?
> Yes, it's probably the best you can do. I think the worst threat to
> generalisability is probably the risk of 'experimenter effect', where the
> students do better in the group that you want to do better. I don't know how
> to minimise that risk. If Sally Fincher or Mark Guzdial is listening, or
> anyone else with a good knowledge of these issues, I hope they'll join in.
> Good luck! Make sure to tell us how it goes.
> 73 Huntington Rd, York YO31 8RL
> The Open University is incorporated by Royal Charter (RC 000391), an exempt
> charity in England & Wales and a charity registered in Scotland (SC 038302).