But consider that the run of participations you're about to make could just be a pilot.

I understand. To clarify my test, I'm not going to compare Scratch/BYOB with my tool. I'm going to compare miniC (a minimal C implementation built on BYOB) vs regular C environments. I want to test if students that learned C by using miniC:
- do less syntactic mistakes
- remember to declare their variables more often
- use sequence/loop/conditionals in a more consistent way

HCI designers regularly deal with these kinds of issues all the time

I did it myself several times. What I'm not sure is if they are using the correct instruments to get correct, unbiased data from their tests. Do they look only at what is commonly seen as a programming base-skill (logic/math)? Or do they take into account also other -maybe apparently unrelated- but important dimensions? Which are their most recent studies on this subject? Did they publish their test-bed (as Dehani-Bornat did)?

After all, from these tests they should claim that their work is successful. Are they doing it in the right way? Shouldn't we have a common, clearly-understood test-bed on which this kind of experimentation should be performed? This doesn't mean that the test-bed should be unupdateable, but at least important part of it should. Otherwise our tests wouldn't be comparable as they should.

I'm I dreaming too much?

Thanks again


Citando "Guzdial, Mark" <guzd...@cc.gatech.edu>:

I'm listening, but have little to add. I just point back to Marian's list -- those are the issues, Stefano. Gather what data you can and hope that your two groups are roughly equivalent on the variables you're hoping to control for.

A bigger issue (again, reflecting back to Marian's list) is the design of the task. I'm guessing that this is Mini-Logo vs. Scratch? Or MiniLogo vs. BYOB-Scratch vs. Scratch? In any case, there's going to likely be an interaction between the task and the environment -- some tasks are impossible in Scratch that the others can do, so a task that all three can do is likely to NOT take advantage of whatever makes MiniLogo and BYOB-Scratch unique. It's hard to do these kinds of tests. Even after this experiment, some think-aloud protocols with the environments that you most want to compare might give you the most insightful results about the real impact of the tool.

Stefano, computer scientists do this, too. In particular, HCI designers regularly deal with these kinds of issues all the time. I'm a professor in a School of Interactive Computing in a College of Computing -- my colleagues and I do work with human subjects regularly. I think it's terrific for Programming Language designers to care about user testing and experiments with human participants! But consider that the run of participations you're about to make could just be a pilot. Make your mistakes now, and the next iteration will be publishable and head-turning.


On Mar 19, 2011, at 6:50 AM, Thomas Green wrote:

On 19 Mar 2011, at 09:55, Stefano Federici wrote:

what I claim is the easiest programming environment ever designed so far).

Er, yes. You might need to restrict what you mean by 'programming' ..... I regard using spreadsheets as programming. But Scratch is very good at its job, to be sure.

I take it then that you're trying to out-do Scratch.

Does this sound reasonable?

Yes, it's probably the best you can do. I think the worst threat to generalisability is probably the risk of 'experimenter effect', where the students do better in the group that you want to do better. I don't know how to minimise that risk. If Sally Fincher or Mark Guzdial is listening, or anyone else with a good knowledge of these issues, I hope they'll join in.

Good luck! Make sure to tell us how it goes.


73 Huntington Rd, York YO31 8RL

The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302).

Stefano Federici
Università degli Studi di Cagliari
Facoltà di Scienze della Formazione
Dipartimento di Scienze Pedagogiche e Filosofiche
Via Is Mirrionis 1, 09123 Cagliari, Italia
Cell: +39 349 818 1955 Tel.: +39 070 675 7815
Fax: +39 070 675 7113

Reply via email to