But consider that the run of participations you're about to make
could just be a pilot.
I understand. To clarify my test, I'm not going to compare
Scratch/BYOB with my tool. I'm going to compare miniC (a minimal C
implementation built on BYOB) vs regular C environments. I want to
test if students that learned C by using miniC:
- do less syntactic mistakes
- remember to declare their variables more often
- use sequence/loop/conditionals in a more consistent way
HCI designers regularly deal with these kinds of issues all the time
I did it myself several times. What I'm not sure is if they are using
the correct instruments to get correct, unbiased data from their
tests. Do they look only at what is commonly seen as a programming
base-skill (logic/math)? Or do they take into account also other
-maybe apparently unrelated- but important dimensions? Which are their
most recent studies on this subject? Did they publish their test-bed
(as Dehani-Bornat did)?
After all, from these tests they should claim that their work is
successful. Are they doing it in the right way? Shouldn't we have a
common, clearly-understood test-bed on which this kind of
experimentation should be performed? This doesn't mean that the
test-bed should be unupdateable, but at least important part of it
should. Otherwise our tests wouldn't be comparable as they should.
I'm I dreaming too much?
Thanks again
stefano
Citando "Guzdial, Mark" <guzd...@cc.gatech.edu>:
I'm listening, but have little to add. I just point back to
Marian's list -- those are the issues, Stefano. Gather what data
you can and hope that your two groups are roughly equivalent on the
variables you're hoping to control for.
A bigger issue (again, reflecting back to Marian's list) is the
design of the task. I'm guessing that this is Mini-Logo vs.
Scratch? Or MiniLogo vs. BYOB-Scratch vs. Scratch? In any case,
there's going to likely be an interaction between the task and the
environment -- some tasks are impossible in Scratch that the others
can do, so a task that all three can do is likely to NOT take
advantage of whatever makes MiniLogo and BYOB-Scratch unique. It's
hard to do these kinds of tests. Even after this experiment, some
think-aloud protocols with the environments that you most want to
compare might give you the most insightful results about the real
impact of the tool.
Stefano, computer scientists do this, too. In particular, HCI
designers regularly deal with these kinds of issues all the time.
I'm a professor in a School of Interactive Computing in a College of
Computing -- my colleagues and I do work with human subjects
regularly. I think it's terrific for Programming Language designers
to care about user testing and experiments with human participants!
But consider that the run of participations you're about to make
could just be a pilot. Make your mistakes now, and the next
iteration will be publishable and head-turning.
Cheers,
Mark
On Mar 19, 2011, at 6:50 AM, Thomas Green wrote:
On 19 Mar 2011, at 09:55, Stefano Federici wrote:
what I claim is the easiest programming environment ever designed so far).
Er, yes. You might need to restrict what you mean by 'programming'
..... I regard using spreadsheets as programming. But Scratch is
very good at its job, to be sure.
I take it then that you're trying to out-do Scratch.
Does this sound reasonable?
Yes, it's probably the best you can do. I think the worst threat to
generalisability is probably the risk of 'experimenter effect',
where the students do better in the group that you want to do
better. I don't know how to minimise that risk. If Sally Fincher or
Mark Guzdial is listening, or anyone else with a good knowledge of
these issues, I hope they'll join in.
Good luck! Make sure to tell us how it goes.
Thomas
73 Huntington Rd, York YO31 8RL
01904-673675
http://homepage.ntlworld.com/greenery/
--
The Open University is incorporated by Royal Charter (RC 000391),
an exempt charity in England & Wales and a charity registered in
Scotland (SC 038302).
Stefano Federici
-------------------------------------------------
Università degli Studi di Cagliari
Facoltà di Scienze della Formazione
Dipartimento di Scienze Pedagogiche e Filosofiche
Via Is Mirrionis 1, 09123 Cagliari, Italia
-------------------------------------------------
Cell: +39 349 818 1955 Tel.: +39 070 675 7815
Fax: +39 070 675 7113
