Depending on your aims, you might want to measure transfer to other
problems: that is, do participants who used tool A for the sorting
task, then do better when tackling a new problem, possibly with a
different tool, than participants who used tool B?
You might also want to look at memory and savings: how do the
participants manage two months later? Occasionally cognitive tasks
like yours show no effect at the time but produce measurable
differences when the same people do the same tasks later.
Pretty hard to create a truly fair test, but things to think about are
controlling for practice and order effects, which should be easy, and
controlling for experimenter expectation effects. The hardest thing to
balance for is sometimes the training period: people using a new tool
have to learn about it, and that gives them practice effects that the
controls might not get. Sometimes people create a dummy task for the
control condition to avoid that problem; or you can compare different
versions of the tools, with differing features.
I suggest you try to avoid the simple A vs B design and instead look
for a design when you can predict a trend: find A, B, C such that your
theory says A > B > C. The statistical power is much better.
Don't forget to talk to the people afterwards and get their opinions.
Sometimes you can find they weren't playing the same game that you were.
Good luck
Thomas Green
On 1 Mar 2011, at 11:20, Stefano Federici wrote:
Dear Collegues,
I need to plan an evaluation of the improvements brought by the
usage of specific software tools when learning the basic concepts of
computer programming (sequence, loop, variables, arrays, etc) and
the specific topic of sorting algorithms.
Which are the best practises for the necessary steps? I guess the
steps should be: selection of test group, test of initial skills,
partition of the test group in smaller homogenous groups, delivery
of learning materials by or by not making use of the tools, test of
final skills, comparative analysis.
What am I supposed to do to perform a fair test?
Any help or reference is welcome.
Best Regards
Stefano Federici
-------------------------------------------------
Professor of Computer Science
University of Cagliari
Dept. of Education and Philosophy
Via Is Mirrionis 1, 09123 Cagliari, Italy
-------------------------------------------------
Tel: +39 349 818 1955 Fax: +39 070 675 7113
--
The Open University is incorporated by Royal Charter (RC 000391), an
exempt charity in England & Wales and a charity registered in
Scotland (SC 038302).
73 Huntington Rd, York YO31 8RL
01904-673675
http://homepage.ntlworld.com/greenery/
