As an (ex) compiler writer and one of the authors of the Camel paper,
I feel impelled to wade in. Except I no longer think I have nothing
interesting to say about understanding compilers: it's fun if you
like it is all.
The Camel paper was written very over-enthustiastically, for which I
apologise. That enthused far too many people and annoyed many others.
Both effects are to be regretted. Our results (Dehnadi's results,
really, I'm just a part-time scribe) are interesting but not world
shaking.
In most of the experiments (four out of six) for which we now have
data, we've found that the subgroup which scores 'consistent
C0' (using the same mental model in eight out of twelve questions) is
about half the sample, and has a strikingly significantly different
failure rate to the rest of the sample. Typically we find the C0
group has a 20% failure rate or less, and the rest have about 50%
failure rate. This isn't enough for us to claim (as I did, and I
regret it) that we can divide sheep from goats, because the 50% false
negative rate substantially undermines the test's efficacy as a
predictive instrument.
When we look more closely at the data we find more intriguing
information. In three of the experiments there are enough females for
us to be able to contrast them with males. The C0/not C0 distinction
holds up well for males, but NOT for females. In females the test
doesn't work: the inconsistent group does better than the consistent.
Numbers are so small that we can't say much more, and given my
apologies I'd better not start speculating again. (Since you ask, in
the fourth experiment almost all of the females had previous
programming experience and didn't score inconsistent, so no such
effect.)
We've also found, since Simon inspired us to include questions about
sex and previous programming experience, that there is a significant
group which has previous programming experience and already has the
Java/C models of equals-sign-as-right-to-left-copy and juxtaposition-
as-sequential-execution. This group -- call it CM2 -- skews the
results somewhat by dragging the C0 average upward, but even when we
exclude it from our analysis we can still see significant effects.
On the whole, in those four experiments, my (feeble) grasp of
statistics and the SPSS controls leads me to believe that Dehnadi's
test predicts about 25% of the variance between groups. That may not
sound much, but I have heard it said that since education is so
multifactorial, 30% is about as much as you can hope for. So, as I
said, at the moment interesting but not earth-shattering.
The anomalous two experiments, of which the Aarhus experiment is one,
do seem to have very different starting conditions. In both
experiments almost all the intake scores consistent/C0 at the
beginning (and at Aarhus it may be they score CM2, but I haven't had
clarification of that point from the experimenters). We have some
notion of why that happens in one experiment, which is in the UK, but
I don't know how the Danish education system works, so can't
speculate about Aarhus.
One intriguing result which hasn't yet been replicated, because it
was overlooked at first, is the 'week 3 effect' which Dehnadi found
in his first experiment. Forget for the moment about consistency:
just test whether students have the correct models of assignment and
sequence in week 3, when they've been taught them. The single time it
was tried this was more predictive than the original test, with much
smaller false negative and false positive trails. One collaborator
picked this out of the Camel paper, modified his teaching, and
promises us some data about his results soon (but we beware the
Hawthorne effect, as ever, and I'm aware that maybe the Hawthorne
experiment didn't show the Hawthorne effect, so please don't set that
dog on me).
For a long time, ever since I had to teach a group of Masters
students who had paid for their own course and were trying to convert
to computer science to get a job, I haven't believed that motivation
is the major underlying cause of the two-hump phenomenon. It must be
that a proportion of 18-year olds are unmotivated and don't try, but
there are so many who are desperate to succeed and don't seem to be
able to. Perhaps the week 3 effect will point to a remedy (come on
Ron! give us your data!) but more likely there is a weirdness that
we've hardly begin to understand let alone fix.
Richard Bornat
----------------------------------------------------------------------
PPIG Discuss List (discuss@ppig.org)
Discuss admin: http://limitlessmail.net/mailman/listinfo/discuss
Announce admin: http://limitlessmail.net/mailman/listinfo/announce
PPIG Discuss archive: http://www.mail-archive.com/discuss%40ppig.org/
