Oops - I mistated part of the conclusions when I gave the "between 27% and
39%" figure, as the 27% was actually the overall (student and faculty)
percentage correct for the second question (it'd be easier if I actually had
the article here - it's well worth reading). My point stands, though.
Paul Smith
Alverno College
Milwaukee
> The May and Hunter article abstracted below points to a possible
> source of the misconception among students: a shockingly high
> prevalence
> (between 27% and 39%) of the same misconception among faculty
> (though the
> source is by now a bit dated - I'd love to see what the
> numbers look like
> today). I don't have the article in front of me, and so I
> don't know what it
> reported about the fields from which the faculty were drawn
> (and I hope to
> hell that they weren't generally in the behavioral sciences...).
>
> Paul Smith
> Alverno College
> Milwaukee
>
> ========================================
> May, R. B., & Hunter, M. A. (1988). Interpreting students'
> interpretations of research. Teaching of Psychology, 15, 156-158.
>
> Purpose
> This article describes a study investigating students'
> and faculty
> members' understandings of the purposes of random selection and random
> assignment to groups. Students often confuse the two concepts, feeling
> relatively familiar with random selection and its role in
> providing external
> validity (generalizability). However, the role of random assignment to
> groups in providing internal validity is far less clear to
> students. Earlier
> published work indicates that this overemphasis on external
> validity extends
> beyond students: one author cited points out that as a
> journal editor, he
> noticed that contributing authors and reviewers alike made
> the same error.
>
> Methods
> Two multiple-choice questions, both with the same
> response options
> were asked of 18 faculty and 46 students. The faculty were generally
> tenured, holding Ph.D.s and having a median of 17
> publications. The students
> were in their second statistics course, and included 18
> graduate students
> and 28 undergraduates.
> The first question was
> Random sampling of subjects from a specified population of interest
> facilitates:
> (a) generality interpretations of results
> (b) cause-effect interpretations of results
> (c) both a and b, but a more than b
> (d) both a and b, but b more than a
> (e) both a and b equally
>
> The second question was
> Random assignment of subjects to groups receiving different treatments
> facilitates:
> (same response options)
>
> Results
> Participants were far more likely to correctly answer
> the sampling
> question than the assignment to groups question (75% vs. 27%). The
> proportions for students and faculty were roughly similar on
> question 1, and
> while faculty were more likely to get question 2 correct,
> only 39% did so.
>
> Conclusions
> The results reveal a likely bias in training, such that
> students are
> exposed to far more work involving random sampling and
> generality than on
> threats to internal validity. The authors (and others)
> suggest teaching "the
> random assignment model" before "the normal model" in order
> to avoid the
> effects of this bias.
>
