If you reply to this long (11 kB) post please don't hit the reply
button unless you prune copy of this post that may appear in your
reply down to a few relevant lines, otherwise the entire already
archived post may be needlessly resent to subscribers.
****************************************
ABSTRACT: TIPster Paul Bernhardt pointed out a serious omission in my
post "Randomized Trials. . . .," viz.: "Hake does not . . . describe
the conditions under which non-randomized assignment designs can
establish causation and whether those conditions are present in the.
. .[pre/post]. . . . studies under examination." The U.S. Dept. of
Education's (USDE's) psychologist-dominated "What Works Clearing
House" would certainly NOT regard these studies as meeting their
much-contested causation "gold standard" - randomized control trials.
Nevertheless, the USDE would probably regard the studies as meeting
their lesser "silver standard" - "Meets Evidence Standards with
Reservations" since the studies are quasi-experimental studies
[Shadish et al. (2002)] of especially strong design." The strength of
the design derives partly from: (a) the use of reasonably
well-matched control groups offered by the "traditional" courses, and
(b) corroboration of the results by many different research groups.
****************************************
I thank TIPster Paul Bernhardt (2008) for pointing out a serious
omission in my post "Randomized Trials (was Can Pre-to-posttest Gains
Gauge Course Effectiveness?)" [Hake (2008a)]. Paul wrote:
"Dr. Hake simply asserts that Shadish, Cook, & Campbell (2002) and
other writers agree that random assignment is not necessary to
establish causation, and he's right. There are several design
conditions that, when met, indicate a causal relationship. What Dr.
Hake does not do is describe the conditions under which
non-randomized assignment designs can establish causation and whether
those conditions are present in the studies under examination."
The studies under examination - pre/post studies by physics education
researchers - [Hake (1998a,b) and about 25 other pre/post studies
referenced in Hake (2008b)] - do not meet the psychologist-dominated
U.S. Dept. of Education's (USDE's) much contested - see e.g., the
Cook quote in the signature below - "gold standard" of randomized
control trials, but would nevertheless probably pass muster at the
USDE's "What Works Clearing House" <http://ies.ed.gov/ncee/wwc/> as
"quasi-experimental studies [Shadish et al. (2002)] of especially
strong design" [see
<http://ies.ed.gov/ncee/wwc/references/standards/>].
The strength of the design derives partly from:
(a) the use of reasonably well-matched control groups offered by the
"traditional" courses, and
(b) corroboration of the results by many different research groups
[see Hake (2008b)].
Despite rampant pre/post paranoia [Hake (2006)], pre/post assessments
of student learning are being more and more utilized in fields such
as astronomy, economics, biology, biomechanics, calculus, chemistry,
geoscience, statistics, statics, and engineering (circuits, computer
Engineering, dynamics, electromagnetics, electronics, fluid
mechanics, heat transfer, materials, signals and systems, strength
of materials, thermodynamics, and waves) - see Hake (2008b, 2008d,
2008f) - but evidently NOT in psychology :-( .
Richard Hake, Emeritus Professor of Physics, Indiana University
24245 Hatteras Street, Woodland Hills, CA 91367
Honorary Member, Curmudgeon Lodge of Deventer, The Netherlands.
<[EMAIL PROTECTED]>
<http://www.physics.indiana.edu/~hake/>
<http://www.physics.indiana.edu/~sdi/>
"In some quarters, particularly medical ones, the randomized
experiment is considered the causal 'gold standard.' It is clearly
not that in educational contexts, given the difficulties with
implementing and maintaining randomly created groups, with the
sometimes incomplete implementation of treatment particulars, with
the borrowing of some treatment particulars by control group units,
and with the limitations to external validity that often follow from
how the random assignment is achieved."
Thomas Cook and Monique Payne in "Evidence Matters" [Mosteller &
Boruch (2002)]
REFERENCES [Tiny URL's courtesy <http://tinyurl.com/create.php>.]
Bernhardt, P.C. 2008. :Re: Randomized Trials (was Can
Pre-to-posttest Gains Gauge Course Effectiveness?)," TIPS post of 22
Oct 2008 05:22:44-0700; online on the OPEN! TIPS archives at
<http://tinyurl.com/5a7hzk>
Hake, R.R. 1998a. "Interactive-engagement vs traditional methods: A
six thousand- student survey of mechanics test data for introductory
physics courses," Am. J. Phys. 66(1): 64-74; online as an 84 kB pdf
at <http://tinyurl.com/3xuyqe> . See also Hake (1998b).
Hake, R.R. 1998b. "Interactive- engagement methods in introductory
mechanics courses," online as a 108 kB pdf at
<http://tinyurl.com/2tg5d9> - a crucial companion paper to Hake
(1998a).
Hake, R.R. 2005. "Cross-Posting - Synergistic or Sinful?" Post of 1
Nov 2005 08:37:12-0800 to ITFORUM and AERA-L; online at at
<http://tinyurl.com/2m59v4>.
Hake, R.R. 2006. Possible Palliatives for the Paralyzing Pre/Post
Paranoia that Plagues Some PEP's, Journal of MultiDisciplinary
Evaluation 6, November, online at
<http://survey.ate.wmich.edu/jmde/index.php/jmde_1/issue/view/22>.
This even despite the admirable anti-alliteration advice at
psychologist Donald Zimmerman's site
<http://mypage.direct.ca/z/zimmerma/> to "Always assiduously and
attentively avoid awful, awkward, atrocious, appalling, artificial,
affected alliteration." This is a severely truncated version of
"Should We Measure Change? Yes! [Hake (2008c).
Hake, R.R. 2008a. "Randomized Trials (was Can Pre-to-posttest Gains
Gauge Course Effectiveness?)" online on the OPEN! AERA-D archives at
<http://tinyurl.com/5wp582>. Post of 2-22 Oct 2008 to AERA-D, ASSESS,
EdResMeth, EdStat-L, EvalTalk, IFETS, Net-Gold, Phys-L, PhysLrnR,
POD, STLHE-L (abstract only), TIPS, & WBTOLL-L.
Hake, R.R. 2008b. "Design-Based Research in Physics Education
Research: A Review," in "Handbook of Design Research Methods in
Education: Innovations in Science, Technology, Engineering, and
Mathematics Learning and Teaching" [Kelly, Lesh, & Baek (2008)] -
publisher's information at <http://tinyurl.com/4eazqs>; a
pre-publication version of Hake's chapter is online at
<http://www.physics.indiana.edu/~hake/DBR-Physics3.pdf> (1.1 MB).
Hake, R.R. 2008c. 'Should We Measure Change? Yes!" online at
<http://www.physics.indiana.edu/~hake/MeasChangeS.pdf> or as ref. 43
at <http://www.physics.indiana.edu/~hake>. To appear as a chapter in
"Evaluation of Teaching and Student Learning in Higher Education"
[Hake, in preparation].
Hake, R.R. 2008d. "Over Eighty Annotated and Hot-linked References to
Sources Containing or Alluding to Data Demonstrating the Benefit of
Reform Pedagogy in STEM Disciplines," online on the OPEN! AERA-C
archives at <http://tinyurl.com/3qaut3>. Post of 10 October 2008 to
AERA-A, AERA-B, AERA-C, AERA-D, AERA-GS, AERA-H, AERA-J, AERA-K,
AP-Physics, ASSESS, Biopi-L, Chemed-L, DrEd, EdResMeth, EvalTalk,
IFETS, Math-Learn, Math-Teach, Net-Gold, Physhare, Phys-L, PhysLrnR,
POD, PsychTeacher (rejected), RUME, STLHE-L, TeachEdPsych, TIPS, &
WBTOLL-L. For a guide to discussion lists see Hake (2008e). For a
defense of cross-posting see Hake (2005).
Hake, R.R. 2008e. "Over Sixty Academic Discussion Lists: List
Addresses and URL's for Archives & Search Engines" online at
<http://www.physics.indiana.edu/~hake/ADL-L.pdf> (637 kB).
Hake, R.R. 2008f. "Re: Formative Pre/post Tests For Various
Disciplines," online on the OPEN! AERA-D archives at
<http://tinyurl.com/6z8ntp>. Post of 7 Jul 2008 15:50:28 -0700 to
AERA-D, ASSESS, Dr-Ed, EdRedMeth, EdStat, EvalTalk, and POD.
Mosteller, F. & R. Boruch, eds. 2002. "Evidence Matters: Randomized
Trials in Education Research." Brookings Institution. Amazon.com
information at <http://tinyurl.com/59gp6o>.
Shadish, W.R., T.D. Cook, & D.T. Campbell. 2002."Experimental and
Quasi- Experimental Designs for Generalized Causal Inference."
Houghton Mifflin. Amazon.com information at
<http://tinyurl.com/6kel78>. Note the"Search Inside" feature. A
goldmine of references on social science research.
---
To make changes to your subscription contact:
Bill Southerly ([EMAIL PROTECTED])
