Dr. Hake simply asserts that Shadish, Cook, & Campbell (2002) and other writers agree that random assignment is not necessary to establish causation, and he's right. There are several design conditions that, when met, indicate a causal relationship.
What Dr. Hake does not do is describe the conditions under which non-randomized assignment designs can establish causation and whether those conditions are present in the studies under examination. On 10/21/08 8:03 PM, "Richard Hake" <[EMAIL PROTECTED]> wrote: > If you reply to this long (14 kB) post please don't hit the reply > button unless you prune copy of this post that may appear in your > reply down to a few relevant lines, otherwise the entire already > archived post may be needlessly resent to subscribers. > > *************************************** > ABSTRACT: In a recent post "Can Pre-to-posttest Gains Gauge Course > Effectiveness? #2," I wrote: "These [pre/post studies] have been > carried out on many different instructors, in many different > institutions, using many different texts, and working with many > different types of student populations from rural high schools to > Harvard." In response, AERA-D's Jeremy Miles asked "WERE THESE > RANDOMIZED TRIALS?" The short answer is "NO." The long answer > explains that: (a) randomized control trials (RCT's) are almost > impossible to carry out in undergraduate physics education research, > and (b) careful non-RCT research can establish causality to a > reasonable degree - as argued by Shadish, Cook, & Campbell; Shavelson > & Towne; Schneider, Carnoy, Kilpatrick, Schmidt, & Shavelson; and > Michael Scriven. > *************************************** > > In my post of 19 October 2008 titled "Can Pre-to-posttest Gains Gauge > Course Effectiveness? #2" [Hake (2008a)], I wrote: > > "These investigations. . . . .[Hake (1998a,b) and about 25 other > pre/post studies referenced in Hake (2008b)]. . . . have been > carried out on many different instructors, in many different > institutions, using many different texts, and working with many > different types of student populations from rural high schools to > Harvard." > > In response, AERA-D's Jeremy Miles (2008a) asked: "WERE THESE > RANDOMIZED TRIALS?" > > The short answer is "NO!" > > The long answer is: > > In undergraduate physics education research the use of randomized > control trials (RCT's) is nearly impossible. In "Re: Should > Randomized Control Trials Be the Gold Standard of Educational > Research? " [Hake (2005a)] I wrote: > > "Could physics education researchers (PER's) whose work is > predominately in UNDERGRADUATE education utilize RCT's? PER's deal > with populations of UP (university professors) and US (Undergraduate > Students). Most UP's demand autonomy in the way they teach courses > since they obviously know best how to lecture. Most of the US's (or > their parents) paid good money to be lectured at. No one that I know > of has been insane enough to even suggest that subjects from > populations UP and US be randomly assigned to different curricula in > a RCT, especially if one curriculum de-emphasizes lectures. Also the > average UP, thrown into an IE course would be a total disaster. If > anyone has some ideas on how to accomplish an RTC among UP's and US's > while avoiding dismissal or execution please let me know. Of course > one could PAY the subjects, but this might bias the results towards > the greedy and/or impecunious." > > That Miles is asking a leading question might be inferred from the > end of Section 2.2: "The structure of experiments" under "Collecting > Data - Experimental Design" in his valuable "Psychology Research > Methods Wiki" [Miles (2008b)]. There Miles gives the canonical > argument for randomized control trials: > > "One of the most important aspects of experimental design is random > assignment of different participants to different conditions - if you > do not randomly assign participants to conditions, you cannot be sure > that and differences in the conditions are due to the independent > variable." [Nor can you be sure even if there IS random assignment! - > see the signature quote by Thomas Cook.] > > But Miles (2008b) moderates the above in the "digression box" in > "Cause and Effect: The Advantage of Experiments" in Section 3.1: > > "You need to be aware that some people take a very strict line, and > say that without an experiment . . . .[i.e., "Randomized Control > Trial]. . . , statements about cause and effect can never be justly > made. Others take a less strict line, and say that statements about > cause and effect relationships can be made occasionally, if care is > taken (yes, you do not get any less strict than that). I hold that > causal statements can be made, if care is taken and appropriate > techniques are used, but your tutors may not agree." > > Among "tutors" who DO agree are: > > a. Shadish, Cook, & Campbell (2002) in "Experimental and > Quasi-Experimental Designs for Generalized Causal Inference"; > > b. Shavelson & Towne (2002) in "Scientific Research in Education," > National Academy Press; > > c. Schneider et al. (2007) in "Estimating Causal Effects Using > Experimental and Observational Designs"; > > d. Michael Scriven (2007) in "The Logic Of Causal Investigations." > > > Richard Hake, Emeritus Professor of Physics, Indiana University > 24245 Hatteras Street, Woodland Hills, CA 91367 > Honorary Member, Curmudgeon Lodge of Deventer, The Netherlands. > <[EMAIL PROTECTED]> > <http://www.physics.indiana.edu/~hake/> > <http://www.physics.indiana.edu/~sdi/> > > "In some quarters, particularly medical ones, the randomized > experiment is considered the causal 'gold standard.' It is clearly > not that in educational contexts, given the difficulties with > implementing and maintaining randomly created groups, with the > sometimes incomplete implementation of treatment particulars, with > the borrowing of some treatment particulars by control group units, > and with the limitations to external validity that often follow from > how the random assignment is achieved." > Thomas Cook and Monique Payne in "Evidence Matters" [Mosteller & > Boruch (2002)] > > > REFERENCES [Tiny URL's courtesy <http://tinyurl.com/create.php>.] > Cook, T.D. & M.R. Payne. 2002. "Objecting to the Objections to Using > Random Assignment in Educational Research," in Mosteller & Boruch > (2002). See also Cook (2001). > > Cook, T.D. 2001. "A critical appraisal of the case against using > experiments to assess school (or community) effects" Education Next, > No. 3 (Hoover Institute, Stanford, 2001), online at > <http://media.hoover.org/documents/ednext20013unabridged_cook.pdf> > (128 kB). CAUTION: a severely truncated popularization of this > scholarly paper appears under the provocative title "Sciencephobia: > Why education rejects randomized experiments," Education Next 1(3): > 62-68 (2001), online at > <http://www.hoover.org/publications/ednext/3384446.html>. One can > progress to the more academic "A critical appraisal of the case > against using experiments to assess school (or community) effects" by > clicking on "Unabridged PDF" in the right-hand column. For a > discussion of this paper see Scriven (2007). > > Hake, R.R. 1998a. "Interactive-engagement vs traditional methods: A > six thousand- student survey of mechanics test data for introductory > physics courses," Am. J. Phys. 66(1): 64-74; online as an 84 kB pdf > at <http://tinyurl.com/3xuyqe> . See also Hake (1998b). > > Hake, R.R. 1998b. "Interactive- engagement methods in introductory > mechanics courses," online as a 108 kB pdf at > <http://tinyurl.com/2tg5d9> - a crucial companion paper to Hake > (1998a). > > Hake, R.R. 2005a. Re: Should Randomized Control Trials Be the Gold > Standard of Educational Research? online on the OPEN! AERA-D archives > at <http://tinyurl.com/55deua>. Post of 17/18 Apr 2005 to AERA-C, > AERA-D, AERA-G, AERA-H, AERA-J, AERA-K, AERA-L, AP-Physics, ASSESS, > Biopi-L, Chemed-L, EvalTalk, Math-Learn, Phys-L, PhysLrnR, Physhare, > POD, STLHE-L, & TIPS. For a guide to discussion lists see Hake > (2008c). For a defense of cross-posting see Hake (2005b). > > Hake, R.R. 2005b. "Cross-Posting - Synergistic or Sinful?" Post of 1 > Nov 2005 08:37:12-0800 to ITFORUM and AERA-L; online at at > <http://tinyurl.com/2m59v4>. > > Hake, R.R. 2008a. "Can Pre-to-posttest Gains Gauge Course > Effectiveness? #2" online on the OPEN! AERA-D archives at > <http://tinyurl.com/5dx45q>. Post of 19 Oct 2008 16:08:08-0700 to > AERA-D, ASSESS, EdResMeth, EvalTalk, PhysLrnR, and POD. > > Hake, R.R. 2008b. "Design-Based Research in Physics Education > Research: A Review," in "Handbook of Design Research Methods in > Education: Innovations in Science, Technology, Engineering, and > Mathematics Learning and Teaching" [Kelly, Lesh, & Baek (2008)] - > publisher's information at <http://tinyurl.com/4eazqs>; a > pre-publication version of Hake's chapter is online at > <http://www.physics.indiana.edu/~hake/DBR-Physics3.pdf> (1.1 MB). > > Hake, R.R. 2008c. "Over Sixty Academic Discussion Lists: List > Addresses and URL's for Archives & Search Engines" online at > <http://www.physics.indiana.edu/~hake/ADL-L.pdf> (637 kB). > > Miles, J. 2001. " Research Methods and Statistics in Psychology: > Success in Your Psychology Degree," Crucial Publishers. Amazon.com > information at <http://tinyurl.com/6ouepa>. > > Miles, J. 2008a. "Re: Can Pre-to-posttest Gains Gauge Course > Effectiveness? #2" on the OPEN! AERA-D archives at > <http://tinyurl.com/6quapc>. > > Miles, J. 2008b. "Psychology Research Methods Wiki," online at > <http://www.researchmethodsinpsychology.com>. Based on Miles (2001). > > Mosteller, F. & R. Boruch, eds. 2002. "Evidence Matters: Randomized > Trials in Education Research." Brookings Institution. Amazon.com > information at <http://tinyurl.com/59gp6o>. > > Schneider, B.M. Carnoy, J. Kilpatrick, W.H. Schmidt, R.J. Shavelson. > 2007. "Estimating Causal Effects Using Experimental and Observational > Designs." AERA, publisher's information and FREE download at > <http://www.aera.net/publications/Default.aspx?menu_id=46&id=3360>. > > Scriven, M. 2007. "The Logic Of Causal Investigations," probably in > press; online at <http://homepages.wmich.edu/~mscriven/>; especially > Quasi-Experimental Designs (QXTs), pp. 21-23. The reference to Cook > (2000) should be Cook (2001) - see above. Scriven is now at Claremont > Graduate University <http://www.cgu.edu/pages/4745.asp>. See also > Scriven (2008). > > Scriven, M. 2008. Re: A Primer on Medical Studies, EvalTalk post of > 15 Oct 2008 10:25:41-0700; online at <http://tinyurl.com/56ttjl>. > Scriven writes: ". . . . many alleged RCTs have given incorrect > conclusions (some of them at least because of the Hawthorne effect, > present in virtually all so-called RCTs in social cases under IRBs), > and that's still true even of many truly double-blind RCTs; see the > placebo literature (eg, but not only for this reason, because they're > not triple-blind). The use of a 'third arm' that is, another group > that receives no treatment at all, placebo or experimental, is > increasingly popular in order to avoid the above epidemic of mistaken > interpretations, and even then, the measurement needs to be > unobtrusive or you merely have a second placebo group. Now that's not > too hard to do with drug studies, but it's extremely hard with > studies of social interventions." > > Shadish, W.R., T.D. Cook, & D.T. Campbell. 2002."Experimental and > Quasi-Experimental Designs for Generalized Causal Inference." > Houghton Mifflin. Amazon.com information at > <http://tinyurl.com/6kel78>. Note the"Search Inside" feature. A > goldmine of references on social science research. > > Shavelson, R.J. & L. Towne, eds. 2002. "Scientific Research in > Education," National Academy Press; online at > <http://www.nap.edu/catalog/10236.html>. > > > > > > > --- > To make changes to your subscription contact: > > Bill Southerly ([EMAIL PROTECTED]) --- To make changes to your subscription contact: Bill Southerly ([EMAIL PROTECTED])
