Are we voting? :-) I also take this approach. As for the bogus treatment example, Michael is well advised to consider all of the various threats to validity that might befall a poorly-designed research study.
Suppose the hypothesis is that Vitamin C improves academic achievement. A design that would be highly susceptible to regression effects would be one in which a group of students selected because of their poor academic achievement is "treated" with a regimen of daily doses of Vitamin C. They are measured on a pretest on an academic topic (to make sure they are really low-achieving and don't know this material) and are given the same test as a posttest following 4 weeks of treatment. To see if they have been "fixed" and are able to perform like typical students, they are compared to a group of students with average academic achievement that does not receive Vitamin C but takes the posttest. Lo - the treated group scores higher on the posttest than on the pretest and looks a lot like the typical students! We can explain this result in terms of regression to the mean, but that isn't the only threat to validity lurking in this design. The pretest itself can be reactive and produce change in the treated student's performance. The students learned something from taking it or just became better practiced with this type of test procedure. (Yes, giving the control group the pretest as well will help control this, but I built a bad design on purpose here.) The parents of the treated students, alarmed at the poor performance on the pretest, might have implemented their own intervention with at-home tutoring. The students were curious about the questions on the pretest and spent the following weeks reading Wikipedia (or an even more reliable web source), watching Nova and the Discovery channel, etc. The students matured for unknown reasons and took the posttest more seriously. (Maybe the researcher was extra encouraging during the posttest administration and warned them against Christmas-treeing their response sheets - we could spin some expectation biases into this.) No doubt, there are additional threats (Campbell and Stanley have a nice long list!). Now if someone wants to defend Vitamin C as a treatment for poor academic performance . . . have at it! I do a version of this as a demonstration of regression effects using a deck of cards or a random number generator on my calculator to "measure" achievement (and create a "deficient" group for treatment and a "high achieving" group for comparison based on the pretest measure) The deficient group always gets better in the post test (only takes a 30-second "treatment" that involves much waving of my hands) and the high achieving group shows some slippage. The effect of random error is more obvious in this demonstration. Claudia J. Stanny, Ph.D. Director, Center for University Teaching, Learning, and Assessment Associate Professor, Psychology University of West Florida Pensacola, FL 32514 - 5751 Phone: (850) 857-6355 or 473-7435 e-mail: [email protected] CUTLA Web Site: http://uwf.edu/cutla/ Personal Web Pages: http://uwf.edu/cstanny/website/index.htm -----Original Message----- From: Stuart McKelvie [mailto:[email protected]] Sent: Monday, February 09, 2009 12:13 PM To: Teaching in the Psychological Sciences (TIPS) Subject: RE: [tips] [tips]Regression to the mean (was Bogus treatments) Rick asked if anyone else teaches regression to the mean basic on classical test theory assumptions. This is exactly what I do too. Stuart ___________________________________________________________________ Stuart J. McKelvie, Ph.D., Phone: (819)822-9600, Extension 2402 Department of Psychology, Fax: (819)822-9661 Bishop's University, 2600 College Street, Sherbrooke, Québec J1M 1Z7, Canada. E-mail: [email protected] Bishop's University Psychology Department Web Page: http://www.ubishops.ca/ccc/div/soc/psy ___________________________________________________________ --- To make changes to your subscription contact: Bill Southerly ([email protected]) --- To make changes to your subscription contact: Bill Southerly ([email protected])
