Some years ago I did a simulation on the pretest-posttest control group design lokking at three methods of analysis, ANCOVA, repeated measures ANOVA, and treatment by block factorial ANOVA (blocking on the pretest using a median split). I found that that with typical sample sizes, the repeated measures ANOVA was a bit more powerful than the ANCOVA procedure when the correlation between pretest and posttest was fairly high (say .90). As noted below, this is because the ANCOVA and ANOVA methods are approaching the same solution but ANCOVA loses a degree of freedom estimating the regression parameter when the ANOVA doesn't. Of course this effect diminshes as the sample size gets larger because the loss of one df is diminished. On the other hand, the treatment by block design tends to have a bit more power when the correlation between pretest and posttest is low (< .30). I tried to publish the results at the time but aimed a bit too high and received such a scathing review (what kind of idiot would do this kind of study?) that I shoved it a drawer and it has never seen the light of day since. I did the syudy because it seemed at the time that everyone was using this design but were unsure of the analysis and I thought a demonstration would be helpful. SO, to make a long story even longer, the ANCOVA seems to be most powerful in those circumstances one is likely to run into but does have somewhat rigid assumptions about homogeneity of regression slopes. Of course the repeated measures ANOVA indirectly makes the same assumption but at such high correlations, this is really a homogenity of variance issue as well. The second thought is for you reviewers out there trying to soothe your own egos by dumping on someone else's. Remember, the researcher you squelch today might be turned off to research and fail to solve a meaty problem tomorrow.
Paul R. Swank, Ph.D. Professor Developmental Pediatrics UT Houston Health Science Center -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of jim clark Sent: Thursday, September 27, 2001 7:00 AM To: [EMAIL PROTECTED] Subject: Re: Analysis of covariance Hi On 26 Sep 2001, Burke Johnson wrote: > R Pretest Treatment Posttest > R Pretest Control Posttest > In the social sciences (e.g., see Pedhazur's popular > regression text), the most popular analysis seems to be to > run a GLM (this version is often called an ANCOVA), where Y > is the posttest measure, X1 is the pretest measure, and X2 is > the treatment variable. Assuming that X1 and X2 do not > interact, ones' estimate of the treatment effect is given by > B2 (i.e., the partial regression coefficient for the > treatment variable which controls for adjusts for pretest > differences). > Another traditionally popular analysis for the design given > above is to compute a new, gain score variable (posttest > minus pretest) for all cases and then run a GLM (ANOVA) to > see if the difference between the gains (which is the > estimate of the treatment effect) is statistically > significant. > The third, and somewhat less popular (?) way to analyze the > above design is to do a mixed ANOVA model (which is also a > GLM but it is harder to write out), where Y is the posttest, > X1 is "time" which is a repeated measures variable (e.g., > time is 1 for pretest and 2 for posttest for all cases), and > X2 is the between group, treatment variable. In this case one > looks for treatment impact by testing the statistical > significance of the two-way interaction between the time and > the treatment variables. Usually, you ask if the difference > between the means at time two is greater than the difference > at time one (i.e., you hope that the treatment lines will not > be parallel) > Results will vary depending on which of these three > approaches you use, because each approach estimates the > counterfactual in a slightly different way. I believe it was > Reichardt and Mark (in Handbook of Applied Social Research > Methods) that suggested analyzing your data using more than > one of these three statistical methods. Methods 2 and 3 are equivalent to one another. The F for the difference between change scores will equal the F for the interaction. I believe that one way to think of the difference between methods 1 and 2/3 is that in 2/3 you "regress" t2 on t1 assuming slope=1 and intercept=0 (i.e., the "predicted" score is the t1 score), whereas in method 1 you estimate the slope and intercept from the data. Presumably it would be possible to simulate the differences between the two analyses as a function of the magnitude of the difference between means and the relationship between t1 and t2. I don't know if anyone has done that. Best wishes Jim ============================================================================ James M. Clark (204) 786-9757 Department of Psychology (204) 774-4134 Fax University of Winnipeg 4L05D Winnipeg, Manitoba R3B 2E9 [EMAIL PROTECTED] CANADA http://www.uwinnipeg.ca/~clark ============================================================================ ================================================================= Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ ================================================================= ================================================================= Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =================================================================
