The correct statistical test is called McNemar's Test: http://www.medcalc.org/manual/mcnemar_test.php It is specifically for dichotomous outcome data which is paired (repeated or matched). A modified Bonferroni more or less as you describe looks easy enough to do manually, after extracting the p-values for each of the 25 McNemar tests. http://www.psych-it.com.au/Psychlopedia/article.asp?id=251
Paul On Jan 15, 2013, at 6:21 PM, Annette Taylor wrote: > I know this is a basic question but here goes: > > I have categorical data, 0,1 which stands for incorrect (0) or correct (1) on > a test item. > > I have 25 items and I have a pretest and a posttest and I want to know on > which items students improved significantly, and not just by chance. Just > eyeballing the data I can tell that there are some on which the improved > quite a bit, some not at all and some are someplace in the middle and I can't > make a guess at all. That is why we have statistics. Yeah! .... > hmmmm....bleh..... > > As far as I know, the best thing to do is a chi-square test for each of 25 > items; but of course that will mean that with a .05 sig level I will have at > least one false positive, maybe more, but most assuredly at least one. This > seems to be a risk. At any rate I can use SPSS and the crosstabs command > allow for calculation of the chi-square. > > I know that when I do planned comparisons with multiple t-tests, I can do a > Simes' correction in which I can rank order my final, obtained alphas, and > adjust for the number of comparisons and reject from the point from which the > obtained alpha failed to exceed the corrected-for-number-of-comps alpha. But > as far as I know, I cannot do that with 25 chi square tests. There is > probably some reason why I can no more do that, that relates to the reason > for why I cannot do 25 t-tests in this situation with categorical data. > > Is there a better way to answer my research question? I need a major > professor! Oh wait, that's me... drat! I need to hire a statistician. Oh > wait, I'd need $$ for that and I don't have any. So I hope tipsters can stand > in as a quasi-hired-statistician and help me out. > > Oh, I get the digest. I don't mind waiting until tomorrow or the next day for > a response, but a backchannel is fine. [email protected] > > I will be at APS this year. Any other tipsters planning to be there? Let's > have a party! I'd love to put personalities to names. > > Thanks > > Annette > > Annette Kujawski Taylor, Ph. D. > Professor, Psychological Sciences > University of San Diego > 5998 Alcala Park > San Diego, CA 92110 > [email protected] > --- > You are currently subscribed to tips as: [email protected]. > To unsubscribe click here: > http://fsulist.frostburg.edu/u?id=13441.4e79e96ebb5671bdb50111f18f263003&n=T&l=tips&o=23044 > or send a blank email to > leave-23044-13441.4e79e96ebb5671bdb50111f18f263...@fsulist.frostburg.edu > --- You are currently subscribed to tips as: [email protected]. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=23066 or send a blank email to leave-23066-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
