That was my understanding too ... though while washing dishes last night I warmed up to Jim's (was it Jim Clark?? Sorry if I forgot!) suggestion (imperfect memory here) of treating item as a random factor, get a CI, and then noting which improvements lies outside the CI. Also, a very simple thing, purely exploratory and descriptive, is just to note how much improvement for each item, corrected for pretest differences, such as (Posttest % correct - Pretest % correct)/(100% - Pretest % correct). Another is to get an effect size, such as phi coefficient, for each chi square from McNemar's test.
Cronbach's alpha is great for indicating internal homogeneity on a bunch of items presumed to measure the same thing (which is not the case here) but it won't help identify which items are changing more than others which is what is needed I believe. ========================== John W. Kulig, Ph.D. Professor of Psychology Coordinator, University Honors Plymouth State University Plymouth NH 03264 ========================== ----- Original Message ----- From: "Karl L Wuensch" <[email protected]> To: "Teaching in the Psychological Sciences (TIPS)" <[email protected]> Sent: Thursday, January 17, 2013 1:28:53 PM Subject: RE: Re:[tips] my crummy knowledge of stats My understanding of the intent of the analysis was to find items which were most affected, not a test for an omnibus effect across items. > ----- Original Message ----- > > From: "Annette Taylor"<[email protected]> > To: "Teaching in the Psychological Sciences > (TIPS)"<[email protected]> > Sent: Tuesday, January 15, 2013 6:21:42 PM > Subject: [tips] my crummy knowledge of stats > > I know this is a basic question but here goes: > > I have categorical data, 0,1 which stands for incorrect (0) or correct (1) on > a test item. > > I have 25 items and I have a pretest and a posttest and I want to know on > which items students improved significantly, and not just by chance. Just > eyeballing the data I can tell that there are some on which the improved > quite a bit, some not at all and some are someplace in the middle and I can't > make a guess at all. That is why we have statistics. Yeah! .... > hmmmm....bleh..... > > As far as I know, the best thing to do is a chi-square test for each of 25 > items; but of course that will mean that with a .05 sig level I will have at > least one false positive, maybe more, but most assuredly at least one. This > seems to be a risk. At any rate I can use SPSS and the crosstabs command > allow for calculation of the chi-square. > > I know that when I do planned comparisons with multiple t-tests, I can do a > Simes' correction in which I can rank order my final, obtained alphas, and > adjust for the number of comparisons and reject from the point from which the > obtained alpha failed to exceed the corrected-for-number-of-comps alpha. But > as far as I know, I cannot do that with 25 chi square tests. There is > probably some reason why I can no more do that, that relates to the reason > for why I cannot do 25 t-tests in this situation with categorical data. > > Is there a better way to answer my research question? I need a major > professor! Oh wait, that's me... drat! I need to hire a statistician. Oh > wait, I'd need $$ for that and I don't have any. So I hope tipsters can stand > in as a quasi-hired-statistician and help me out. > > Oh, I get the digest. I don't mind waiting until tomorrow or the next > day for a response, but a backchannel is [email protected] > > I will be at APS this year. Any other tipsters planning to be there? Let's > have a party! I'd love to put personalities to names. > > Thanks > > Annette > > Annette Kujawski Taylor, Ph. D. > Professor, Psychological Sciences > University of San Diego > 5998 Alcala Park > San Diego, CA 92110 > [email protected] --- You are currently subscribed to tips as: [email protected]. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13060.c78b93d4d09ef6235e9d494b3534420e&n=T&l=tips&o=23097 or send a blank email to leave-23097-13060.c78b93d4d09ef6235e9d494b35344...@fsulist.frostburg.edu --- You are currently subscribed to tips as: [email protected]. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13338.f659d005276678c0696b7f6beda66454&n=T&l=tips&o=23121 or send a blank email to leave-23121-13338.f659d005276678c0696b7f6beda66...@fsulist.frostburg.edu --- You are currently subscribed to tips as: [email protected]. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=23124 or send a blank email to leave-23124-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
