I think step 1 would still be an answer to the question "Was the change large enough to be considered "real" or statistically reliable."
Step 2 would be to ask which of the statistically reliable changes were largest . . . perhaps an estimate of effect size and rank ordering of the questions. Outside of Cohen and others categories of effect sizes as small, medium, or large, is there any quantitative method for comparing the relative sizes of effects? _____________________________________________ Claudia J. Stanny, Ph.D. Director Center for University Teaching, Learning, and Assessment Associate Professor NSF UWF Faculty ADVANCE Scholar School of Psychological and Behavioral Sciences University of West Florida 11000 University Parkway Pensacola, FL 32514 – 5751 Phone: (850) 857-6355 (direct) or 473-7435 (CUTLA) [email protected] CUTLA Web Site: http://uwf.edu/cutla/ Personal Web Pages: http://uwf.edu/cstanny/website/index.htm On Thu, Jan 17, 2013 at 12:28 PM, Wuensch, Karl L <[email protected]> wrote: > My understanding of the intent of the analysis was to find items > which were most affected, not a test for an omnibus effect across items. > > ----- Original Message ----- > > > > From: "Annette Taylor"<[email protected]> > > To: "Teaching in the Psychological Sciences > > (TIPS)"<[email protected]> > > Sent: Tuesday, January 15, 2013 6:21:42 PM > > Subject: [tips] my crummy knowledge of stats > > > > I know this is a basic question but here goes: > > > > I have categorical data, 0,1 which stands for incorrect (0) or correct > (1) on a test item. > > > > I have 25 items and I have a pretest and a posttest and I want to know > on which items students improved significantly, and not just by chance. > Just eyeballing the data I can tell that there are some on which the > improved quite a bit, some not at all and some are someplace in the middle > and I can't make a guess at all. That is why we have statistics. Yeah! .... > hmmmm....bleh..... > > > > As far as I know, the best thing to do is a chi-square test for each of > 25 items; but of course that will mean that with a .05 sig level I will > have at least one false positive, maybe more, but most assuredly at least > one. This seems to be a risk. At any rate I can use SPSS and the crosstabs > command allow for calculation of the chi-square. > > > > I know that when I do planned comparisons with multiple t-tests, I can > do a Simes' correction in which I can rank order my final, obtained alphas, > and adjust for the number of comparisons and reject from the point from > which the obtained alpha failed to exceed the corrected-for-number-of-comps > alpha. But as far as I know, I cannot do that with 25 chi square tests. > There is probably some reason why I can no more do that, that relates to > the reason for why I cannot do 25 t-tests in this situation with > categorical data. > > > > Is there a better way to answer my research question? I need a major > professor! Oh wait, that's me... drat! I need to hire a statistician. Oh > wait, I'd need $$ for that and I don't have any. So I hope tipsters can > stand in as a quasi-hired-statistician and help me out. > > > > Oh, I get the digest. I don't mind waiting until tomorrow or the next > > day for a response, but a backchannel is [email protected] > > > > I will be at APS this year. Any other tipsters planning to be there? > Let's have a party! I'd love to put personalities to names. > > > > Thanks > > > > Annette > > > > Annette Kujawski Taylor, Ph. D. > > Professor, Psychological Sciences > > University of San Diego > > 5998 Alcala Park > > San Diego, CA 92110 > > [email protected] > > > --- > You are currently subscribed to tips as: [email protected]. > To unsubscribe click here: > http://fsulist.frostburg.edu/u?id=13060.c78b93d4d09ef6235e9d494b3534420e&n=T&l=tips&o=23097 > or send a blank email to > leave-23097-13060.c78b93d4d09ef6235e9d494b35344...@fsulist.frostburg.edu > > --- > You are currently subscribed to tips as: [email protected]. > To unsubscribe click here: > http://fsulist.frostburg.edu/u?id=13144.1572ed60024e708cf21c4c6f19e7d550&n=T&l=tips&o=23121 > or send a blank email to > leave-23121-13144.1572ed60024e708cf21c4c6f19e7d...@fsulist.frostburg.edu > --- You are currently subscribed to tips as: [email protected]. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=23123 or send a blank email to leave-23123-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
