Hi I would consider an alternative approach. For each ITEM, calculate the percentage of students who passed that item. Then do a paired-difference test of significance for pre vs post with items as the random factor (i.e., "subjects"). This will tell you whether there was an overall change. Then you might calculate a change score and a confidence interval about the mean change score to identify items that scored unexpectedly high or low.
Or you could do some analysis just of post-test scores for items. What follows would depend on the nature of the items. If objective questions (TF, MultChoice), then determine the chance probability of correct and confidence intervals about that value to see which items fall outside the CI. If not objective but just scored 0 and 1, then perhaps determine one-sided CI about 0 (assuming such a statistic exists)? Or if you could a priori (i.e., without looking at results or using factor analysis or something like that) group some of the items together (e.g., from same chapter, same concept, ...), you could produce a score rather than a 0/1 value and do some more standard statistical tests, such as within-s factorial (time, scale) or simple effects of time within scale. Take care Jim James M. Clark Professor & Chair of Psychology [email protected] Room 4L41A 204-786-9757 204-774-4134 Fax Dept of Psychology, U of Winnipeg 515 Portage Ave, Winnipeg, MB R3B 0R4 CANADA >>> John Kulig <[email protected]> 16-Jan-13 9:24 AM >>> Hi Annette Perhaps McNemar's Test for significance of changes, for dichotomous data. For each item, set up a table that looks like a 2*2 chi square but has "pretest" and "post-test" as variables (in texts its usually labelled "before" and "after") . Posttest - + + A B Pretest - C D So every S appears as one count in the table and A+B+C+D = total N. But the key cells are A and D. Since (A+D) equals the people who changed, we expect half of (A+D) in A and D if the Null is true. The frequency expected for both A and D is (A+D)/2 ... Then you just do a chi square on those two cells. The formula simplifies to chi square = (A - D)squared / (A + D) with df = 1. The Wikipedia link is http://en.wikipedia.org/wiki/McNemar%27s_test, and I just noticed it reorders the rows and columns so that B and C are the "change" variables. It also reminds us about Yates correction .. Unfortunately, I don't use SPSS much these days so I don't know how to find it or code the variables for the chi square! Not sure about the chance issue. The p values may be all you need ,.. but you may want a correction for chance?? Others will know more .... ========================== John W. Kulig, Ph.D. Professor of Psychology Coordinator, University Honors Plymouth State University Plymouth NH 03264 ========================== ----- Original Message ----- From: "Annette Taylor" <[email protected]> To: "Teaching in the Psychological Sciences (TIPS)" <[email protected]> Sent: Tuesday, January 15, 2013 6:21:42 PM Subject: [tips] my crummy knowledge of stats I know this is a basic question but here goes: I have categorical data, 0,1 which stands for incorrect (0) or correct (1) on a test item. I have 25 items and I have a pretest and a posttest and I want to know on which items students improved significantly, and not just by chance. Just eyeballing the data I can tell that there are some on which the improved quite a bit, some not at all and some are someplace in the middle and I can't make a guess at all. That is why we have statistics. Yeah! .... hmmmm....bleh..... As far as I know, the best thing to do is a chi-square test for each of 25 items; but of course that will mean that with a .05 sig level I will have at least one false positive, maybe more, but most assuredly at least one. This seems to be a risk. At any rate I can use SPSS and the crosstabs command allow for calculation of the chi-square. I know that when I do planned comparisons with multiple t-tests, I can do a Simes' correction in which I can rank order my final, obtained alphas, and adjust for the number of comparisons and reject from the point from which the obtained alpha failed to exceed the corrected-for-number-of-comps alpha. But as far as I know, I cannot do that with 25 chi square tests. There is probably some reason why I can no more do that, that relates to the reason for why I cannot do 25 t-tests in this situation with categorical data. Is there a better way to answer my research question? I need a major professor! Oh wait, that's me... drat! I need to hire a statistician. Oh wait, I'd need $$ for that and I don't have any. So I hope tipsters can stand in as a quasi-hired-statistician and help me out. Oh, I get the digest. I don't mind waiting until tomorrow or the next day for a response, but a backchannel is fine. [email protected] I will be at APS this year. Any other tipsters planning to be there? Let's have a party! I'd love to put personalities to names. Thanks Annette Annette Kujawski Taylor, Ph. D. Professor, Psychological Sciences University of San Diego 5998 Alcala Park San Diego, CA 92110 [email protected] --- You are currently subscribed to tips as: [email protected]. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13338.f659d005276678c0696b7f6beda66454&n=T&l=tips&o=23044 or send a blank email to leave-23044-13338.f659d005276678c0696b7f6beda66...@fsulist.frostburg.edu --- You are currently subscribed to tips as: [email protected]. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13251.645f86b5cec4da0a56ffea7a891720c9&n=T&l=tips&o=23065 or send a blank email to leave-23065-13251.645f86b5cec4da0a56ffea7a89172...@fsulist.frostburg.edu --- You are currently subscribed to tips as: [email protected]. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=23069 or send a blank email to leave-23069-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
<<attachment: Jim_Clark.vcf>>
