There are also ways to compare whether two r effect sizes are significantly different, and can be found in summaries of meta-analysis techniques ... as this is often done when comparing the effect sizes of two studies. But my preference is to do something simple and descriptive at first (like rank ordering) and noting which are highest and lowest, as this is a great way to spot trends, generate hypotheses, discover truth etc. And to save the formal inferential stats for publication, when you convince others of the correctness of your conclusions arrived at by poking through the data .. oops! imagine putting that in a methods text!
========================== John W. Kulig, Ph.D. Professor of Psychology Coordinator, University Honors Plymouth State University Plymouth NH 03264 ========================== ----- Original Message ----- From: "Claudia Stanny" <[email protected]> To: "Teaching in the Psychological Sciences (TIPS)" <[email protected]> Sent: Thursday, January 17, 2013 1:41:34 PM Subject: Re: Re:[tips] my crummy knowledge of stats I think step 1 would still be an answer to the question "Was the change large enough to be considered "real" or statistically reliable." Step 2 would be to ask which of the statistically reliable changes were largest . . . perhaps an estimate of effect size and rank ordering of the questions. Outside of Cohen and others categories of effect sizes as small, medium, or large, is there any quantitative method for comparing the relative sizes of effects? _____________________________________________ Claudia J. Stanny, Ph.D. Director Center for University Teaching, Learning, and Assessment Associate Professor NSF UWF Faculty ADVANCE Scholar School of Psychological and Behavioral Sciences University of West Florida 11000 University Parkway Pensacola, FL 32514 – 5751 Phone: (850) 857-6355 (direct) or 473-7435 (CUTLA) [email protected] CUTLA Web Site: http://uwf.edu/cutla/ Personal Web Pages: http://uwf.edu/cstanny/website/index.htm On Thu, Jan 17, 2013 at 12:28 PM, Wuensch, Karl L < [email protected] > wrote: My understanding of the intent of the analysis was to find items which were most affected, not a test for an omnibus effect across items. > ----- Original Message ----- > > From: "Annette Taylor"< [email protected] > > To: "Teaching in the Psychological Sciences > (TIPS)"< [email protected] > > Sent: Tuesday, January 15, 2013 6:21:42 PM > Subject: [tips] my crummy knowledge of stats > > I know this is a basic question but here goes: > > I have categorical data, 0,1 which stands for incorrect (0) or correct (1) on > a test item. > > I have 25 items and I have a pretest and a posttest and I want to know on > which items students improved significantly, and not just by chance. Just > eyeballing the data I can tell that there are some on which the improved > quite a bit, some not at all and some are someplace in the middle and I can't > make a guess at all. That is why we have statistics. Yeah! .... > hmmmm....bleh..... > > As far as I know, the best thing to do is a chi-square test for each of 25 > items; but of course that will mean that with a .05 sig level I will have at > least one false positive, maybe more, but most assuredly at least one. This > seems to be a risk. At any rate I can use SPSS and the crosstabs command > allow for calculation of the chi-square. > > I know that when I do planned comparisons with multiple t-tests, I can do a > Simes' correction in which I can rank order my final, obtained alphas, and > adjust for the number of comparisons and reject from the point from which the > obtained alpha failed to exceed the corrected-for-number-of-comps alpha. But > as far as I know, I cannot do that with 25 chi square tests. There is > probably some reason why I can no more do that, that relates to the reason > for why I cannot do 25 t-tests in this situation with categorical data. > > Is there a better way to answer my research question? I need a major > professor! Oh wait, that's me... drat! I need to hire a statistician. Oh > wait, I'd need $$ for that and I don't have any. So I hope tipsters can stand > in as a quasi-hired-statistician and help me out. > > Oh, I get the digest. I don't mind waiting until tomorrow or the next > day for a response, but a backchannel is [email protected] > > I will be at APS this year. Any other tipsters planning to be there? Let's > have a party! I'd love to put personalities to names. > > Thanks > > Annette > > Annette Kujawski Taylor, Ph. D. > Professor, Psychological Sciences > University of San Diego > 5998 Alcala Park > San Diego, CA 92110 > [email protected] --- You are currently subscribed to tips as: [email protected] . To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13060.c78b93d4d09ef6235e9d494b3534420e&n=T&l=tips&o=23097 or send a blank email to leave-23097-13060.c78b93d4d09ef6235e9d494b35344...@fsulist.frostburg.edu --- You are currently subscribed to tips as: [email protected] . To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13144.1572ed60024e708cf21c4c6f19e7d550&n=T&l=tips&o=23121 or send a blank email to leave-23121-13144.1572ed60024e708cf21c4c6f19e7d...@fsulist.frostburg.edu --- You are currently subscribed to tips as: [email protected] . To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13338.f659d005276678c0696b7f6beda66454&n=T&l=tips&o=23123 (It may be necessary to cut and paste the above URL if the line is broken) or send a blank email to leave-23123-13338.f659d005276678c0696b7f6beda66...@fsulist.frostburg.edu --- You are currently subscribed to tips as: [email protected]. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=23126 or send a blank email to leave-23126-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
