Hi, All -- I need help. (Our library doesn't have access to the most recent 12 months of Nat. Neurosci.) I read the piece in the Guardian and read Cohen's paper recommended by Mike, but I just can't understand this (from the piece in the Guardian):
"The impact of combining low power and low pre-study odds has important consequences for the likelihood that the research finding is actually true. In our analysis, we show that for exploratory studies with an average 20% power, together with average one in four pre-study odds that the effect being sought is actually true, the likelihood that any claimed effect (based on passing a conventional level of statistical significance) actually is true is only 50%. That's a 50/50 chance that any positive effects are spurious. For a confirmatory study with four to one pre-study odds, the chance that any positive effects are spurious is reduced to 25%." Can someone help me understand? This is how I'm thinking (and how I was taught): Power is the probability of rejecting a false null, as I understand it (1-beta, where beta is p(miss)). If my data suggest I should reject the null, why is low power a concern? If I *fail* to reject then the first thing I look at is power, but if I can reject with confidence, then I'm not concerned about the power of the test. Can someone explain why I'm not thinking about this in the right way? If we were talking effect sizes, then I can understand why small effect sizes would mitigate the importance of the result (and make is less likely to be replicable and/or more likely to be the result of an alpha-error). But low power increasing the likelihood of an alpha-error? I really don't understand. I've learned much from you all and hope to keep doing so, here: can someone clue me in? I don't want to be teaching the wrong stuff.... Thanks, m -- Marc Carter, PhD Associate Professor of Psychology Chair, Department of Behavioral and Health Sciences College of Arts & Sciences Baker University -- > -----Original Message----- > From: Jim Clark [mailto:[email protected]] > Sent: Wednesday, April 10, 2013 12:15 PM > To: Teaching in the Psychological Sciences (TIPS) > Subject: Re: [tips] Why Neuroscience Research Sucks > > Hi > > I would say the problem is less with individual studies not having > enough power than with (a) the difficulty of getting replications > published, and (b) premature conclusion drawing by researchers (often > aided and abetted by their institution's communication / advertising > departments) who cannot wait for sufficient replications to be > confident about the effects. > > It seems pretty clear that insisting on increases in power will > decrease the number of published studies, including replications. Not > everyone will have the resources to have the perhaps very large sample > size necessary to carry out and publish "ideal" studies. But what we > need is more replication rather than less. > > I remember with fondness the old J of Experimental Psychology and J of > Verbal Learning and Verbal Behavior, which would contain numerous > studies, some of which would be only a few pages long as the important > info was Methods and Results. And one still finds these kinds of > journals in the natural sciences, perhaps accounting for the much > better acceptance rates (i.e., lower rejection rates) than psychology > journals. > > Science is a collective effort, but psychology and affiliated > disciplines appear to have turned it too much towards an emphasis on > individual achievement (where individual includes research teams). > > Take care > Jim > > James M. Clark > Professor & Chair of Psychology > [email protected] > Room 4L41A > 204-786-9757 > 204-774-4134 Fax > Dept of Psychology, U of Winnipeg > 515 Portage Ave, Winnipeg, MB > R3B 0R4 CANADA > > > >>> Michael Palij <[email protected]> 10-Apr-13 7:20 AM >>> > A paper published in Nature Reviews Neuroscience reports a meta- > analysis of neuroscience research studies and, in keeping with old > problems with experimental designs used by people who perhaps don't > know what they're doing (e.g., failing to appreciate the role of > statistical power), report that they find (a) low levels of statistical > power (around .20), > (b) exaggerated effect sizes, and (c) lack or reproducibility. > But don't take my word for it, here is a link to research article: > http://www.nature.com/nrn/journal/vaop/ncurrent/full/nrn3475.html > > NOTE: you'll need to use you institution's library to access the > article. > > There are popular media articles that focus on this article which may > be useful in classes such as critical thinking and maybe even > neuroscience; see: > http://www.guardian.co.uk/science/sifting-the- > evidence/2013/apr/10/unreliable-neuroscience-power-matters > > Jack Cohen pointed out some of the problems back in his 1962 review as > well as updated them in subsequent publications; see: > http://classes.deonandan.com/hss4303/2010/cohen%201992%20sample%20size. > pdf > > Of course, this is problem of researcher education, the politics of > funding research and publishing, and perhaps sociological factors, such > trying to appear more "scientific" -- focusing on brain is after all > more "scientific" than focusing on just behavior or the mind. > > -Mike Palij > New York University > [email protected] > > --- > You are currently subscribed to tips as: [email protected]. > To unsubscribe click here: > http://fsulist.frostburg.edu/u?id=13251.645f86b5cec4da0a56ffea7a891720c > 9&n=T&l=tips&o=24913 > or send a blank email to leave-24913- > [email protected] > > > --- > You are currently subscribed to tips as: [email protected]. > To unsubscribe click here: > http://fsulist.frostburg.edu/u?id=13029.76c7c563b32ad9d8d09c72a2d17c90e > 1&n=T&l=tips&o=24923 > or send a blank email to leave-24923- > [email protected] The information contained in this e-mail and any attachments thereto ("e-mail") is sent by Baker University ("BU") and is intended to be confidential and for the use of only the individual or entity named above. The information may be protected by federal and state privacy and disclosures acts or other legal rules. If the reader of this message is not the intended recipient, you are notified that retention, dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error please immediately notify Baker University by email reply and immediately and permanently delete this e-mail message and any attachments thereto. Thank you. --- You are currently subscribed to tips as: [email protected]. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=24924 or send a blank email to leave-24924-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
