On Tue, 08 Nov 2011 06:25:24 -0800, Michael Britt wrote: >This article looks timely: >http://pss.sagepub.com/content/early/2011/10/17/0956797611417632.full
The article that Michael B. links to is the following: Joseph P. Simmons, Leif D. Nelson, and Uri Simonsohn False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant Psychological Science October 2011 0956797611417632, first published on October 17, 2011 doi:10.1177/0956797611417632 NOTE: A more recent version of this article was published on - Nov 7, 2011 Members of APS receive weekly listings of articles in the APS journals and when the one for Psychological Science came this morning, this was the first article I read (though I was tempted to read the working memory article first). The authors examine how researchers may engage in certain activities that maximize the probability of getting a statistically significant result but, it is argued, that this process actually produces more false positive or Type I errors. The authors use the phrase of "degrees of freedom" in two different ways (1) the usual sense associated with a statistical test, and (2) the number of ways that one can use variables, conduct analyses, and other "methodological" procedures that can be used to increase the likelihood of a statistically significant result and, by extension, increase the Type I/false positive rate. I am not totally convinced by the argument but I have only read the article once. Nonetheless, the authors propose 6 rules for researchers or authors such as (quoting from the article):: |1. Authors must decide the rule for terminating data |collection before data collection begins and report |this rule in the article. |2. Authors must collect at least 20 observations per |cell or else provide a compelling cost-of-datacollection |justification. |3. Authors must list all variables collected in a study |4. Authors must report all experimental conditions, |including failed manipulations. |5. If observations are eliminated, authors must also |report what the statistical results are if those |observations are included. |6. If an analysis includes a covariate, authors must |report the statistical results of the analysis without |the covariate. There also are some suggested rules for journal reviewers but I'll leave it to the interested reader to read them. I think that a number of points made by Simmons et al are valid but I think that their presentation is lacking. I think that their recommendations are somewhat reasonable but arbitrary (e.g., rule 2 above specifies 20 observations per cell but shouldn't one conduct an a priori power analysis to determine what the minimum sample size should be to detect a specific effect at specific levels of power?). I'd like to see what others think of the article. -Mike Palij New York University [email protected] --- You are currently subscribed to tips as: [email protected]. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=13987 or send a blank email to leave-13987-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
