I'd like to get Tipsters' opinion of a new analysis of Bem's data that focuses on whether a series of experiments report more statistically significant results than expected given the reported effect sizes and level of power associated with each of the experiments. The reference is given below and the article is available via Springerlink (members of the psychonomic society have free access and nonmembers should have access through their institution's library). See:
Francis, Gregory (2012, online). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review. 1-6, Url: http://dx.doi.org/10.3758/s13423-012-0227-9 Doi: 10.3758/s13423-012-0227-9 Abstract: Empirical replication has long been considered the final arbiter of phenomena in science, but replication is undermined when there is evidence for publication bias. Evidence for publication bias in a set of experiments can be found when the observed number of rejections of the null hypothesis exceeds the expected number of rejections. Application of this test reveals evidence of publication bias in two prominent investigations from experimental psychology that have purported to reveal evidence of extrasensory perception and to indicate severe limitations of the scientific method. The presence of publication bias suggests that those investigations cannot be taken as proper scientific studies of such phenomena, because critical data are not available to the field. Publication bias could partly be avoided if experimental psychologists started using Bayesian data analysis techniques. The other series of studies referred to is Jonathan Schooler's verbal overshadowing effect which was one of sets of results focused on by Jonah Lehrer in his "The New Yorker" article which can be found here: http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer Jonathan Schooler found that the effect size for verbal overshadowing effect was reduced over time until it was very difficult to replicate at all. Schooler called this the "decline effect", a finding the J.B. Rhine first ifentified in his studies of PSI/parapsychology. Schooler reflects on this in an article in Nature; see: http://www.nature.com/news/2011/110223/full/470437a.html One question that arises is whether uncritical acceptance of meta-analysis might lead to wrong conclusions if one does not do analyses like that suggested by Francis and which are inspired by the work of Ioannidis; see: http://www.ncbi.nlm.nih.gov/pubmed/18633328?dopt=Abstract and http://www.ncbi.nlm.nih.gov/pubmed/17715249?dopt=Abstract I wonder what such an analysis and review of DiCara's work, as shown in Figure 1 of Dworkin and Miller 1986, would reveal? -Mike Palij New York University [email protected] --- You are currently subscribed to tips as: [email protected]. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=16265 or send a blank email to leave-16265-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
