CALL: CORTEX FORUM ON PEER-REVIEW Cortex regularly publishes a section called Discussion Forum on science-related issues (see a recent one on The Use and Misuse of Impact Factors in Cortex 37, Issue 4, 2001; also available on the web, for both subscribers and non subscribers alike, at: http://www.cortex-online.org ).
The next Forum will be on The Role of Peer-Review in Neuropsychology. A preliminary draft of the accompanying editorial is attached for your perusal. If you wish to contribute to the discussion on any issue relevant to peer-reviewing, please let us know ([email protected]) by March 1st. Each contribution should be no longer than 1000 words plus 1 figure or table and no more than ten references (see previous Forum for examples). Deadline for final submission is: March 29th, 2002. Further information can be obtained from the editors: Sergio Della Sala: [email protected] Jordan Grafman: [email protected] Contributions should be sent to: Cortex Department of Psychology William Guild Building University of Aberdeen Aberdeen AB24 2UB Refereeing mortus est, vivat refereeing Sergio Della Sala and Jordan Grafman We are all well aware of some of the shortcomings in the peer-review process of scientific work. The system has been revealed to be riddled with prejudice, subject to nepotism (Forsdyke, 1993, Calza and Gerbisa, 1995; Perez-Enciso, 1995; Wenner s and Wold, 1997), sexism (Wallston and OLeary, 1982; Wenner s and Wold, 1997), and influenced by the national language of the authors (Bakewell, 1992; Nylenna et al., 1994), not to mention the problems of broken confidentiality (Maddox, 1984) and conflict of interest (sometimes financial; sometimes scientific competition). Careful analyses of the review process also showed a very poor inter-referee reliability (Zuckerman and Merton, 1971; Cole et al., 1981) - indeed little greater than chance (Gordon, 1977; Inglefinger, 1974; Rothwell and Martyn, 2000) - a clear association with the reviewers experience and age (younger reviewers producing more thorough reviews, Nylenna et al., 1994), a significant tendency to favour positive findings (Mahoney, 1977) and a proclivity towards projects and findings in line with the referees own ideas or their "knee-jerk" adherence to current theoretical dogma (Ernst et al., 1992), a phenomenon labelled "confirmatory bias" (Mahoney, 1977). The process is hampered by a further bias: the preference for "normal science" (Crawford, 1998). Refereeing tends to favour straightforward, uncontroversial, even prosaic science, over more venturesome and speculative arguments (Allen and Grant, 1998). Moreover, detecting plagiarism is either based on a suspicion or on a systematic review of all submissions to a journal - a trying and lengthy process we rarely bother undertaking (Manweel and Baker, 1982; Marshall, 1998). Time is a crucial factor. The mean time spent on reviewing a manuscript is 1.5 hours (Lock and Smith, 1990; Nylenna et al., 1994) with an average of one manuscript per month (Yankauer, 1990). Why should reviewers put more effort into a process which carries no fame, no money, very little extra knowledge and for which they bear no responsibility? We all work for the benefit of the publishers; but in all other instances of this slavish subjugation (publishing papers, reviewing books) our exacting ego is rewarded, which is hardly the case in the anonymity of the refereeing process. The problem is far from new (Huxley, 1901) and it has been discussed before (Wilson, 1978; Peters and Ceci, 1980; Lock, 1985), though not so much within Neuropsychology. The result of the reviewing process is unsatisfying: referees are slow, hurried, often inaccurate (Horton, 1998) and sometimes cantankerous. Yet the process warrants good science, no matter the means of publication (paper or web) or whether the manuscripts are meant for restricted audience or free access. We are sure that we could all amuse one another with endless anecdotes on errors and misjudgements by referees and editors. Some experimental evidence of the unfairness of peer-review comes from the provocative study of Peter and Ceci (1982). They selected 12 psychology articles by prestigious investigators and institutions and re-submitted them (changing the names of the authors and using fictitious affiliations) to the same 12 top American journals which had originally published them some two years before. Three of these re-submissions were detected as spoofs. Of the remaining nine, eight were rejected, not because of a feeling of dej vu (lack of originality was never mentioned), but on the basis of one or another major flaw in the study design. Surely a matter for more than mere amusement. Furthermore, in one journal or another, or even on the Internet, almost everything is eventually published. Indeed, the few studies available show that over 90% of the papers rejected by one journal are eventually, and unaltered, published, not necessarily in a lower status journal. The reviewing system appears to influence where but not what should be published (Wilson, 1978): the mountain labours brings forth ... an undignified shuffling of manuscripts. One attempt to blatantly and overtly deal with this phenomena is to create a central repository of articles with cursory reviews thus placing the burden of deciding whether a paper or study is important to the consumer Let the buyer beware! This musing is to open a debate and to make a (preliminary) proposal. We are convinced that peer-review is central to scientific credibility. However, as it stands the process is far from watertight. Is there any way we can improve it by suggesting any modification, either radical or minimal? Time is ripe for such a discussion to be launched (see the JAMA and BMJ four congresses on peer review in biomedical publication: www.jama-peer.org). One possible alternative is to substitute referees with sponsors, chosen by the authors, who overtly review and promote the papers (with their names as sponsors on it) they regard worthy of publication prior to submission. The manuscript would therefore be submitted to the journal ready to be published, although the editorial board could still reject it on scientific or non-scientific grounds (e.g., inappropriateness for the journal audience). On the one hand this will make the refereeing process overt and rewarding for the reviewers. On the other hand, it will require more responsibility, because the sponsors names would be on it. Some problems are however immediately apparent with this kind of review process: nepotism, favouritism towards better known authors, over-commitment of the most popular sponsors, awkwardness in requesting the review from sponsors, and, more dangerously, scientific or personal covert blackmailing. All true, though not dissimilar from the imperfections of the current system. Would this proposal, all things considered, improve the present cumbersome and not indisputable process? If not, does anybody have a better idea? Is it even worth thinking about possible modifications of the present system? References Allen L, Grant J. Funding innovations. Wellcome News, 1998, 16: 24-25. Bakewell D. French research - publish in English or perish. Nature, 1992, 356:648. Calza L, Garbisa S. Italian Professorships. Nature, 1995, 374: 492. Cole S, Cole JR, Simon GA. Chance and consensus in peer-review. Science, 1981, 214: 881-886. Crawford E. Nobel: Always the winners, never the losers. Science, 1998, 282: 1256-1257. Ernst E, Resch KL, Uher EM. Reviewer bias. Annals of Internal Medicine, 1992. 116: 958. Forsdyke DR. On giraffes and peer-review. FASEB Journal, 1993. 7:619-621. Gordon M. Evaluating the evaluators. New Scientist, 1977. 73: 342-343, Horton R. Disseminating systematic reviews:working with medical journals. Paper presented at the Symposium "Systematic Reviews: Beyond the Basic", St.John College, Oxford, 8-9 January, 1998. Huxley L. Life and letters of Thomas Henry Huxley. New York: D. Appleton and Co.,1901, p.105. Inglefinger FJ. Peer review in biomedical publication. American Journal of Medicine, 1974, 56: 686-692, Lock S. A difficult balance. Editorial peer review in Medicine. The Nuffield Provincial Hospital Trust, 1985. Lock S, Smith J. What do peer-reviewers do? Journal of the American Medical Association, 1990, 263:1341-1343. Maddox J. Privacy and the peer-review system. Nature, 1984. 312: 497. Mahoney MJ. Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1977. 1: 161-175, Manwell C, Baker MA. Reform peer review. The Behavioral and Brain Science, 1982, 5: 211-225. Marshall E. The Internet: A powerful tool for plagiarism sleuths. Science, 1998, 279: 474. Nylenna M, Riis P, Karlsson Y. Multiple blinded reviews of the two manuscripts - effects of referee characteristics and publication language. Journal of the American Medical Association, 1994, 272: 149-151. Perez-Enciso M. Spanish Practices. Nature, 1995, 378: 760. Peters DP, Ceci, SJ. A manuscript masquerade. How well does the review process work? The Sciences, 1980, 20: 16-19. Peters DP, Ceci SJ. Peer-review practices of psychological journals: The fate of published articles, submitted again. The Behavioral and Brain Science, 1982. 5: 187-195. [See also the Commentary pp. 196-255]. Rothwell PM, Martyn CN. Reproducibility of peer review in clinical neuroscience. Is agreement between reviewers any greater than would be expected by chance alone? Brain, 123: 1964-1969. Wallston Strudler B, O¹Leary VE. Sex makes a difference: differential perceptions of women and men. Review of Personality and Social Psychology, 1982, 2: 9-43. Wennerå s C, Wold A. Nepotism and sexism in peer-review. Nature, 1997, 387: 341-343. Wilson JD. Peer review and publication. Journal of Clinical Investigation, 1978, 58: 1697-1701. Yankauer A. Who are the peer reviewers and how much do they review? Journal of the American Medical Association, 1990, 263: 1338-1340.
