On Wed, 10 Feb 1999, Alexandre Soares wrote: > I've read the original articles mentioned in your message and also your > article on peer review (Harnad,S.,1998. The invisible hand of peer > review. Nature [online] (c. 5 Nov. 1998) > http://www.ecs.soton.ac.uk/~harnad/nature2.html). If you will, I > would like to add another possibility. Why not let the authors > themselves do the job of editors and publishers?
Some of your points are already considered and answered in the Nature online paper above. Here is a list of the problems with such proposals: (1) A large part of the function of refereed journal publication (including the level of the journal in question, in the quality hierarchy of journals) is to allow the otherwise overwhelmed reader to calibrate his reading in the enormous and growing mass of raw manuscripts. If all publication were self-publication -- even if accompanied by referee reports (and even if the referees were the appropriate experts) -- this would provide no calibration for the reader. Everything would be on a par, and one would have to plow through paper by paper, reading now not only the papers, but the referee reports, in the hope that it was a paper one would have wanted to read in the first place, had it been accepted by a refereed journal of a designated quality. (2) So the filtering and tagging function of refereeing would be lost. The correcting and improving function of the referee reports would also be compromised or lost, because authors could pick and choose whether they followed the referees' advice (having already picked and chosen the referees in the first place). I expressed some doubts about the self-policing capacities of human nature in the above article. Ask yourself how likely you yourself would be, under the weight of a heavy workload like that of most of us, to voluntarily elect to do further work on your paper at the behest of a referee, when you have the option of "publishing" it without doing so -- with an accompanying scolding public referee report being the only penalty? Surely there is a reason why self-monitoring has not been adopted as the (sole) means of quality-control in any other expert field of human endeavour either: At the population level (we are not speaking of the occasional individual saint, here) we must be protected from ourselves if quality standards are to be maintained. Otherwise there are just too many incentives to cut corners. (3) Public commentary (whether peer or non-peer, qualified or unqualified) is after-the-fact. It cannot serve as the advance quality filter. (4) Why should referees, with their loads, be any happier to lend their services to such an unpoliced, untagged system than readers would be to lend it their eyes? Today, when a referee is called upon to sacrifice some of his precious time to peer review, the call comes from the Editor of a journal of some distinction who has judged that referee suited to review that paper, and will see to it that the referee's recommendations -- if he judges them justified -- will be followed. In the proposed alternative, it is not a (presumably) qualified and respected and answerable editor who calls for the referee's services and will see to it that they are put to use, it is the author, who can then pick and choose. Will referees lend their time to such a self-policing system? Indeed, as referees are human too, will they then not rather ingratiate themselves to the great and good among authors, and leave the calls of lesser authors for review unanswered? These too are the wages of self-policing. (5) Last, what improvement is this over my own proposed alternative, which is to solicit expert opinion informally to the best of one's ability, do the recommended revisions one judges justified, and then archive the preprint publicly on the Web, not calling it a "refereed journal publication" (which it isn't) but an unrefereed preprint (which it is), while at the same time submitting it for refereeing to the appropriate peer-reviewed journal? The final, revised outcome of that process too (assuming it is successful) could then also follow the preprint onto the Web tagged as a refereed reprint. Volunteer peer commentary could be invited or spontaneously provided for both the preprint and the reprint, likewise on the Web. What would be lost in such a system that would be gained by the proposed self-policing system -- whose main casualty would be a peer-reviewed literature? > For example, the author > would write her/his paper and choose 3 referees randomly from a pool of > expert referees (see 1). Can a pool of qualified experts be constituted in advance for every potential specialty and subspecialty? (If so, why have journal editors not yet discovered and capitalized on this remarkable resource?) Are we more expert to pick the relevant experts who will adjudge our own expertise than qualified, respected disinterested experts, appointed by and answerable to the learned community would be? (Will referees respond to our call?) Are there any empirical data that confirm the assumption that anonymity need not be an option in order to elicit full, frank, competent reviews of the quality of a raw manuscript? > The referees would give their reports and the > author would make the changes which are agreed upon and the ones which > are not agreed would appear as appendix on the paper in a section called > "Referees comments" together with the author's answer. Is there any evidence that this is what quality control is or ought to be: The candidate picks and chooses what to fix, and the rest appears as a warning label? Would referees see this as a good use of their time? > Obviously, in > this section would appear general comments related to ideas (hypothesis, > methods, applicability), not structure (grammar, etc) of the paper. > There could be also comments from the referees about the importance of > the paper, and referee questions and authors answers (along the lines of > some proceedings of symposia). This sounds like peer commentary to me: Potentially useful, potentially even more-so on the Web; but is it a substitute for peer review? > The referees are therefore non-anonymous > to both authors and public (their addresses SHOULD appear in the paper > for contacts). Any data at all that (1) referees would go for this? or that (2) the results would be better/worse/no quality control? Or does non-anonymity have some sort of face validity? Should our names be published on the walls with our votes in elections too? > The author would then make the final layout of her/his paper (using any > of the almost infinite variety of available softwares) and would > "publish" the final version in her/his web site. Why not just "publish" it on the website and then go on to get it peer reviewed too? > In this way we achieve several highly desirable qualities : > > 1- authors would have the copyright of their work, (as suggested by > Bachrach S. et al. (1998). Intellectual Property: Who Should Own > Scientific Papers? Science 281 (5382):1459-1460. in > http://www.sciencemag.org/cgi/content/full/281/5382/1459), They could have that anyway, if they insisted -- and they will. Surely quality control need not be sacrificed in order to retain ownership of one's intellectual property. (That Bargain would be a Faustian one indeed!) > 2- publication would be free of costs for both authors and readers, > rapidly "published" and easily accessible and distributed, It would also be free of both quality control and the signposting that went with it, to guide, respectively, the author and the reader. > and > > 3- the hand of the referees would be "visible", eliminating "invisible > friends" and "enemies". On the assumption that visibility is an end in itself and would be at least as effective as optional anonymity in eliciting valid feedback... > Rejected papers Rejected? Who would be rejecting? Referees only recommend: In your system it is the author who (self)-accepts or rejects! > could be re-submitted with new randomly choosen > referees, In other words, one could keep shopping for referees till one found more favorable ones. (An excellent formula for getting statistical significance too: keep calculating the cumulative F ratio and raising the N by gathering more data till you hit a point where p happens to peak momentarily above .05, and then promptly stop testing and publish! Our journals would be filled with a lot more significant effects then, assuming referees could eventually be found to endorse them...) > but the reports of all referees SHOULD appear. Even in the > extreme case that the paper is rejected several times, the author > her/himself would have the final decision on "publishing" or not, as > long as she/he would always include the veridict of the referees. The > wide public would then judge whether to trust or not in the results of > the paper. If they had the time to first wade through all those papers and referee reports. (It almost makes you want to bring back Editors to do that for you...) > Notes: > > (1) This pool of referees could be organized by the appropriate > scientific society and could be available in the web in a especial > software. This can and will be done anyway, but not to put at the beck and call of a self-policing system, but in the service of classical peer review, upgraded and streamlined for online implementation. (This was discussed in the Nature online article.) > Referees would have numbers and each time an author would > enquire for referees, his name and the title of the article would be > registered and 3 (or more) referee names would be returned to the author > after automatic referee number randomisation and combination choice. Some parametric version of this would be fine, but in the heads of a respected and capable Editor and Editorial Board, not in the hands of the authorship at large. > Referees could establish in this software their availability (for > example, no more than 4 articles per month) and this would be > considered by the software during referee choice. Each of these "pool > of referees" would somehow act as a feedback control center for how many > times an article was "submitted" and the information would be freely > accessed by any interested person. All reasonable suggestions, already being implemented, but in the service of peer review, not self-policing. -------------------------------------------------------------------- Stevan Harnad [email protected] Professor of Cognitive Science [email protected] Department of Electronics and phone: +44 1703 592-582 Computer Science fax: +44 1703 592-865 University of Southampton http://www.ecs.soton.ac.uk/~harnad/ Highfield, Southampton http://www.princeton.edu/~harnad/ SO17 1BJ UNITED KINGDOM ftp://ftp.princeton.edu/pub/harnad/
