> Bruce Edmonds <[email protected]> > > A key difference between Stevan's and my proposals is that in his one > either gets a generic quality constraint on the flow of information > (via journals) or none at all (using public pre-print archives). In my > proposal papers are `tagged' with relatively rich judgemental > information so that the reader can determine the selection of > information to suit their own needs - mixing judgemental and > content-based information. Readers could emulate the journal system by > choosing selection criteria so as to cover a single topic area allowing > only papers of high general quality, but they would have the choice!
As we have been back and forth on this before, let me just repeat that your proposal does not, as it suggests, capture the classical quality control and tagging system as a special case. It loses it completely, and offers no tested control/tag system in its place, but rather an untested one with a number of prima facie reasons, described repeatedly, to expect that it will fail (both as a way of eliciting reliable peer review and as a way of filtering and signposting the literature for the user). > This early and inflexible constraint on the flow of papers from author > to reader that occurs in the journal system was necessary because > publishing used to be costly - this is no longer the case. Reviewers > already produce this information - why not release it to readers so > that they can determine their own selection criteria? Same story: They CAN release it to readers (by publicly archiving their papers), but with no need either to sacrifice classical peer review in exchange or to swap it for an untested alternative with many prima facie problems. As repeatedly stated, virtually all the advantages you cite are available without following your proposal, simply by public self-archiving of papers (and, if desired, of referee reports too). > Yes, I accept that a system which involves a dialogue between author > and journal is different [from] a one-shot review. However I disagree on > the assessment of the overall impact of this change. The one-shot > system would encourage authors to get feedback from colleagues *before* > submission, thus saving editorial staff a *lot* of time. Most authors already get advance feedback, by circulating their papers to experts for feedback before submitting them for publication. Self-archiving will increase the scope of this presubmission feedback. Again, you have it without any need for the system you propose. Submission/refereeing/revision/re-refereeing/revision... is not aptly glossed as "a dialogue between author and journal"... > Review boards, could unilaterally decide to review an on-line paper, > they would not have to wait for submitted papers. Thus there is no > danger of authors being afraid of the process - they would have no > choice but to get used to it! This is a non sequitur, and based on an unrealistic idea about the journeyman time available to editorial boards and referees, particularly if they are ever faced with a Global Netful of unfiltered drafts. Some authors might be happy with exhibitionism and taking lumps; but the ones really at risk are the referees -- and the readers. > The balance of work for reviewers is far from obvious. Not obvious? Currently I review only what an expert editor asks me, based on my expertise, to review, and with the understanding that the author is answerable to the review process. Who would review what if there were just a Netful of raw manuscripts waiting for all comers? And what is the point of reviewing if there is no answerability beyond the (temporary) public co-archiving of the reviews themselves, along with the rest of the unfiltered drafts? > Considerable > time would be saved in mark-up and in the management of the review > process. Markup has nothing to do with the peer review process. Peer review assesses and provides (binding) feedback about the quality of the CONTENT of papers; markup is concerned with tagging FORM. > On the other hand papers will be reviewed more than once by > different boards - but this is an appropriate and vital academic > activity, on a par with writing and reading. Do not mix up published post-publication reviews (book reviews, formal commentaries) with peer review. Peer review is a feedback process meant to help shape the paper's contents to meet the quality criteria of a particular journal within the quality hierarchy of journals. Multiple review AFTER publication is nothing like multiple submission BEFORE publication, which is a huge drain on the finite resources of peer review. > I think more effort is > required judging academic output so that it becomes accessible to more > than one audience. At the moment this lack of cross-judging impedes the > flow of information from field to field. Archive publicly. Then anyone can "cross-judge" it. But don't expect the same attention for unfiltered self-archived drafts as for quality- controlled ones, duly tagged by the journal that has done the peer review. You are conflating two features here. > Yes there would be many changes to get used to. One would have to put > up with getting poor reviews on your on-line journal, but if you did not > want it to be judged, why make it available. This is purely a single-author-eye's view of the literature, the view from which most of the unrealistic proposals for new "systems" tend to be made. The literature is not there only, or even primarily, so as to vouchsafe an author the maximum amount of attention and exposure; that's what Hyde Park is for. The literature is there to provide information for us all, information that we can trust and navigate. And for that it needs to be quality controlled by experts in advance. Otherwise we all have to find our way through the unfiltered morass for ourselves. So it is not sufficient to fantasize about authors being kept up to standards by fears of receiving negative public reports: In the system you recommend it is not clear who would bother to read what any more; so it is equally unclear whether negative public feedback would even be systematically noticed by anyone in the vast unfiltered pot-pourri of papers and comments (another kind of paper) that would replace the refereed literature. As an exercise, please step through your proposal as it would apply to papers on cancer treatment, and give some indication of how a practitioner should apply this literature to his ongoing practise. Don't focus on the quack paper that is prominently denounced as such; focus on the vast majority of borderline papers, where it is not clear whether they are or are not reliable. And don't reject the biomedical literature as a special case: Some of us consider it just as important to sort the reliable from the unreliable in any area of scholarly and scientific research to matter. > There would be > compensation in that, with a plethora of review boards, one would get > used to have a variety of reviews depending on its acceptability and > importance for different audiences. Your proposal is like social credit: You are hypothetically disbursing resources (referee time and effort) that there is no evidence you would have left if your system were actually adopted (and strong prima facie reasons to expect you would not). > Life is `tough' for academics, if they go public with ideas they are > responsible for the quality, regardless of whether it is formally > submitted to a journal. Correct, and, as above, this remains just as true with self-archiving PLUS classical peer review as with self-archiving MINUS classical peer review (and some other untested option in its place). > One has to accept one's work will be judged and > not release it (i.e. publish it) until one is ready to take that risk. True again, with or without your proposal. > It is a *problem* with the current system that the level of judgement is > low, unless it is formally submitted. But my way you can have this > judgement _without_ restricting the availability and flow of information > until it almost reaches the reader. This sounds incoherent (what is "almost"?). You can self-archive your paper, and take your lumps publicly. This is true with or without classical peer review. You can even set up an archive and baptize it an "open journal," in which authors get "referee reports" and stars alongside their self-archived papers (though it is not clear who would do the refereeing or why). That's part of taking the lumps too. But for those who would prefer to restrict their reading to papers that (at least) meet existing quality standards, those standards still have to be applied, by qualified experts, selected by and accountable to expert umpires (editors), who ensure that only what has met the standards, possibly after several rounds of revision and re-refereeing, gets tagged as such. Nothing in your proposal fills this need, or shows that it need no longer be filled. "Journal" still seems a reasonable name to reserve for what fills it. -------------------------------------------------------------------- Stevan Harnad [email protected] Professor of Cognitive Science [email protected] Department of Electronics and phone: +44 1703 592-582 Computer Science fax: +44 1703 592-865 University of Southampton http://www.ecs.soton.ac.uk/~harnad/ Highfield, Southampton http://www.princeton.edu/~harnad/ SO17 1BJ UNITED KINGDOM ftp://ftp.princeton.edu/pub/harnad/
