Where I think we agree: 1. What you call the organised process of review is largely irrelevant (review boards, journals etc.).
2. The advent of widely available word processing and internet access has cut many of the costs associated with journals, there is little reason why academics should pay a publisher to manage a journal for them - after all academics do almost all the work (reviewing, marking-up own papers, downloading, writing). 3. That the organisation of the review process (and mark-up of papers where it occurs) are the last major unsubsumed costs (cost of archiving being small). 4. That reviewer's time is precious and not to be squandered. 5. That readers need some system to help them find the quality papers from amongst the multitude, otherwise their time is wasted and they end up ill-informed. 6. That archives, rapid peer-commentary etc. have an important role in promoting active discussion of ideas. 7. That academics need some system to promote their work (if it is good) to others and gain recognition for it. Where I think we disagree: A) (most fundamentally) in our conceptions of the process of knowledge development. (I think) you have a foundationalist conception: each paper is checked and worked on until it can be relied on in the collective construction of knowledge (rather like building a wall out of bricks - you make sure each brick is sound before relying on it to support further such bricks). I have a more evolutionary picture in mind: academics are continually producing variations on ideas, experiments, studies, models etc. (both individually and as part of large ecologies), then selection pressures are applied so that (probably) the better will emerge. B) I see the closed nature of the author-reviewer/editor revision process as somewhat of a hang-over from the days when considerable cost was expended in publishing. In my experience many such discussions are not about simple errors in established fact or poor presentation but involve issues of content and/or subject demarcation. Such discussion would be better as a public discussion rather than a private one where one discussant has power over another. Peer commentary of the type you have championed goes some way in this regard, but not all the way. I do not suppose that such closed-process reviews would disappear, but exist along side (and in competition with) the open evaluation-boards. C) I think that it would be useful to readers to have more ancillary information about a paper, beyond the published/not published duality. Of course, information about the content of the paper is paramount (title, author, institution, abstract, keywords, commentary, outline, references etc.), but there is far more that could be useful. Examples of such are: originality, importance, empirical content, accessibility to non-expert, clarity of argument, readability of language, presentation, amount of previous work assumed etc. These are regularly assessed by referees, both explicitly on many review forms and implicitly in comments, but this useful information is not usually imparted to aid the readers find the information they want. I know that a "star" system is crude (albeit less crude than the present system) - I am arguing for richer information to be made available to the reader, especially where it is in a form which could be utilised in database-style queries and user-specific settings. D) The system where a single paper is judged once for all audiences belies the fact that the same paper will, in effect, be of a different quality for different audiences. If journals are not going to own and hold the papers, but concentrate on selecting them, then there is no reason why different boards should not review the same paper for different audiences. Some mechanism could easily be developed so people knew it was the same paper. I know that there is a current prohibition against repeat publication, but this is for reasons that are now defunct. E) I do not think that my suggested system would end up with more work for reviewers. Authors will be wary of seeing their work get a low assessment in public and will adjust their output to suit. Also each board would quickly devise their own rules for limiting the amount of work to the right level. F) I do not think it would end up with readers losing the reliability of sources of quality papers. I guess that a similar hierarchy of boards would spring up as journals, each offering different styles of assessment, browsing tools and review rigour (in fact I would guess there would be a greater variety than journals because the expertise in mark-up would not be needed). The top-end would undoubtedly be there. Comments: There are already reviewing services of web-pages (e.g. Magellan or Encyclopaedia Brittanica), it is just that they have not been combined with paper archives and web-search engines. It is only a matter of time before they do. The explosion of available information and papers will hasten its appearence. Debating which would be better is literally academic: if the systems were running side by side, readers, reviewers and authors would soon vote with their feet. The system would only adjust to the new if it suited people.
