Dear Andrew, I think our exchange has been useful, if only to highlight the points on which we will have to agree to disagree, and why. I will try to summarize our four main points of disagreement, and then make some comments on your recent replies. I will send our exchange to Fiona, offering to cobble mine into a commentary (and suggesting that if she is interested, she then ask you to do the same with your replies). I look forward to breakfasting with you (and maybe Jean-Claude) on November 22.
One last question: As our differences are, I think, rather important, because we are both very involved in the open-access movement and we have both written a good deal about it, would you give me permission to also post our full exchange in the American Scientist Forum? The BOAI list on which they have appeared is not a public list, and seen only by the dozen or so original signatories of the BOAI. Summary of our differences: (R1) Your primary motivation is (R) to reform research communication, mainly through the posting of all papers online, unrefereed, and then relying on self-selected "vetters," in place of of classical peer review, to evaluate, improve and thereby sign-post paper quality. (A1) My primary motivation is (A) to make the classically peer-reviewed research literature we have now openly accessible online, such as it is, to maximize its accessibility, usage and impact. Qualified journal editors continue to select the referees, referees and author remain answerable to the editor, and the journal-name and its quality track-record sign-post for potential users that an article has met that journal's quality-standards. (R2) You see a causal connection between A and R: A will lead to R. (A2) I see a causal connection between worries about R and not-A: Worries that R would compromise or destroy the quality and usability of the peer-reviewed literature hold researchers back from doing A. (R3) You think peer review is flawed and should be replaced by something better. (A3) I think peer review is no more nor less flawed than any other area of human judgment; it is merely a systematic, answerable method for soliciting qualified judgment, and sign-posting the outcome. If there is a better method, then it should definitely replace classical peer review; but no one has yet tested or demonstrated a better method. (R4) You think that the research advances that occur before peer review through the online posting of pre-refereeing preprints today are evidence that peer review is unnecessary and can be replaced by spontaneous vetting (R) without loss (and perhaps even with a gain) in quality. (A4) I think those research advances are only evidence that online pre-refereeing preprint posting is a very valuable supplement to -- but not a substitute for -- classical peer review, which is still there, unchanged, alongside all preprint posting and exchange today (the "invisible hand" of answerability to classical peer review). No test of what would happen without classical peer review has been done; all your evidence is parasitic on intact peer review. On Tue, 5 Nov 2002, Andrew Odlyzko wrote: > We do have substantially different visions of the future of peer review, > and they have not changed much since we first started corresponding back > in 1993. I agree. Nor has the evidence changed since 1993. >sh> at I am suggesting is that if your prediction [that A will lead to R, >sh> and that R is viable] happens to be wrong, making the prediction >sh> anyway will have a negative, retardant effect on self-archiving and open >sh> access. (Of course, if you are right, then concerns about these changes >sh> will still have a negative, retardant effect on self-archiving.) > > I am not sure it will be negative. It might encourage the transformation > by showing the path to a future in which not only information dissemination, > but also peer review, are improved. If you are right (and if reluctant self-archivers, worried about peer review, believe you). But if you are not right, or not believed, then the effect will be negative. > I hope to hasten the transition [to open access] by pointing out that > it is likely to improve the peer review system. And I hope that voicing your conjecture will not have the opposite effect. > Sorry, but alternatives to classical peer review have been tried, and > are constantly being tried (quite successfully, in my opinion). That > is largely what my article "The fast evolution of scholarly communication" > was about. Testing your hypothesis means performing systematic, controlled experiments on representative and sufficiently large samples of the literature (on a sufficient time-scale) to test whether a system consisting exclusively of self-selected vetting of completely unrefereed papers, with no subsequent or parallel classical peer review, yields a literature of quality comparable to its current quality. Nothing faintly like this has been attempted yet. Online preprint posting and exchange occur within a "parallel economy," with classical peer review still 100% in place and active, before, during and afterward. There is not only this total empirical gap between the data you use and the conclusions you draw, but there are also logical gaps: You have not replied when I have asked how, in a system where classical peer review and journal-names with track-records are no longer there as the back-up and bottom line -- as they are universally and without exception now -- how the annual 2,000,000 papers (which are today refereed and sign-posted by refereed journals) will find their proper vetting, and how this will be sign-posted for potential users? This question does not even come up in the case of pre-refereeing preprints, because those are a "parallel economy," backed up by the classical peer-review and then sign-posted by the names and track-records of the journals to which just about every one of those preprints has been submitted, and in which they will all appear eventually (though perhaps only after several rounds of revision and refereeing, and perhaps not always at the level of the journal to which they were submitted first.) How will the 2,000,000 annual papers (all? some? most? enough? which?) find their qualified vetters? And how will potential vetters know what to vet (and why will they want to do that faithful duty for us all under those ad lib conditions, busy as they all are)? And what will ensure that the papers' authors (all? some? most? enough? which?) hear and heed what needs to be heard and heeded? And how will potential users know whether (and when) the vetting has been heeded? You, Andrew, are imagining that, persisting miraculously under these anarchic conditions, there will still be something much like what we have already, with classical peer review, whereas I am imagining the un-navigable chaos it would quickly decline into, once no longer propped up by the invisible hand of peer review and the sign-posting of the outcome by the journals that implement it. One can, of course, dream up a system of answerability and classification that would systematize and bring order into all this. But call it what you like, it will prove to be a re-invention of something very much like classical peer review. > Well, yes, we don't know whether my prediction is true, but there is > evidence for it, for example in "The fast evolution ...." Peer review > is undergoing change right now, in front of our eyes, even though > few are paying attention. To pretend that nothing will change seems > really short-sighted. I submit that nothing like the test I described above is happening now. It is all just small local supplements to the universal system currently backing it up. > We get a very weak quality and usability signal from > classical peer review, and my contention is that we can obtain many other > signals that collectively, if not individually, can be even more useful. Classical peer review is, as I have agreed repeatedly, fallible, being based on human judgment. Perhaps it can be improved; it can certainly be supplemented. But whether an alternative is truly a substitute -- and whether it can yield a literature of at least equal quality and usability -- must first be tested. None of the data from which you draw your conclusions constitute such a test; they are all parasitic on classical peer review. > I would have to go back and dig up some old messages, but I believe there > are several very reputable scholars who have stopped publishing in traditional > journals. And what follows from that? That this can be safely extrapolated to all the reputable and not-so-reputable authors of the annual 2,000,000? Of course there are always exceptions. If Newton were alive today, he would have no peers, and we could all safely read every raw manuscript he wrote. But nothing whatsoever follows from that for the 2,000,000. Such cases (usually elite ones) simply do not scale! > Furthermore, some estimates have been made of arXiv submissions, > and a noticeable fraction of them do not get submitted to journals. (I can > testify to some really outstanding papers in mathematics that are available > only through arXiv, because their authors simply never bothered to submit > them to journals.) This is partly the Newton effect again (for the elite at the peak of the distribution). As to the noticeable fraction never submitted to journals, we need to know a bit more: (1) How big a fraction? (2) Apart from the Newton-subset (how big a subfraction is that?), how does this fraction compare in quality to the the rest of the papers (i.e., those that did go on to appear in journals)? (3) Were some (many) of these rejected by journals? revised and resubmitted under another name? incorporated instead in later work that was eventually accepted by a journal? Such questions are very interesting, and we too have done and are doing such analyses -- http://opcit.eprints.org/tdb198/opcit/ -- http://opcit.eprints.org/ijh198/ -- but these data are certainly no substitute for (or even predictive of the outcome of) testing what would happen in the complete absence of classical peer review (as sketched above). > Part of our difference is probably rooted in our varying professional > experiences. As just one example (there are others in "Tragic loss ...") > some of the most interesting developments in mathematics over the last > half century were in algebraic geometry (leading to several Fields Medals, > the mathematics equivalent of a Nobel Prize). Much of that work was > based on several thousand pages published in the un-refereed Springer > Lecture Notes in Mathematics (the famous SGA volumes). Why was the > field willing to rely on those? Well, it is a long story, but there > were enough checks around (such as many people, including graduate > student seminars going over those volumes line by line) to convince > experts to rely on those papers. It was examples like these, definitely > involving review by peers, but not classical peer review, that helped > convince me that scholars could thrive under a variety of systems. > I spend an inordinate amount of time on this subject in "Tragic loss ..." I read it, and found it very interesting, but it does not test the hypothesis. It merely confirms that classical peer review can be supplemented in various ways (especially with elite researchers, highly specialized topics with relatively few practitioners and known to one another, and relatively fast-moving developments). It certainly does not follow that these special-case supplements can serve as substitutes for classical peer review, nor that they scale to research as a whole. Here is a logical reductio (since so much of this is a question of scale): If research just consisted of algebraic topology, and there were only 6 algebraic topologists doing all the breakthrough work, we would not need journals and peer review: We could revert to the exchange of learned letters of Newton's day. Unfortunately (perhaps), this does not scale to the full gaussian distribution of fields and quality represented by the 2,000,000 annual papers published in our day. Classical peer review evolved specifically to cope with the increase in scale of research. Advocating posting it all to the skies and assuming that vetting will somehow take care of itself at this scale strikes me as unrealistic in the extreme. In some ways, it is a pity that self-archiving and open-access began with unrefereed preprints. It gave two wrong impressions (diametrically opposite ones, in the event): Some people concluded, wrongly, that self-archiving and open-access are only suitable for unrefereed preprints, whereas refereed postprints should be toll-access. (I think we all agree that that is not only wrong, but nonsense.) Others concluded (equally wrongly), that unrefereed self-archived, open-access preprints are all we need: We can dispense with the peer review and the postprints. How much better it would have been (but alas it is too late to redo it) if the first ones who had "twigged" on the fact that open-access is optimal for all research had self-archived instead their refereed postprints, rather than their unrefereed preprints. The reason this did not happen is that (1) most researchers then (and now) wrongly believed that copyright law prevents them from self-archiving their refereed research and (2) the physicists, though not silly enough to be deterred by such copyright worries, were focussed mainly on the "growing edge" of their work, which precedes the refereed postprint by 8-12 months. So they took to self-archiving their pre-refereeing preprints first (though from the outset, many swapped or added the refereed postprint 8-12 months later, once it was available). Today it is still copyright worries that are holding back self-archiving (except among the much more sensible physicists and mathematicians). But a further worry has been added to retard self-archiving: that it might destroy peer review, and hence the quality and navigability of the research literature. And this worry has been (needlessly) encouraged by the (incorrect) interpretations the self-archiving physicists and mathematicians have made of what they have actually been doing, and what follows from it. They THINK they have shown that peer review is not necessary; in reality what they have shown is that open access is optimal. If self-archiving had (mirabile dictu) begun instead with refereed postprints, we might have spared ourselves these misconstruals, and we might have been further along the road to open access by now.... >sh> Peer review is not a gold standard, but I'm sure you will agree >sh> that any alternative would have to ensure at least the same >sh> standard, if not better: Do you think there is this evidence >sh> for the promise you are holding out above? > > Yes, I do. Examples such as that of algebraic geometry (mentioned in my > previous response above) showed me early on that there is nothing sacred > about classical peer review. The law review system is yet another example. I'm afraid we will have to agree to disagree about that. I have tried to explain why those cases do not test the hypothesis, and what tests are needed. >sh> I honestly can't see how you imagine this scaling to the annual >sh> 2, 000,000 papers that currently appear in the classically >sh> peer reviewed journals! Absent the peer reviewed journal, how >sh> can I know that a paper has been "vetted by experts of a top >sh> caliber"? What tells me that (as the journal-name currently >sh> does) for those 2,000,000 annual papers? And what now gets >sh> the right-calibre experts vetting the right papers (as editors >sh> formerly did, when they invited them to referee?). Do experts >sh> voluntarily spend their precious time trawling the continuum of >sh> raw papers on the net on their own? > > I do not have the time to respond to all these points, but yes, many experts > do "voluntarily spend their precious time trawling the continuum of raw papers > on the net on their own." I do know many people whose day starts with a scan > of the latest arXiv submissions. Moreover, some put a lot of effort into > making what they find more easily digestible for others. (John Baez and his > wonderful "This week's finds in mathematical physics" comes to mind.) I can only repeat: This is all parasitic on a classical peer review system that is still intact behind all of this. It does not and cannot tell us what would happen without it. > I am sorry, Stevan, but you are ignoring some of the most interesting > evolutionary developments in scholarly publishing in your blind faith > that classical peer review is the only thing that stands between us > and chaos. Perhaps. But it is a historical and evolutionary fact that classical peer review is still 100% intact and in place behind all these developments, which therefore makes them supplements to peer review, not substitutes, until such a time as evolution or experimentation actually tests them as substitutes. > Open archives are not a religion to me, just a step towards > a better scholarly communication system, which will also require changes > in peer review. I don't think open-access is a religion for me either (though it has become a bit of an obsession). For me too, it is merely a means to an end. The end, though, is the enhanced impact and interactivity, hence productivity, of research. I don't think it's the flaws of classical peer review that are holding that back (especially now that pre-refereeing preprints can be made openly accessible too), but the access-barriers of the toll-access system. Open access to it all would solve that problem -- and it would also open the door to any evolutionary developments along the line you are hypothesizing, if they turn out to be adaptive -- but only after needless worries about copyright and peer review are overcome. Our point of disagreement is only about the advisability of needlessly exacerbating worries about peer review while self-archiving and open access still have not prevailed. Stevan Harnad
