On Tue, May 18, 2010 at 7:49 AM, Matthew Cockerill <[email protected]> wrote:
> I'm guessing that you are making a ballpark estimate that the accuracy > of the typical 'fact' in Wikipedia is around 50%, while the accuracy of > a typical 'fact' in the peer-reviewed literature is around 99.95%. That's right. But the figures are just guesses. The context was a reply to Richard Poynder's pointing out that neither Wikipedia nor peer reviewed journals were 100% reliable. I agreed, but noted that that they differ greatly in terms of degree of unreliability. > I don't think either of us has published a peer-reviewed systematic > analysis on this, but my sense is that, if anything, the numbers could > be the other way around. I agree there are no objective data, but I greatly doubt the figures are the reverse! The absolute and relative reliability estimates may be wrong, but their direction, and the sizeable difference, are unlikely to be wrong. >The peer-reviewed original research literature is the bleeding edge of > knowledge. As such, the original research literature inevitably contains > myriad assertions that seem to be well supported at the time, but which > turn out, on balance, to have been incorrect interpretations of the data > or over-enthusiastic speculation. Again, I don't know the absolute figures on this, but I greatly doubt that WP entries on a topic will prove to be more reliable than peer-reviewed entries on the topic. I would agree that peer-reviewed *review articles* on a topic, written a few years later, are likely to be more reliable than some of the original primary studies they review, but I doubt that WP entries on the same topic will be as reliable, on average, as either peer-reviewed primary studies or peer-reviewed reviews (though individual WP entries do sometimes turn out to be surprisingly reliable -- at least for a while, as long as entry-rot does not set in). > Wikipedia, via its strict 'no original research' policy, sets itself the > more manageable (though still challenging) task of summarizing consensus > opinion from a neutral PoV. As such, I think the facts/assertions it > contains tend be much less disputed and subject to change than those in > the original research literature. They are indeed *supposed* to be written from a "neutral Point of View." But they are written by anonymous authors who are answerable only to other anonymous authors as well as anonymous adjudicators, all of whom, equally strictly, are answerable only to one another, not to expertise on the topic. The result is the unreliable (yet sometimes surprisingly useful) mixture we get from WP. I highly doubt that, on average, WP content is less "disputed" than peer-reviewed content, whether primary or reviewed. And it is almost axiomatic that it is far more "subject to change" (for better or for worse). (You seem to have a somewhat romanticized, if not ideologized PoV about WP, Matt!) > Obviously, it depends where in the vast resource you look. That's what "reliability" means. For more reliable sources, it depends less where you happen to be looking. > But certainly, you must be using a particularly weird subset of Wikipedia if > you are finding 50% of it to be factually 'wrong' in any significant > way. As I said, I am not referring to the hobbyist or trivial-pursuit sectors of WP, but the scholarly/scientific entries. In the (few) fields where I am myself a qualified expert, I find plenty of errors and omissions. In the far larger number of entries on which I am not an expert, I run into errors, about which I have the impression that although they are not as rampant as in average journalism or most blogs, they are far more frequent than in refereed research articles. (And, I hope we agree, that's what the refereeing is done in order to minimize.) 50% is probably an exaggeration, but the WP unreliability rate is still, I think, substantially greater than the refereed research unreliability rate -- though I agree that the right comparison is probably with refereed secondary reviews rather than with individual pieces of refereed primary research, since WP items are themselves secondary (or tertiary) "reviews." > Given Wikipedia aims to document disputed facts from a neutral POV, by > describing the dispute, it is clear that if you hold a strong absolute > opinions about which side of the debate is correct, you are unlikely to > find Wikipedia expressing direct agreement with you on that topic. That > is by design. I am not talking about differences in opinion. And although there are sometimes matters of opinion and bias in scholarly and scientific research that refereeing sometimes fails to detect and correct, I would have a lot more confidence in reports that have been adjudicated by answerable experts (peer reviewers), themselves answerable to an answerable meta-adjudicator (the editor) whose name, journal-title, and track-record publicly under-writes its accepted publications than to an anonymous sample of authors and adjudicators, answerable only to one another's opinions. Perhaps, Matt, you are unduly influenced here by the special case of my "PoV" on the usefulness of calling the WP entry on OA "OA (publishing)." I agree that my PoV on that is not neutral (which, by the way, certainly does not mean it is wrong, since wrong/right is not a matter of nose-counts -- or notability, or notoriety -- on such questions). But my matchbox estimates of reliability were not based on my own experience in that special case of entry-(mis)naming. They are based on my sample, across time, of WP entries in general, including those on the (much rarer) cases where I am myself an expert on the topic in question. > Not only that, in the case of opinions which are disputed, you not only > get to see what one specific opinionated expert finally decided, but via > the discussion pages and the history, you get to see a summary of the > different strains of thinking that have led to the current consensus. Yes, yes. But for matters of fact in science and scholarship I'd rather just have a reliable report of the facts, adjudicated and reported by qualified and answerable experts, rather than the option of delving through the sad history of anonymous, unanswerable, and unqualified opinion, counter-opinion, and adjudication by those whose adjudicative powers are grounded solely in having done a lot of anonymous, unanswerable, and unqualified adjudication... > Biology Direct's peer review model > http://www.biology-direct.com/content/1/1/1 > and the open peer review on the medical BMC-series journals are both > designed to provide a degree of the same kind of transparency regarding > differences of opinion. In contrast, in most of the peer reviewed > literature you see only the final end product, and unfortunately despite > the best efforts of authors, peer reviewers and editors, it is > inevitable that sometimes that final end product will turn out to be > wrong. The reliability, scaleability and sustainability of various "open peer review" experiments are not yet known. Anonymity is a big difference too. (But I'd rather not re-open the much-aired question of peer review experiments and reform in general here: It's already had a lot of air time, and -- like publishing reform and copyright reform -- it tends to co-opt time and attention from the practical goal of providing OA, to which this Forum is dedicated. I realize I am the culprit who introduced the somewhat off-topic question of the dynamics of WP, but I was just hoping to elicit some reflections on direct experience from a long-time AmSci Forum contributor [David Goodman] who went on to become one of the adjudicators of WP. The AmSci Forum's focus is still on freeing access to the peer-reviewed research literature, such as it is, not to freeing it from peer review. The WP topic arose simply because of WP's unfortunate contribution to the conflation of OA with OA Publishing, by canonizing the conflation in its generic OA entry. -- WP, by the way, generously gives OA two entries: you can look up either "OA (publishing)" or "OA journal": from your PoV, Matt, that may look like a good thing, but not from mine -- nor, I would argue, OA's...) Stevan Harnad > -----Original Message----- > From: American Scientist Open Access Forum > [mailto:[email protected]] On > Behalf Of Stevan Harnad > Sent: 16 May 2010 16:11 > To: [email protected] > Subject: Re: Wikipedia, Open Access and Cognitive Virology > > On Sun, May 16, 2010 at 9:12 AM, Richard Poynder > <[email protected]> wrote: >> SH: But Wikipedia's frequent accuracy and usefulness does not (in my > view) >> counterbalance the fact that it is frequently unreliable and > inaccurate too >> ... >> >> RP: Is that not also a reasonable description of the totality of the >> research corpus today, and a feature of the peer-reviewed literature > that >> will become increasingly apparent as more and more of the content > published >> in the world's 24,000 scholarly journals becomes OA? > > It's a reasonable description of both corpora -- if you ignore the > difference between (say) 50% unreliability and .05% unreliability... >
