On Tue, Oct 21, 2008 at 1:23 PM, J.F.Rowland <[email protected]> wrote:
>>> HM: Literature - authors. There are many researchers studying >>> Shakespeare. A lesser-known local author will be lucky >>> to receive the attention of even one researcher. In a >>> metrics-based system, it seems reasonable to hypothesize >>> that this bias will increase, and the odds of studying local >>> culture decrease. >> >> SH: What bias? If a lesser-known researcher does good work, >> it will be used, and this will be reflected in the metrics. > > Stevan - You misunderstood Heather's point. She didn't say the researcher - > the author of the current research article in question - was little-known. > She said the literary author that (s)he was studying was little-known. > Therefore, not many researchers will be interested in that literary author, > so not many people will cite the article, however good it is. > > There is a real and valid point in Heather's message, and simply saying 'use > other metrics' is vague, to say the least. Please specify what metrics > might be used to provide a valid quality measure to the work of researchers > who study minority subjects which will excite interest, and therefore usage, > and citations, from only a few people worldwide. Fair question. Here is a quick and dirty solution, by way of illustration. (Many more are possible along analogous lines.) (1) Compare like with like. So we are only looking at literary studies. The subject matter can be narrowed down by Boolean means, including taxonomic descriptors. No comparing Shakespeare scholarship works with molecular biology. (2) Now, for illustrative purposes, let us suppose there are only two authors studied by literary scholars: William Shakespeare and Joe Bloggs. (3) Do an OA space search (remember, we are assuming universal OA) to find out how many litcrit papers there are on Shakespeare and on Bloggs: Let's say there are S and B papers, respectively. (4) In comparing the citation counts of Shakespeare scholars with Bloggs scholars, divide the Shakespeare citations by S and the Bloggs citations by B. Notice that I used two metrics, jointly: citation counts and a semiometric content-word count. This is just a simplified illustration. With universal OA there will be a vast palate of potential metrics to use to normalize comparisons of like with like. Call it metric "profiling." Separate this question from the question of how you would do it today, with just ISI web of science metrics and no universal OA. The answer is that you can't (though Google Scholar and Google Books might still allow you to do a quick and dirty approximation). Stevan Harnad PS the absolute number of researchers working on a topic is itself a metric, which may for some fields and some purposes be a valid and informative one, and for other fields not. Compare like with like and validate, initialize and weight metrics fields by field (or, for some purposes, even subfield by subfield).
