Whether metrics are improving, and whether it is a good idea to base decisions about quality of and funding for research and journals entirely on usage metrics, are two separate questions.
In this post, I agree that metrics are improving with potential to advance our understanding of scholarship, but that there are dangers to be considered from over-reliance on usage metrics. Another idea I would like to introduce is cost-efficiency metrics. As Stevan points out, "metrics are becoming far richer, more diverse, more transparent and more answerable than just the ISI JIF". There is indeed potential to develop much richer metrics, and this is a good thing, as it gives us a better means to research scholarship per se. However, there are potential dangers to scholarship from relying too much on metrics. One important point is the distinction between popularity (or temporary importance), and real importance to scholarship or to the world. Consider, for example: Biology - species. There will always, of necessity, be a limited pool of scientists studying any one species in danger of extinction. Do articles and journals in these areas receive fewer citations? If so, what happens if we reward scholars and journals on the basics of metrics? Will these researchers lose their funding? Will journals that publish articles in this area lose their status? Literature - authors. There are many researchers studying Shakespeare. A lesser-known local author will be lucky to receive the attention of even one researcher. In a metrics-based system, it seems reasonable to hypothesize that this bias will increase, and the odds of studying local culture decrease. History - the local versus the global. A reasonable hypothesis is that historical articles and journals with broader potential readership are likely to attract more citations than locally-based historical studies. If this is correct, then local studies would suffer under a metrics-based system. (In the medium to long term, the broader studies would suffer, too, through lack of background that can be supplied by in-depth local research). Medicine - temporary importance: AIDS, bird flu, SARS, are all viral diseases, horrible diseases and pandemics or potential pandemics. Of course, our research communities must prioritize these threats in the short term. This means many articles on these topics, and new journals, receiving many citations. Great stuff, this advances our knowledge and may have already prevented more than one pandemic. But what about other, less-pressing issues, such as the resistance of bacteria to antibiotics and basic research? In the short term, a focus on research usage metrics helps us to prioritize and focus on the immediate danger. In the long term, if usage metrics lead us to undervalue basic research, we could end up with more pressing dangers to deal with, such as rampant and totally untreatable bacterial illnesses, and less basic knowledge to help us figure out what to do. This is speculation, but hopefully enough theoretical substance to illustrate that there are good reasons to think carefully about the impact of metrics-based systems before rushing to implement them. Cost-efficiency metrics, such as average cost per article, is a tool that can be used to examine the relative cost-effectiveness of journals. In the print world, the per-article cost for the small, not-for-profit society publishers has often been a small fraction of the cost of the larger commercial for-profit publishers, often with equal or better quality. If university administrators are going to look at metrics, why not give thought to rewarding researchers for seeking publishing venues that combine high-quality peer review and editing with affordable costs? Any opinion expressed in this e-mail is that of the author alone, and does not represent the opinion or policy of BC Electronic Library Network or Simon Fraser University Library. Heather Morrison, MLIS The Imaginary Journal of Poetic Economics http://poeticeconomics.blogspot.com
