I haven't the time to develop this, so I'll throw it out there in hopes someone 
will run with it. I believe being asked to referee indicates one's standing in 
a field. Journals will always try to get the best referees possible. We simply 
don't have a way to measure or reward reviewing. 

For authors we have a measure of impact (actually several, 
see http://en.wikipedia.org/wiki/H-index for a quick start). I would suggest 
something similar for referees. Journals would produce an annual list of 
reviewers and the number of time each reviewed. The sum of the number of 
reviews by a referee times the impact factor of the journals  they review in 
should give a pretty good index of their standing in their field. Reviewing in 
Science would be rare but earn a high score but more frequent reviewing in high 
ranked but more focused journals would really drive scores. Reviewing in low 
ranked journals would not help one's score much but as at present would be done 
more as moral obligation than for one's career. 

Further indices could correct for time and frequency of reviews, or look at 
mean rank, much as the H-index spawned a wave of refinements.

Once each of us has a number (or various), there will be a natural inclination 
to want to improve one's standing (which can be done by more reviewing or by 
being asked to review by higher impact journals). Administrators, obsessed with 
the quantitative will latch onto this like flies onto roadkill for evaluating. 
The bottom line would be a competition for opportunities to review rather than 
a competition among editors for a limited number of reviewers. We would measure 
those who give back, not just those who publish.

Of course this could be gamed, but the best defense would be editors who don't 
count reviews unless they reach a certain standard of excellence. Of course if 
editors were too picky, we wouldn't bother to review for that particular 

We can continue to bemoan the state of reviewing, and dream up sticks with 
which to beat reviewers into helping, or we can come up with carrots. This 
carrot  is cheap and appeals to both our better and worse angels.

Anyway, I'd appreciate thoughts on it. If it goes anywhere, I hope someone will 
call it the D-Index.


David Duffy

Professor/PCSU Unit Leader/CESU Director
PCSU/CESU/Department of Botany
University of Hawaii Manoa
3190 Maile Way, St John 410
Honolulu, HI 96822 USA
Tel 808-956-8218, FAX 808-956-4710

----- Original Message -----
From: malcolm McCallum <malcolm.mccal...@herpconbio.org>
Date: Saturday, January 7, 2012 4:49 am
Subject: Re: [ECOLOG-L] Is there a referee crisis in ecology?

> Recent joint editorial from all herp societies published in
> Herpetological Conservation and Biology.
> The "peer" in Peer Review.
> http://www.herpconbio.org/Volume_6/Issue_3/Joint_editorial_2011.pdf
> On Thu, Jan 5, 2012 at 4:02 PM, Chris Lortie 
> <lor...@yorku.ca> wrote:
> > Dear Ecologgers,
> >
> > Thank you so much for your feedback on the editorial 'Money 
> for nothing and referees for free'
> > published in Ideas in Ecology and Evolution in December
> > (http://library.queensu.ca/ojs/index.php/IEE/index).  The most 
> compelling and common question
> > I was asked was is there a referee crisis in ecology (or 
> tragedy of the 'reviewers common' as
> > Hochberg et al. proposed).  This is an excellent question.  I 
> propose that whilst there are more
> > perfect ways to test this (total up number of submissions and 
> then estimate total pool of referees,
> > tricky), an interesting indicator would instead to be 
> calculate the decline to review rate (d2rr) in
> > ecology.   I envision the following two primary data streams 
> to calculate this rate: a per capita
> > estimate derived from each of us personally and a mean 
> estimate of rate from the publishing
> > portals (journals).  Hence, let's do it.  Only you know your 
> decline to (accept doing a) review rate
> > across all requests whilst journals track their own net rates 
> and your specific rate with them too.
> >
> > So, please take 30 seconds and fill in this short survey, and 
> we can then assess, to an extent,
> > whether there is a referee crisis in ecology.
> >
> > https://www.surveymonkey.com/s/VD3K36W
> >
> > I have also compiled a long list of emails for every editor I 
> could find for all ecology journals and
> > have contacted them to see if they would share the rate at 
> which individuals decline for each of
> > them, i.e. do they have to ask 5 or 6 people to even secure 
> two reviews?  I will not share the journal
> > names etc. and protect their rates as I recognize the 
> implications.  I would just like to know what
> > our overall mean is from a journal perspective too.
> >
> > Thanks so much for your time and help with these discussions. 
>  I hope you think they are
> > important too, but I also want to assure you that this is my 
> penultimate post on the subject.
> > Warm regards,
> > Christopher Lortie.
> > lor...@yorku.ca
> > www.onepoint.ca
> -- 
> Malcolm L. McCallum
> Department of Molecular Biology and Biochemistry
> School of Biological Sciences
> University of Missouri at Kansas City
> Managing Editor,
> Herpetological Conservation and Biology
> "Peer pressure is designed to contain anyone with a sense of 
> drive" -
> Allan Nation
> 1880's: "There's lots of good fish in the sea"  W.S. Gilbert
> 1990's:  Many fish stocks depleted due to overfishing, habitat loss,
>             and pollution.
> 2000:  Marine reserves, ecosystem restoration, and pollution reduction
>           MAY help restore populations.
> 2022: Soylent Green is People!
> The Seven Blunders of the World (Mohandas Gandhi)
> Wealth w/o work
> Pleasure w/o conscience
> Knowledge w/o character
> Commerce w/o morality
> Science w/o humanity
> Worship w/o sacrifice
> Politics w/o principle
> Confidentiality Notice: This e-mail message, including any
> attachments, is for the sole use of the intended recipient(s) 
> and may
> contain confidential and privileged information.  Any unauthorized
> review, use, disclosure or distribution is prohibited.  If you 
> are not
> the intended recipient, please contact the sender by reply e-
> mail and
> destroy all copies of the original message.

Reply via email to