David's message rings clear, but I am happy to report that he is incorrect on 
one matter.  We DO have a system that measures and rewards reviewing.  A recent 
initiative, called Peerage of Science, has instituted a system that (among 
other things) provides quantitative ratings of review quality.  

I encourage you to read the details about this bold new endeavour at: 
http://www.peerageofscience.org/
I hope the information there can convince many of you to join, or at least 
breathe easier that attempts are being made to divert an (aptly described) 
referee crisis.

In reference to the issue at hand, that of quantifying referee effort, the PoS 
system works along the following lines:
1. A manuscript is submitted to PoS for review
2. Members are alerted to the ms, and can sign up to review it
3. After the manuscript's first submission is reviewed, the reviewers are then 
allowed to see each other's reviews (all anonymous)
4. The reviews are then scored by the other reviewers
5. The manuscript continues on in the process...

Each reviewer then accumulates an average "review quality" score over time.  
Poor reviews are justifiably penalized with low scores.  Excellent reviews 
accrue good scores.  

I am sure that the benefits here are obvious, and perhaps so are a few 
drawbacks.  But, it is the first attempt of which I am aware that is trying to 
create a currency amongst reviewers that is not just an extra bullet on a 
performance review or CV.

Check it out.  Chris, as originator of this thread, I especially think you 
would be interested in this.

Sincerely,
Joe Nocera

(Member of the Board of Governers for Peerage of Science)



----- Original Message -----
From: David C Duffy <ddu...@hawaii.edu>
Date: Saturday, January 7, 2012 2:56 pm
Subject: Re: [ECOLOG-L] Is there a referee crisis in ecology?
To: ECOLOG-L@LISTSERV.UMD.EDU

> I haven't the time to develop this, so I'll throw it out there 
> in hopes someone will run with it. I believe being asked to 
> referee indicates one's standing in a field. Journals will 
> always try to get the best referees possible. We simply don't 
> have a way to measure or reward reviewing. 
> 
> 
> For authors we have a measure of impact (actually several, 
> see http://en.wikipedia.org/wiki/H-index for a quick start). I 
> would suggest something similar for referees. Journals would 
> produce an annual list of reviewers and the number of time each 
> reviewed. The sum of the number of reviews by a referee times 
> the impact factor of the journals  they review in should give a 
> pretty good index of their standing in their field. Reviewing in 
> Science would be rare but earn a high score but more frequent 
> reviewing in high ranked but more focused journals would really 
> drive scores. Reviewing in low ranked journals would not help 
> one's score much but as at present would be done more as moral 
> obligation than for one's career. 
> 
> 
> Further indices could correct for time and frequency of reviews, 
> or look at mean rank, much as the H-index spawned a wave of 
> refinements.
> 
> Once each of us has a number (or various), there will be a 
> natural inclination to want to improve one's standing (which can 
> be done by more reviewing or by being asked to review by higher 
> impact journals). Administrators, obsessed with the quantitative 
> will latch onto this like flies onto roadkill for evaluating. 
> The bottom line would be a competition for opportunities to 
> review rather than a competition among editors for a limited 
> number of reviewers. We would measure those who give back, not 
> just those who publish.
> 
> 
> Of course this could be gamed, but the best defense would be 
> editors who don't count reviews unless they reach a certain 
> standard of excellence. Of course if editors were too picky, we 
> wouldn't bother to review for that particular journal.
> 
> 
> We can continue to bemoan the state of reviewing, and dream up 
> sticks with which to beat reviewers into helping, or we can come 
> up with carrots. This carrot  is cheap and appeals to both our 
> better and worse angels.
> 
> 
> Anyway, I'd appreciate thoughts on it. If it goes anywhere, I 
> hope someone will call it the D-Index.
> 
> 
> Cheers,
> 
> 
> David Duffy
> 
> 
> 
> 
> 
> 
> Professor/PCSU Unit Leader/CESU Director
> PCSU/CESU/Department of Botany
> University of Hawaii Manoa
> 3190 Maile Way, St John 410
> Honolulu, HI 96822 USA
> Tel 808-956-8218, FAX 808-956-4710
> http://www.botany.hawaii.edu/faculty/duffy/
> 
> 
> 
> 
> ----- Original Message -----
> From: malcolm McCallum <malcolm.mccal...@herpconbio.org>
> Date: Saturday, January 7, 2012 4:49 am
> Subject: Re: [ECOLOG-L] Is there a referee crisis in ecology?
> To: ECOLOG-L@LISTSERV.UMD.EDU
> 
> > Recent joint editorial from all herp societies published in
> > Herpetological Conservation and Biology.
> > 
> > The "peer" in Peer Review.
> > http://www.herpconbio.org/Volume_6/Issue_3/Joint_editorial_2011.pdf
> > 
> > On Thu, Jan 5, 2012 at 4:02 PM, Chris Lortie 
> > <lor...@yorku.ca> wrote:
> > > Dear Ecologgers,
> > >
> > > Thank you so much for your feedback on the editorial 'Money 
> > for nothing and referees for free'
> > > published in Ideas in Ecology and Evolution in December
> > > (http://library.queensu.ca/ojs/index.php/IEE/index).  The 
> most 
> > compelling and common question
> > > I was asked was is there a referee crisis in ecology (or 
> > tragedy of the 'reviewers common' as
> > > Hochberg et al. proposed).  This is an excellent question. 
>  I 
> > propose that whilst there are more
> > > perfect ways to test this (total up number of submissions 
> and 
> > then estimate total pool of referees,
> > > tricky), an interesting indicator would instead to be 
> > calculate the decline to review rate (d2rr) in
> > > ecology.   I envision the following two primary data streams 
> > to calculate this rate: a per capita
> > > estimate derived from each of us personally and a mean 
> > estimate of rate from the publishing
> > > portals (journals).  Hence, let's do it.  Only you know your 
> > decline to (accept doing a) review rate
> > > across all requests whilst journals track their own net 
> rates 
> > and your specific rate with them too.
> > >
> > > So, please take 30 seconds and fill in this short survey, 
> and 
> > we can then assess, to an extent,
> > > whether there is a referee crisis in ecology.
> > >
> > > https://www.surveymonkey.com/s/VD3K36W
> > >
> > > I have also compiled a long list of emails for every editor 
> I 
> > could find for all ecology journals and
> > > have contacted them to see if they would share the rate at 
> > which individuals decline for each of
> > > them, i.e. do they have to ask 5 or 6 people to even secure 
> > two reviews?  I will not share the journal
> > > names etc. and protect their rates as I recognize the 
> > implications.  I would just like to know what
> > > our overall mean is from a journal perspective too.
> > >
> > > Thanks so much for your time and help with these 
> discussions. 
> >  I hope you think they are
> > > important too, but I also want to assure you that this is my 
> > penultimate post on the subject.
> > > Warm regards,
> > > Christopher Lortie.
> > > lor...@yorku.ca
> > > www.onepoint.ca
> > 
> > 
> > 
> > -- 
> > Malcolm L. McCallum
> > Department of Molecular Biology and Biochemistry
> > School of Biological Sciences
> > University of Missouri at Kansas City
> > 
> > Managing Editor,
> > Herpetological Conservation and Biology
> > 
> > "Peer pressure is designed to contain anyone with a sense of 
> > drive" -
> > Allan Nation
> > 
> > 1880's: "There's lots of good fish in the sea"  W.S. Gilbert
> > 1990's:  Many fish stocks depleted due to overfishing, habitat loss,
> >             and pollution.
> > 2000:  Marine reserves, ecosystem restoration, and pollution 
> reduction>           MAY help restore populations.
> > 2022: Soylent Green is People!
> > 
> > The Seven Blunders of the World (Mohandas Gandhi)
> > Wealth w/o work
> > Pleasure w/o conscience
> > Knowledge w/o character
> > Commerce w/o morality
> > Science w/o humanity
> > Worship w/o sacrifice
> > Politics w/o principle
> > 
> > Confidentiality Notice: This e-mail message, including any
> > attachments, is for the sole use of the intended recipient(s) 
> > and may
> > contain confidential and privileged information.  Any unauthorized
> > review, use, disclosure or distribution is prohibited.  If you 
> > are not
> > the intended recipient, please contact the sender by reply e-
> > mail and
> > destroy all copies of the original message.
> 

Reply via email to