A naive solution seems reasonable if I am willing to assume that students are
randomly assigned to the preceptors for evaluation. If so, I'd expect the
average rating given by each judge to be the same. So, force the judges'
means to be be equal to the overall mean by dividing each individual score
appropriately. Then calculate the students' averages. Of course the results
are no longer integers 1 to 9, but that's where you, the filter to that
"other body", will have the Procrustean Responsibility!
If there is no balance, if for instance some students is rated 12 times by
the same judge while others are rated by 12 different judges, there probably
is no good model for the process. The literature on rating the strength of
sports teams in a league with an unbalanced schedule might give some hints on
how to proceed.
Doug Federman wrote:
> I have a dilemma which I haven't found a good solution for. I work with
> students who rotate with different preceptors on a monthly basis. A
> student will have at least 12 evaluations over a year's time. A
> preceptor usually will evaluate several students over the same year.
> Unfortunately, the preceptors rarely agree on the grades. One preceptor
> is biased towards the middle of the 1-9 likert scale and another may be
> biased towards the upper end. Rarely, does a given preceptor use the 1-9
> range completely. I suspect that a 6 from an "easy" grader is equivalent
> to a 3 from a "tough" grader.
>
> I have considered using ranks to give a better evaluation for a given
> student, but I have a serious constraint. At the end of each year, I
> must submit to another body their evaluation on the original 1-9 scale,
> which is lost when using ranks.
>
> Any suggestions?
>
> --
> "It has often been remarked that an educated man has probably forgotten
> most of the facts he acquired in school and university. Education is what
> survives when what has been learned has been forgotten."
> - B.F. Skinner New Scientist, 31 May 1964, p. 484
--
Neil W. Henry
Department of Sociology and Anthropology
Department of Statistical Sciences and Operations
Research
Box 843083
Virginia Commonwealth University
Richmond VA 23284-3083
(804)828-1301 x124 FAX: 828-8785
http://www.people.vcu.edu/~nhenry
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
