On Wed, 19 Dec 2001 02:11:01 GMT, Doug Federman <[EMAIL PROTECTED]> wrote:
> I have a dilemma which I haven't found a good solution for. I work with > students who rotate with different preceptors on a monthly basis. A > student will have at least 12 evaluations over a year's time. A > preceptor usually will evaluate several students over the same year. > Unfortunately, the preceptors rarely agree on the grades. One preceptor > is biased towards the middle of the 1-9 likert scale and another may be > biased towards the upper end. Rarely, does a given preceptor use the 1-9 > range completely. I suspect that a 6 from an "easy" grader is equivalent > to a 3 from a "tough" grader. Huge rater-differences? that implies that you can't keep the original 1-9 and be fair. Doesn't it? And you can't trust the labels stuck with the original 1-9. > > I have considered using ranks to give a better evaluation for a given > student, but I have a serious constraint. At the end of each year, I > must submit to another body their evaluation on the original 1-9 scale, > which is lost when using ranks. > > Any suggestions? The original 1-9 seems to have no rational anchor, if the raters can vary between by 3 points on a standard. So, you average your ranks; sort the students that way; compute and list their raw averages along with rank averages. If extreme HIGH scores or LOW scores are important, you could avoid ranks. (a) Simply rescore each preceptor to have an average of 5, or (b) normalize each preceptor by mean and standard deviation. Assume ranking. I think, from that step, I might assign the mid-average to the mid-rank -- this would have the effect of anchoring my eventual, final report to the same range the raters used. (Assuming this matters.) Then I would work my way to the extremes, 'grading by the curve.' -- If you are trying to respect the labels on the 1-9 Likert, you have to consider what they actually *say*. You might justify giving the best student a '9' despite an average of 6.9 if the label says "best, consistent performance." -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html ================================================================= Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =================================================================
