A classic problem of 'norming' or 'standardizing' the scale and the preceptors. Can you find a couple students who fall near the bottom and tops of the scale? Preferably ones whose final rankings are not 'permanent record'?
then you would have each preceptor use these two students as 'baseline' indicators of what a 2 means, and what an 8 means. then have each person do the regular ranking of students, using these as your indicators. It might be possible for the attendant group of preceptors to agree on the ranking of a pair of students, in each specialty or area. then use these for ranking within that specialty. Failing this kind of development for mutual agreement, you might be able to describe a 2 or 3 rating, and a 7 or 8 rating, in such a way that generalized agreement would be obtained, and each grade would be set in comparison to this descriptive scale. This is essentially what the Baldrige Criteria does, for industrial/ educational/ health care operations. Of course, if it's grades we are discussing, it is entirely likely that virtually nobody gets grades in certain ranges, such as the equivalent of C or below on an A- F scale. If Harvard can graduate over half a class as Cum Laude, the rest of us can skew grades anywhere we like. Jay Doug Federman wrote: > I have a dilemma which I haven't found a good solution for. I work with > students who rotate with different preceptors on a monthly basis. A > student will have at least 12 evaluations over a year's time. A > preceptor usually will evaluate several students over the same year. > Unfortunately, the preceptors rarely agree on the grades. One preceptor > is biased towards the middle of the 1-9 likert scale and another may be > biased towards the upper end. Rarely, does a given preceptor use the 1-9 > range completely. I suspect that a 6 from an "easy" grader is equivalent > to a 3 from a "tough" grader. > > I have considered using ranks to give a better evaluation for a given > student, but I have a serious constraint. At the end of each year, I > must submit to another body their evaluation on the original 1-9 scale, > which is lost when using ranks. > > Any suggestions? > > -- > "It has often been remarked that an educated man has probably forgotten > most of the facts he acquired in school and university. Education is what > survives when what has been learned has been forgotten." > - B.F. Skinner New Scientist, 31 May 1964, p. 484 > > ================================================================= > Instructions for joining and leaving this list and remarks about > the problem of INAPPROPRIATE MESSAGES are available at > http://jse.stat.ncsu.edu/ > ================================================================= -- Jay Warner Principal Scientist Warner Consulting, Inc. 4444 North Green Bay Road Racine, WI 53404-1216 USA Ph: (262) 634-9100 FAX: (262) 681-1133 email: [EMAIL PROTECTED] web: http://www.a2q.com The A2Q Method (tm) -- What do you want to improve today? ================================================================= Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =================================================================
