At 01:51 PM 12/7/00 GMT, Jean-Pierre Guay wrote:
>Good morning everybody,
>
>Considering that the value of ICC (2,k) is influenced by the number of
>judges, would any of you know a correction that would allow me to
>compare results based on 2 diffeent sets of judges (for example compare
>a set of analyses based on 4 judges with a set based on 7)?
>
>Thanks
>Jean-Pierre
I don't think I would say that the ICC estimate is influenced by the number
of judges so much as I would say that it reflects the reliability with that
number of judges.
Generalizability theory offers a multitude of options for evaluating which
is traditionally called reliability.
For example, in evaluating the reliability of clinicians to grade x-rays
you might have several physicians read a set of films on multiple occasions.
>From this experiment you can obtain the reliability of the score that would
be produced if you averaged across all judges within and across occasions.
However, perhaps the most interesting reliability estimate would be that for
a single judge on a single occasion as this might most accurately reflect
what you would expect in practice. Generalizability theory allows you to
do this. If you have Brennan's GENOVA program (pronounced genover by some)
it is fairly simple to generate a wide rage of estimates under varying
conditions of number of judges, occasions, and when assuming these effects to
be fixed or random.
One other problem the Generalizability does not address, however, concerns
the issue of whether the scores or judgements being made are representative
of what would be expected in "the wild". The notion that judges might behave
differently when in a study to evaluate their reliability is generally well
accepted and in many cases, designing a reliability study where judges are
blind
to this fact that they are in a reliability study is not easy.
If these issues are interesting to you, e-mail me directly and I will provide
you with additional information.
________________________________________________________________
_____________
Kevin F. Spratt | |
Iowa Testing Programs (ITP) ( )
University of Iowa Back Care (UIBC) \ \_ University
Iowa Spine Research Center (ISRC) \ * \ of Iowa
224-D Lindquist Center \ / Hawkeyes
The University of Iowa |____________ )
Iowa City, Iowa 52242 V
(319) 335-5572 (voice)
(319) 335-6399 (fax)
[EMAIL PROTECTED] (e-mail)
________________________________________________________________
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
