Stat folks,
I have an interrater reliability dilemma. We are examining a 3-item
scale (each item scored 1 to 5) used to rate compliance behavior of
patients. Two separate raters have used the scale to rate patients'
behavior, and we now want to calculate the interrater agreement for the
scale. Two problems:
1) The majority of patients are compliant, and receive either a 4 or
5 for each of the three items from both of the raters. While this is high
agreement, values for ICC are very low due to the limited range of scores.
Are there any indexes that would reflect the high agreement of the raters
under these conditions? Perhaps something that accounts for the full range
of the scale (1 to 5)?
2) The dataset contains a total of about 100 observations, but there
are multiple observations on the same patients at different times, probably
about 5 to 6 observations per patient. Does this repeated assessment need
to be accounted for in the interrater agreement, or can each observation be
treated as independent for the purpose of interrater agreement?
Any suggestions or references addressing this problem would be
appreciated. Thanks.
Allen Cornelius
