Hi all, I have an experiment where 5 raters assessed the quality of 24 web sites. (each rater rated each site once). I want to come up with a measure of reliability of the ratings for the web sites ie to what extent does each rater give the same (or similar) rating to each web site. My idea was to fit a random effects model using lme and from that, calculate the intraclass correlation as a measure of rater reliability. So I analysed it using a two-factor, unreplicated random effects model, with sitenumber and rater as crossed random effects. The call to lme was:
fit <- lme(score ~ 1, random=list(level=pdBlocked(list(pdIdent(~sitenumber-1), pdIdent(~rater-1)))),data=scores) which makes sense, and I then calculated the intraclass correlation using the within sitenumber (ie among raters) variance, the within rater variance and the residual variance (using VarCorr(fit)): rho = var_among/(var_among + var_within + var_resid) Now, the raters can be divided into two groups. I would like to include group as a fixed factor, to test whether there is a difference in scores between groups (I know the number of raters is too small. This is a pilot study and I am really just trying to get the analysis right in case we want to expand the study). raters therefore are nested within group. My initial guess was: fit2 <- lme(score ~ group, random=list(level=pdBlocked(list(pdIdent(~sitenumber-1), pdIdent(~rater-1)))),data=scores) but this fit implies (I think) that sitenumber AND rater are nested within group (sitenumber is not nested within group), and results in the wrong denominator degrees of freedom for the test of group (118). I then tried a slightly different approach: fit3 <- lme(score ~ group, random=list(level=pdIdent(~sitenumber-1), rater=pdIdent(~group-1)),data=scores) which gives the correct denominator degrees of freedom for the group effect (3). But it implies that rater is nested within level. Given that level is a dummy factor with only one level, I don't think this matters and I can "pretend" that it is really a crossed design. Is that correct? Or am I way off beam here? Can anyone suggest a more appropriate lme call? The more I think about it the more confused I get. Thanks in advance, Simon. Simon Blomberg, PhD Depression & Anxiety Consumer Research Unit Centre for Mental Health Research Australian National University http://www.anu.edu.au/cmhr/ [EMAIL PROTECTED] +61 (2) 6125 3379 ______________________________________________ [EMAIL PROTECTED] mailing list https://www.stat.math.ethz.ch/mailman/listinfo/r-help
