Hello -
I am using glmnet to generate a model for multiple cohorts i. For each i, I
run 5 separate models, each with a different x variable. I want to compare
the fit statistic for each i and x combination.
When I use auc, the output is in some cases is < .5 (.49). In addition, if
I compare mean MSE (with upper and lower bounds) ... there is no difference
across my various x variables, but mean AUC (with upper and lower bounds)
shows differentiation. My basic questions are, should I not expect AUC to
lie between .5 and 1 and, which model fit measurement is most appropriate
for comparing across models (if the various statistics are producing a
somewhat inconsistent story).
Thanks in advance for any advice.
Below is my code and sample output for AUC/MSE.
xc <- split(dataS$P1_retained, dataS$TotalHours_R)
yc <- split(dataS$x, dataS$TotalHours_R)
for (i in 1:length(yc))
{
fit=cv.glmnet(as.matrix(yc[[i]]), y=xc[[i]], alpha=.05, type="mse",
nfolds=10, standardize=TRUE,family="binomial")
c_output =
c(i,fit$cvlo[fit$lambda==fit$lambda.1se],fit$cvm[fit$lambda==fit$lambda.1se],
fit$cvup[fit$lambda==fit$lambda.1se])
names(c_output) = names(output_x)
output_x = rbind(output_x, t(c_output))
fit1=cv.glmnet(as.matrix(yc[[i]]), y=xc[[i]], alpha=.05, type="auc",
nfolds=10, standardize=TRUE,family="binomial")
c_output1 =
c(i,fit1$cvlo[fit1$lambda==fit1$lambda.1se],fit1$cvm[fit1$lambda==fit1$lambda.1se],
fit1$cvup[fit1$lambda==fit1$lambda.1se])
names(c_output1) = names(output_x1)
output_x1 = rbind(output_x1, t(c_output1))
fit2=cv.glmnet(as.matrix(yc[[i]]), y=xc[[i]], alpha=.05, type="class",
nfolds=10, standardize=TRUE,family="binomial")
c_output2 =
c(i,fit2$cvlo[fit2$lambda==fit2$lambda.1se],fit2$cvm[fit2$lambda==fit2$lambda.1se],
fit2$cvup[fit2$lambda==fit2$lambda.1se])
names(c_output2) = names(output_x2)
output_x2 = rbind(output_x2, t(c_output2))
}
COHORT LB_MSE_X MEAN_MSE_X UB_MSE_X LB_AUC_X MEAN_AUC_X UB_AUC_X
LB_CLASS_X MEAN_CLASS_X UB_CLASS_X 0 0.44 0.44 0.44 0.50 0.50 0.50 0.33
0.33 0.33 1 0.42 0.42 0.42 0.51 0.51 0.52 0.30 0.30 0.30 2 0.40 0.40 0.40
0.50 0.50 0.50 0.28 0.28 0.28 3 0.36 0.37 0.37 0.51 0.51 0.51 0.24 0.24
0.24 4 0.35 0.35 0.35 0.51 0.51 0.51 0.22 0.23 0.23 5 0.33 0.33 0.33 0.51
0.51 0.52 0.21 0.21 0.21 6 0.32 0.32 0.32 0.51 0.51 0.51 0.20 0.20 0.20 7
0.30 0.31 0.31 0.52 0.52 0.52 0.19 0.19 0.19 8 0.29 0.29 0.30 0.52 0.52
0.52 0.18 0.18 0.18 9 0.28 0.29 0.29 0.52 0.52 0.52 0.17 0.17 0.17 10 0.28
0.28 0.28 0.52 0.53 0.53 0.17 0.17 0.17
[[alternative HTML version deleted]]
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
