Gad Abraham wrote:
This approach leaves much to be desired. I hope that its
practitioners start gauging it by the mean squared error of predicted
probabilities.
Is the logic here is that low MSE of predicted probabilities equals a
better calibrated model? What about discrimination? Perfect calibration
Almost. I was addressed more the wish for the use of strategies that
maximize precision while keeping bias to a minimim.
implies perfect discrimination, but I often find that you can have two
That doesn't follow. You can have perfect calibration in the large with
no discrimination.
competing models, the first with higher discrimination (AUC) and worse
calibration, and the the second the other way round. Which one is the
better model?
I judge models on the basis of both discrimination (best measured with
log likelihood measures, 2nd best AUC) and calibration. It's a
two-dimensional issue and we don't always know how to weigh the two.
For many purposes calibration is a must. In those we don't look at
discrimination until calibration-in-the-small is verified at high
resolution.
Frank
--
Frank E Harrell Jr Professor and Chair School of Medicine
Department of Biostatistics Vanderbilt University
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
