[EMAIL PROTECTED] wrote:
Hi Frank,
Thanks for your feedback! But I think we are talking about two different
things.
1) Validation: The generalization performance of the classifier. See,
for example, "Studies on the Validation of Internal Rating Systems" by
BIS.
I didn't think the desire was for a classifier but instead was for a
risk predictor. If prediction is the goal, classification methods or
accuracy indexes based on classifications do not work very well.
2) Calibration: Correct calibration of a PD rating system means that the
calibrated PD estimates are accurate and conform to the observed default
rates. See, for instance, An Overview and Framework for
PD Backtesting and Benchmarking, by Castermans et al.
I'm unclear on what you mean here. Correct calibration of a predictive
system means that the UNcalibrated estimates are accurate (i.e., they
don't need any calibration). (What is PD?)
Frank, you are referring the #1 and I am referring to #2.
Nonetheless, I would never create a rating system if my model doesn't
discriminate better than a coin toss.
For sure
Frank
Regards,
Pedro
-----Original Message-----
From: Frank E Harrell Jr [mailto:[EMAIL PROTECTED]
Sent: Tuesday, October 07, 2008 11:02 AM
To: Rodriguez, Pedro
Cc: [EMAIL PROTECTED]; r-help@r-project.org
Subject: Re: [R] How to validate model?
[EMAIL PROTECTED] wrote:
Usually one validates scorecards with the ROC curve, Pietra Index, KS
test, etc. You may be interested in the WP 14 from BIS (www.bis.org).
Regards,
Pedro
No, the validation should be done using an absolute reliability
(calibration) curve. You need to verify that at all levels of predicted
risk there is agreement with the true probability of failure. An ROC
curve does not do that, and I doubt the others do. A
resampling-corrected loess calibration curve is a good approach as
implemented in the Design package's calibrate function.
Frank
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]
On Behalf Of Maithili Shiva
Sent: Tuesday, October 07, 2008 8:22 AM
To: r-help@r-project.org
Subject: [R] How to validate model?
Hi!
I am working on scorecard model and I have arrived at the regression
equation. I have used logistic regression using R.
My question is how do I validate this model? I do have hold out sample
of 5000 customers.
Please guide me. Problem is I had never used Logistic regression
earlier
neither I am used to credit scoring models.
Thanks in advance
Maithili
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
--
Frank E Harrell Jr Professor and Chair School of Medicine
Department of Biostatistics Vanderbilt University
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
