Re: [R] predict.lrm ( Design package) poor performance?

2010-09-22 Thread Frank Harrell
% correct is an improper scoring rule and a discontinuous one to boot. So it will not always agree with more proper scoring rules. When you have a more difficult task, e.g., discriminating more categories, indexes such as the generalized c-index that utilize all the categories will recognize

Re: [R] predict.lrm ( Design package) poor performance?

2010-09-22 Thread Chris Mcowen
Thats great thanks I guess it is hard to not use % as a performance measure when that is what is commonly used in everyday life. So when i come to predicting the response of new data ( using the estimated mean Y ) which i am more comfortable with i can say - Species A - 2.12 - Therefore this

Re: [R] predict.lrm ( Design package) poor performance?

2010-09-22 Thread Chris Mcowen
Thats great thanks, I suppose it is hard to move away from a more traditional measure of performance such a percentage correct, at least for the relatively amateur statisticians among us who have been graded on such a system. The difficulty comes in reporting the effectiveness of the model to

[R] predict.lrm ( Design package) poor performance?

2010-09-21 Thread Chris Mcowen
Thanks Frank, I have one small question regarding this, understand you are very busy and if you cant answer i would greatly appreciate any thoughts from the list. Split-sample validation is not reliable unless you have say 10,000 samples to begin with I am a little confused. When i ran

Re: [R] predict.lrm ( Design package) poor performance?

2010-09-21 Thread Chris Mcowen
Thanks Frank, I have one small question regarding this, understand you are very busy and if you cant answer i would greatly appreciate any thoughts from the list. Split-sample validation is not reliable unless you have say 10,000 samples to begin with I am a little confused. When i ran the