[R] Best performance measure?

Hello, I working on a model to predict probabilities. I don't really care about binary prediction accuracy. I do really care about the accuracy of my probability predictions. Frank was nice enough to point me to the val.prob function from the Design library. It looks very promising for my

Re: [R] Best performance measure?

Noah Silverman wrote: Hello, I working on a model to predict probabilities. I don't really care about binary prediction accuracy. I do really care about the accuracy of my probability predictions. Frank was nice enough to point me to the val.prob function from the Design library. It looks

Re: [R] Best performance measure?

Thanks for the suggestion. You explained that Briar combines both accuracy and discrimination ability. If I understand you right, that is in relation to binary classification. I'm not concerned with binary classification, but the accuracy of the probability predictions. Is there some kind

Re: [R] Best performance measure?

Frank, That makes sense. I just had a look at the actual algorithm calculating the Briar score. One thing that confuses me is how the score is calculated. If I understand the code correctly, it is just: sum((p - y)^2)/n If I have an example with a label of 1 and a probability prediction of

Re: [R] Best performance measure?

Noah Silverman wrote: Thanks for the suggestion. You explained that Briar combines both accuracy and discrimination ability. If I understand you right, that is in relation to binary classification. I'm not concerned with binary classification, but the accuracy of the probability

Re: [R] Best performance measure?

Noah Silverman wrote: Frank, That makes sense. I just had a look at the actual algorithm calculating the Briar score. One thing that confuses me is how the score is calculated. If I understand the code correctly, it is just: sum((p - y)^2)/n If I have an example with a label of 1 and a

Re: [R] Best performance measure?

Frank, Visually, the loess curve really helps me see how the model is doing. That leads me to two more questions: 1) Can I somehow summarize the loess curve into a single value? (If I'm comparing a few hundred models/parameters it would be nice to have a single performance value to use.) 2)

Re: [R] Best performance measure?

Noah Silverman wrote: Frank, Visually, the loess curve really helps me see how the model is doing. That leads me to two more questions: 1) Can I somehow summarize the loess curve into a single value? (If I'm comparing a few hundred models/parameters it would be nice to have a single