Re: [R] Package for .632 (and .632+) bootstrap and the cross-validation of ROC Parameters
spime wrote:
Suppose I have
Training data: my.train
Testing data: my.test
The bootstrap does not need split samples.
I want to calculate bootstrap error rate for logistic model. My wrapper
function for prediction
pred.glm - function(object, newdata) {
ret - as.factor(ifelse(predict.glm(object, newdata,
type='response') 0.4, 0, 1))
return(ret)
}
But i thing i cant understand if i want to calculate misclassification error
for my testing data what will be in my data in the following formula.
Misclassification error has many problems because it is not a proper
scoring rule, i.e., it is optimized by bogus models.
Frank
errorest(RES ~., data=???, model=glm, estimator=boot, predict=pred.glm,
est.para=control.errorest(nboot = 10))
Using my.test got following error,
Error in predict(mymodel, newdata = outbootdata) :
unused argument(s) (newdata = list(RES = c(1, 0, 0, 0, 1, 0, 0, 0,
0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1,
1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0,
0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1,
0), CAT01 = c(4, 4, 2, 4, 4, 4, 4, 4, 4, 2, 1, 2, 2, 4, 4, 4, 1, 1, 2, 2, 1,
4, 1, 4, 1, 4, 2, 4, 1, 4, 2, 3, 1, 1, 3, 3, 4, 2, 4, 2, 1, 2, 2, 1, 1,
please reply...
Frank E Harrell Jr wrote:
spime wrote:
Hi users,
I need to calculate .632 (and .632+) bootstrap and the cross-validation
of
area under curve (AUC) to compare my models. Is there any package for the
same. I know about 'ipred' and using it i can calculate misclassification
errors.
Please help. It's urgent.
See the validate* functions in the Design package.
Note that some simulations (see http://biostat.mc.vanderbilt.edu/rms)
indicate that the advantages of .632 and .632+ over the ordinary
bootstrap are highly dependent on the choice of the accuracy measure
being validated. The bootstrap variants seem to have advantages mainly
if an improper, inefficient, discontinuous scoring rule such as the
percent classified correct is used.
--
Frank E Harrell Jr Professor and Chair School of Medicine
Department of Biostatistics Vanderbilt University
__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
--
Frank E Harrell Jr Professor and Chair School of Medicine
Department of Biostatistics Vanderbilt University
__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Package for .632 (and .632+) bootstrap and the cross-validation of ROC Parameters
Suppose I have
Training data: my.train
Testing data: my.test
I want to calculate bootstrap error rate for logistic model. My wrapper
function for prediction
pred.glm - function(object, newdata) {
ret - as.factor(ifelse(predict.glm(object, newdata,
type='response') 0.4, 0, 1))
return(ret)
}
But i thing i cant understand if i want to calculate misclassification error
for my testing data what will be in my data in the following formula.
errorest(RES ~., data=???, model=glm, estimator=boot, predict=pred.glm,
est.para=control.errorest(nboot = 10))
Using my.test got following error,
Error in predict(mymodel, newdata = outbootdata) :
unused argument(s) (newdata = list(RES = c(1, 0, 0, 0, 1, 0, 0, 0,
0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1,
1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0,
0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1,
0), CAT01 = c(4, 4, 2, 4, 4, 4, 4, 4, 4, 2, 1, 2, 2, 4, 4, 4, 1, 1, 2, 2, 1,
4, 1, 4, 1, 4, 2, 4, 1, 4, 2, 3, 1, 1, 3, 3, 4, 2, 4, 2, 1, 2, 2, 1, 1,
please reply...
Frank E Harrell Jr wrote:
spime wrote:
Hi users,
I need to calculate .632 (and .632+) bootstrap and the cross-validation
of
area under curve (AUC) to compare my models. Is there any package for the
same. I know about 'ipred' and using it i can calculate misclassification
errors.
Please help. It's urgent.
See the validate* functions in the Design package.
Note that some simulations (see http://biostat.mc.vanderbilt.edu/rms)
indicate that the advantages of .632 and .632+ over the ordinary
bootstrap are highly dependent on the choice of the accuracy measure
being validated. The bootstrap variants seem to have advantages mainly
if an improper, inefficient, discontinuous scoring rule such as the
percent classified correct is used.
--
Frank E Harrell Jr Professor and Chair School of Medicine
Department of Biostatistics Vanderbilt University
__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
--
View this message in context:
http://www.nabble.com/Package-for-.632-%28and-.632%2B%29-bootstrap-and-the-cross-validation-of-ROC-Parameters-tf4068544.html#a11578129
Sent from the R help mailing list archive at Nabble.com.
__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] Package for .632 (and .632+) bootstrap and the cross-validation of ROC Parameters
Hi users, I need to calculate .632 (and .632+) bootstrap and the cross-validation of area under curve (AUC) to compare my models. Is there any package for the same. I know about 'ipred' and using it i can calculate misclassification errors. Please help. It's urgent. -- View this message in context: http://www.nabble.com/Package-for-.632-%28and-.632%2B%29-bootstrap-and-the-cross-validation-of-ROC-Parameters-tf4068544.html#a11561405 Sent from the R help mailing list archive at Nabble.com. __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Package for .632 (and .632+) bootstrap and the cross-validation of ROC Parameters
spime wrote: Hi users, I need to calculate .632 (and .632+) bootstrap and the cross-validation of area under curve (AUC) to compare my models. Is there any package for the same. I know about 'ipred' and using it i can calculate misclassification errors. Please help. It's urgent. See the validate* functions in the Design package. Note that some simulations (see http://biostat.mc.vanderbilt.edu/rms) indicate that the advantages of .632 and .632+ over the ordinary bootstrap are highly dependent on the choice of the accuracy measure being validated. The bootstrap variants seem to have advantages mainly if an improper, inefficient, discontinuous scoring rule such as the percent classified correct is used. -- Frank E Harrell Jr Professor and Chair School of Medicine Department of Biostatistics Vanderbilt University __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
