Re: [R] ROC curve for each fold in one plot
Hi I did not see any response but what about par(mfrow=c(2,2)) before plotting? Cheers Petr > -Original Message- > From: R-help [mailto:r-help-boun...@r-project.org] On Behalf Of Elahe chalabi > via R-help > Sent: Monday, October 16, 2017 2:26 PM > To: R-help Mailing List <r-help@r-project.org> > Subject: [R] ROC curve for each fold in one plot > > Hi all, > > I have tried a 5 fold cross validation using caret package with random forest > method on iris dataset as example. Then I need ROC curve for each fold: > > > > set.seed(1) > > train_control <- trainControl(method="cv", number=5,savePredictions = > TRUE,classProbs = TRUE) > > output <- train(Species~., data=iris, trControl=train_control, > method="rf") > > library(pROC) > > selectedIndices <- output$pred$Resample == "Fold1" > > > plot.roc(output$pred$obs[selectedIndices],output$pred$setosa[selectedIndices > ]) > selectedIndices <- output$pred$Resample == "Fold2" > > > plot.roc(output$pred$obs[selectedIndices],output$pred$setosa[selectedIndices > ]) > > selectedIndices <- output$pred$Resample == "Fold3" > > > plot.roc(output$pred$obs[selectedIndices],output$pred$setosa[selectedIndices > ]) > > and the same for Fold4 and Fold5,now how can I bring all the plots in one plot > with labels for each fold? > > Thanks for any help! > Elahe > > __ > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. Tento e-mail a jakékoliv k němu připojené dokumenty jsou důvěrné a jsou určeny pouze jeho adresátům. Jestliže jste obdržel(a) tento e-mail omylem, informujte laskavě neprodleně jeho odesílatele. Obsah tohoto emailu i s přílohami a jeho kopie vymažte ze svého systému. Nejste-li zamýšleným adresátem tohoto emailu, nejste oprávněni tento email jakkoliv užívat, rozšiřovat, kopírovat či zveřejňovat. Odesílatel e-mailu neodpovídá za eventuální škodu způsobenou modifikacemi či zpožděním přenosu e-mailu. V případě, že je tento e-mail součástí obchodního jednání: - vyhrazuje si odesílatel právo ukončit kdykoliv jednání o uzavření smlouvy, a to z jakéhokoliv důvodu i bez uvedení důvodu. - a obsahuje-li nabídku, je adresát oprávněn nabídku bezodkladně přijmout; Odesílatel tohoto e-mailu (nabídky) vylučuje přijetí nabídky ze strany příjemce s dodatkem či odchylkou. - trvá odesílatel na tom, že příslušná smlouva je uzavřena teprve výslovným dosažením shody na všech jejích náležitostech. - odesílatel tohoto emailu informuje, že není oprávněn uzavírat za společnost žádné smlouvy s výjimkou případů, kdy k tomu byl písemně zmocněn nebo písemně pověřen a takové pověření nebo plná moc byly adresátovi tohoto emailu případně osobě, kterou adresát zastupuje, předloženy nebo jejich existence je adresátovi či osobě jím zastoupené známá. This e-mail and any documents attached to it may be confidential and are intended only for its intended recipients. If you received this e-mail by mistake, please immediately inform its sender. Delete the contents of this e-mail with all attachments and its copies from your system. If you are not the intended recipient of this e-mail, you are not authorized to use, disseminate, copy or disclose this e-mail in any manner. The sender of this e-mail shall not be liable for any possible damage caused by modifications of the e-mail or by delay with transfer of the email. In case that this e-mail forms part of business dealings: - the sender reserves the right to end negotiations about entering into a contract in any time, for any reason, and without stating any reasoning. - if the e-mail contains an offer, the recipient is entitled to immediately accept such offer; The sender of this e-mail (offer) excludes any acceptance of the offer on the part of the recipient containing any amendment or variation. - the sender insists on that the respective contract is concluded only upon an express mutual agreement on all its aspects. - the sender of this e-mail informs that he/she is not authorized to enter into any contracts on behalf of the company except for cases in which he/she is expressly authorized to do so in writing, and such authorization or power of attorney is submitted to the recipient or the person represented by the recipient, or the existence of such authorization is known to the recipient of the person represented by the recipient. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] ROC curve for each fold in one plot
Hi all, I have tried a 5 fold cross validation using caret package with random forest method on iris dataset as example. Then I need ROC curve for each fold: > set.seed(1) > train_control <- trainControl(method="cv", number=5,savePredictions = TRUE,classProbs = TRUE) > output <- train(Species~., data=iris, trControl=train_control, method="rf") > library(pROC) > selectedIndices <- output$pred$Resample == "Fold1" > plot.roc(output$pred$obs[selectedIndices],output$pred$setosa[selectedIndices]) > selectedIndices <- output$pred$Resample == "Fold2" > plot.roc(output$pred$obs[selectedIndices],output$pred$setosa[selectedIndices]) > selectedIndices <- output$pred$Resample == "Fold3" > plot.roc(output$pred$obs[selectedIndices],output$pred$setosa[selectedIndices]) and the same for Fold4 and Fold5,now how can I bring all the plots in one plot with labels for each fold? Thanks for any help! Elahe __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] ROC Curve: negative AUC
Thanks, Do you know how to correct it? Cheers, Bruno -- View this message in context: http://r.789695.n4.nabble.com/ROC-Curve-negative-AUC-tp4650469p4650530.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] ROC Curve: negative AUC
Hi all, does anyone know why the area under the curve (AUC) is negative? I'm using ROC function with a logistic regression, package Epi. First time it happens... Thanks a lot! Bruno -- View this message in context: http://r.789695.n4.nabble.com/ROC-Curve-negative-AUC-tp4650469.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] ROC Curve: negative AUC
wrong direction for ranking On Thu, Nov 22, 2012 at 1:58 PM, brunosm brunos...@gmail.com wrote: the area under the curve (AUC) is negative? I'm using ROC function with a logistic regression, package Epi. First time it happens... Thanks a lot! Bruno -- == WenSui Liu Credit Risk Manager, 53 Bancorp wensui@53.com 513-295-4370 == [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] ROC Curve: negative AUC
On Fri, Nov 23, 2012 at 7:58 AM, brunosm brunos...@gmail.com wrote: Hi all, does anyone know why the area under the curve (AUC) is negative? I'm using ROC function with a logistic regression, package Epi. First time it happens... Have you looked at the ROC curve? That should tell you whether there's some strange non-convexity going on or whether the variable is just being put into the calculations backwards. -thomas -- Thomas Lumley Professor of Biostatistics University of Auckland [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] ROC curve
On 01/30/2012 08:52 PM, Josiane NJIWA wrote: Hello all, I am very new to R and i am facing two problems. First i didn't succeed changing the konsole language in english even after trying the line command set language='en'. I would like to plot ROC curves. I have a serie of 10 threshold tests that i do for 10 patients. The prediction for the patients is always the same but the status can change given to the considered threshold. I have 11 columns of 10 rows, the first colums containing the10 lines of the predicted status of the patients (0=cured, 1=non cured). Then follow 10 columns (10 thresholds) containing the found status using the threshold. Please do someone know how i can use those values with R to plot ROC curves? Hi Josiane, The function roc.from.table in the epicalc package will produce a basic ROC curve. You will have to get the table that you send to the function in the correct form. I think you will have to lay out the table like this: Predicted Above threshold Threshold1 x x Threshold2 x x Threshold3 x x Threshold4 x x Threshold5 x x Threshold6 x x Threshold7 x x Threshold8 x x Threshold9 x x Threshold10 x x I may be quite wrong as I don't know exactly what the successive threshold values may mean. You will probably want to run the function with grid=FALSE. Jim __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] ROC curve
Hello all, I am very new to R and i am facing two problems. First i didn't succeed changing the konsole language in english even after trying the line command set language='en'. I would like to plot ROC curves. I have a serie of 10 threshold tests that i do for 10 patients. The prediction for the patients is always the same but the status can change given to the considered threshold. I have 11 columns of 10 rows, the first colums containing the10 lines of the predicted status of the patients (0=cured, 1=non cured). Then follow 10 columns (10 thresholds) containing the found status using the threshold. Please do someone know how i can use those values with R to plot ROC curves? I thank you for your understanding, Josiane. Everything should be made as simple as possible, but not simpler.Albert Einstein. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] ROC curve
Hi Josiane, Concerning ROC curves, the package ROCR should do what you want to do. Use install.packages to add it to you library. Getting you data into a text file format, use read.delim to read into an data frame. Once you have a data frame, you can use the methods in ROCR to analyze the data. Best, Corey On Mon, Jan 30, 2012 at 1:52 AM, Josiane NJIWA joa...@yahoo.com wrote: Hello all, I am very new to R and i am facing two problems. First i didn't succeed changing the konsole language in english even after trying the line command set language='en'. I would like to plot ROC curves. I have a serie of 10 threshold tests that i do for 10 patients. The prediction for the patients is always the same but the status can change given to the considered threshold. I have 11 columns of 10 rows, the first colums containing the10 lines of the predicted status of the patients (0=cured, 1=non cured). Then follow 10 columns (10 thresholds) containing the found status using the threshold. Please do someone know how i can use those values with R to plot ROC curves? I thank you for your understanding, Josiane. Everything should be made as simple as possible, but not simpler. Albert Einstein. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- *The mark of a successful man is one that has spent an entire day on the bank of a river without feeling guilty about it.* [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] ROC curve
On Jan 30, 2012, at 4:52 AM, Josiane NJIWA wrote: Hello all, I am very new to R and i am facing two problems. First i didn't succeed changing the konsole language in english even after trying the line command set language='en'. R is a functional language, so it shouldn't surprise you that issuing a command does not do what you apparently expected based on your experience with macro languages. You should read: ?locales I would like to plot ROC curves. I have a serie of 10 threshold tests that i do for 10 patients. The prediction for the patients is always the same but the status can change given to the considered threshold. I have 11 columns of 10 rows, the first colums containing the10 lines of the predicted status of the patients (0=cured, 1=non cured). Then follow 10 columns (10 thresholds) containing the found status using the threshold. Please do someone know how i can use those values with R to plot ROC curves? I thank you for your understanding, Josiane. Everything should be made as simple as possible, but not simpler.Albert Einstein. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. David Winsemius, MD Heritage Laboratories West Hartford, CT __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] ROC curve in R
Hi, i have a fairly large amount of genomic data. I have created a dataframe which has Reference as one column and Variation as another. I want to plot a ROC curve based on these 2 columns. I have serached the R manual but I could not understand. Can anybody help me with the R code for plotting ROC curve. Thnx ashu6886 -- View this message in context: http://r.789695.n4.nabble.com/ROC-curve-in-R-tp2275431p2275431.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] ROC curve in R
On Jul 1, 2010, at 12:50 PM, ashu6886 wrote: Hi, i have a fairly large amount of genomic data. I have created a dataframe which has Reference as one column and Variation as another. I want to plot a ROC curve based on these 2 columns. I have serached the R manual but I could not understand. Can anybody help me with the R code for plotting ROC curve. Not sure what sort of R manual you consulted, but surely any basic introduction must have covered how to search. In the interest of moving you a bit faster perhaps: ?RSiteSearch -- David Winsemius, MD West Hartford, CT __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] ROC curve in R
Did you try googling on R ROC function (or something similar) or using the RSiteSearch() function? ?RSiteSearch RSiteSearch(ROC,restr=func) Learn to use R's various search capabilities before posting, please! Bert Gunter Genentech Nonclinical Biostatistics -Original Message- From: r-help-boun...@r-project.org [mailto:r-help-boun...@r-project.org] On Behalf Of ashu6886 Sent: Thursday, July 01, 2010 9:51 AM To: r-help@r-project.org Subject: [R] ROC curve in R Hi, i have a fairly large amount of genomic data. I have created a dataframe which has Reference as one column and Variation as another. I want to plot a ROC curve based on these 2 columns. I have serached the R manual but I could not understand. Can anybody help me with the R code for plotting ROC curve. Thnx ashu6886 -- View this message in context: http://r.789695.n4.nabble.com/ROC-curve-in-R-tp2275431p2275431.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] ROC curve in R
Try the ROCR package. http://rocr.bioinf.mpi-sb.mpg.de/ROCR.pdf Saeed On Thu, Jul 1, 2010 at 9:50 AM, ashu6886 ashu.infy.m...@gmail.com wrote: Hi, i have a fairly large amount of genomic data. I have created a dataframe which has Reference as one column and Variation as another. I want to plot a ROC curve based on these 2 columns. I have serached the R manual but I could not understand. Can anybody help me with the R code for plotting ROC curve. Thnx ashu6886 -- View this message in context: http://r.789695.n4.nabble.com/ROC-curve-in-R-tp2275431p2275431.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] ROC curve in R
Regarding search, There is also the sos packege. And yesterday someone wrote a nice R code to enable searching inside the description of all packages: http://www.r-bloggers.com/cran-search/ Cheers, Tal Contact Details:--- Contact me: tal.gal...@gmail.com | 972-52-7275845 Read me: www.talgalili.com (Hebrew) | www.biostatistics.co.il (Hebrew) | www.r-statistics.com (English) -- On Thu, Jul 1, 2010 at 8:09 PM, Bert Gunter gunter.ber...@gene.com wrote: Did you try googling on R ROC function (or something similar) or using the RSiteSearch() function? ?RSiteSearch RSiteSearch(ROC,restr=func) Learn to use R's various search capabilities before posting, please! Bert Gunter Genentech Nonclinical Biostatistics -Original Message- From: r-help-boun...@r-project.org [mailto:r-help-boun...@r-project.org] On Behalf Of ashu6886 Sent: Thursday, July 01, 2010 9:51 AM To: r-help@r-project.org Subject: [R] ROC curve in R Hi, i have a fairly large amount of genomic data. I have created a dataframe which has Reference as one column and Variation as another. I want to plot a ROC curve based on these 2 columns. I have serached the R manual but I could not understand. Can anybody help me with the R code for plotting ROC curve. Thnx ashu6886 -- View this message in context: http://r.789695.n4.nabble.com/ROC-curve-in-R-tp2275431p2275431.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] ROC curve in R
Read the ROCR package, it is very good. On Thu, Jul 1, 2010 at 9:50 AM, ashu6886 ashu.infy.m...@gmail.com wrote: Hi, i have a fairly large amount of genomic data. I have created a dataframe which has Reference as one column and Variation as another. I want to plot a ROC curve based on these 2 columns. I have serached the R manual but I could not understand. Can anybody help me with the R code for plotting ROC curve. Thnx ashu6886 -- View this message in context: http://r.789695.n4.nabble.com/ROC-curve-in-R-tp2275431p2275431.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Sincerely, Changbin -- Changbin Du DOE Joint Genome Institute Bldg 400 Rm 457 2800 Mitchell Dr Walnut Creet, CA 94598 Phone: 925-927-2856 [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] ROC curve in R
On 07/01/2010 01:33 PM, Changbin Du wrote: Read the ROCR package, it is very good. Just be sure you really need an ROC curve. More often than not it gets in the way of understanding. Frank On Thu, Jul 1, 2010 at 9:50 AM, ashu6886ashu.infy.m...@gmail.com wrote: Hi, i have a fairly large amount of genomic data. I have created a dataframe which has Reference as one column and Variation as another. I want to plot a ROC curve based on these 2 columns. I have serached the R manual but I could not understand. Can anybody help me with the R code for plotting ROC curve. Thnx ashu6886 -- View this message in context: http://r.789695.n4.nabble.com/ROC-curve-in-R-tp2275431p2275431.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Frank E Harrell Jr Professor and ChairmanSchool of Medicine Department of Biostatistics Vanderbilt University __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] ROC curve
Dear Changbin, I want to know how to select the optimal decision threshold from the ROC curve? Depends on what optimal means. I think there are a bunch of different criteria used: - point closest to the ideal model - point furthest from the guessing model - these criteria may include costs, i.e. a FP/FN ratio != 1 - ... More practical: If you use ROCR: the help of the performance class explains the slots in the object. You find there the data of the curve, incl. the thresholds. At what threshold will give the highest accuracy? to know that, optmize the accuracy as function of the threshold. Remember: finding the optimal threshold from a ROC curve is a data-driven optimization. You need to validate the resulting model with independent test data afterwards. -- Claudia Beleites Dipartimento dei Materiali e delle Risorse Naturali Università degli Studi di Trieste Via Alfonso Valerio 6/a I-34127 Trieste phone: +39 0 40 5 58-37 68 email: cbelei...@units.it __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] ROC curve
On 05/24/2010 02:14 AM, Claudia Beleites wrote: Dear Changbin, I want to know how to select the optimal decision threshold from the ROC curve? Depends on what optimal means. I think there are a bunch of different criteria used: - point closest to the ideal model - point furthest from the guessing model - these criteria may include costs, i.e. a FP/FN ratio != 1 - ... More practical: If you use ROCR: the help of the performance class explains the slots in the object. You find there the data of the curve, incl. the thresholds. At what threshold will give the highest accuracy? to know that, optmize the accuracy as function of the threshold. Remember: finding the optimal threshold from a ROC curve is a data-driven optimization. You need to validate the resulting model with independent test data afterwards. That point is excellent. In addition, such decision analysis assumes that (1) a forced yes/no decision is acceptable, i.e., a predicted probability in the middle is forced to be categorized as low or high as opposed to no decision; get more data, and (2) the utility/cost/loss function is identical across subjects (which it almost never is). Frank -- Frank E Harrell Jr Professor and ChairmanSchool of Medicine Department of Biostatistics Vanderbilt University __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] ROC curve
HI, Dear R community, I want to know how to select the optimal decision threshold from the ROC curve? At what threshold will give the highest accuracy? Thanks! -- Sincerely, Changbin -- [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] ROC curve in randomForest
require(randomForest) rf.pred-predict(fit, valid, type=prob) rf.pred[1:20, ] 0 1 16 0. 1. 23 0.3158 0.6842 43 0.3030 0.6970 52 0.0886 0.9114 55 0.1216 0.8784 75 0.0920 0.9080 82 0.4332 0.5668 120 0.2302 0.7698 128 0.1336 0.8664 147 0.4272 0.5728 148 0.0490 0.9510 153 0.0556 0.9444 161 0.0760 0.9240 162 0.4564 0.5436 172 0.5148 0.4852 176 0.1730 0.8270 215 0.0100 0.9900 222 0.0104 0.9896 255 0.1358 0.8642 267 0.1000 0.9000 rf.pred-predict(fit, valid, type=prob) rf.roc - prediction(rf.pred[,2], valid[,32]) rf.auc - performance(rf.roc, 'tpr', 'fpr') plot(rf.auc) Dear R community, I am plot the ROC curve for randomforest result, I am not sure the above code is right or not? I just used the second column of the probabilities, the pr(outcome=1). Can anyone give some comments? someguys use the following codes to plot, but I can not use the treeresponse commad in randomforest object directly. # create model using random forest and bagging ensemble using conditional inference trees 035x.cf - cforest(Class ~ ., data=BreastCancer[ind == 1,], control = cforest_unbiased(mtry = ncol(BreastCancer)-2)) 036x.cf.pred - predict(x.cf, newdata=BreastCancer[ind == 2,]) 037x.cf.prob - 1- unlist(treeresponse(x.cf, BreastCancer[ind == 2,]), use.names=F)[seq(1,nrow(BreastCancer[ind == 2,])*2,2)] 038 -- Sincerely, Changbin -- Changbin Du DOE Joint Genome Institute Bldg 400 Rm 457 2800 Mitchell Dr Walnut Creet, CA 94598 Phone: 925-927-2856 [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] ROC curve in randomSurvivalForest
I'm using randomSurvivalForest to predict survival from a rather small sample. As it's not enough data to have training and validation set, I rely on the Estimate of error rate computed by the randomForest. If I understand the method correctly, it repeatedly partitions the data into varying training/validation sets during the learning steps, which also produces the estimate of error. My questions is, would it be possible to compute a ROC curve during RF training? A possible approach I considered would be to train the RF on a subset of the data and create a ROC curve from the prediction on the remaining data. By repeating this process, I would get the variation of the ROC curve for the different possible data subsets. But this seems to be not such an elegant solution, as this could be done during one instance of RF learning. It would be very helpful, if someone could point to an approach on how to do this. Nick Fankhauser __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] ROC curve and gains/lift chart
What is the difference between ROC curve and gains/lift chart? how to do them in R? Thanks. -- View this message in context: http://www.nabble.com/ROC-curve-and-gains-lift-chart-tp25083979p25083979.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] ROC curve and gains/lift chart
Have look in the ROCR package and Example's. HTH Christian What is the difference between ROC curve and gains/lift chart? how to do them in R? Thanks. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] ROC curve using epicalc (after logistic regression) (re-sent)
Dear R-help, I am resending as I believe I screwed up the e-mail address to R-help earlier. Sorry for my lack of attention to detail, and for any inconvenience. I have also sent the question to the package maintainer, as suggested in the posting guide. Regards, Cliff -- Forwarded message -- From: Clifford Long gnolff...@gmail.com Date: Sun, Jul 26, 2009 at 8:46 PM Subject: Fwd: ROC curve using epicalc (after logistic regression) To: cvira...@medicine.psu.ac.th Dear Virasakdi Chongsuvivatwong, After sending the message below to the R-help mailing list, it occurred to me that I probably should also have sent a copy to you, per R posting guidance. I would be interested in any thoughts or suggestions that you might have regarding my difficulty using the ROCR routine in the epicalc package. (I've used this before, and find it to be a very helpful package ... thanks.) Is my issue related to the way the data is structured for the glm routine - meaning not with individual cases, but instead by counts (per DOE treatment) of pass, fail, and total? Or perhaps I've made another error? I'll understand if you don't have the time to look this over. In case you do, any direction/guidance will be appreciated. Thank you for your time, and for this excellent package. Regards, Cliff Long -- Forwarded message -- From: Clifford Long gnolff...@gmail.com Date: Sun, Jul 26, 2009 at 3:52 PM Subject: ROC curve using epicalc (after logistic regression) To: R-help@r-project.org Dear R-help list, I'm attempting to use the ROC routine from the epicalc package after performing a logistic regression analysis. My code is included after the sessionInfo() result. The datafile (GasketMelt1.csv) is attached. I updated both R and the epicalc packages and tried again before sending this request. sessionInfo result: R version 2.9.1 (2009-06-26) i386-pc-mingw32 locale: LC_COLLATE=English_United States.1252;LC_CTYPE=English_United States.1252;LC_MONETARY=English_United States.1252;LC_NUMERIC=C;LC_TIME=English_United States.1252 attached base packages: [1] splines stats graphics grDevices utils datasets methods [8] base other attached packages: [1] caret_4.19 lattice_0.17-25 epicalc_2.9.1.2 survival_2.35-4 [5] foreign_0.8-36 loaded via a namespace (and not attached): [1] grid_2.9.1 tools_2.9.1 Header information from package 'epicalc': Package: epicalc Version: 2.9.1.2 Date: 2009-07-14 My code ... # # Logistic Regression (the model result is as expected) # dfile = 'GasketMelt1.csv' gmelt.df = read.csv(dfile, header = TRUE, as.is = TRUE) names(gmelt.df) gmelt.df$p = gmelt.df$Pass / gmelt.df$Total gmelt.glm = glm(p ~ Time + Temperature + Depth + Time*Temperature + Time*Depth + Temperature*Depth, family = binomial(link = logit), data=gmelt.df, weight=Total) summary(gmelt.glm) # # ROC # library(epicalc) lroc(gmelt.glm, graph = TRUE, line.col = red) The error message: lroc(gmelt.glm, graph = TRUE, line.col = red) Error in dimnames(x) - dn : length of 'dimnames' [2] not equal to array extent Have I overlooked something? Many thanks to anyone who might have a suggestion. Cliff __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] ROC curve using epicalc (after logistic regression)
Dear R-help list, I'm attempting to use the ROC routine from the epicalc package after performing a logistic regression analysis. My code is included after the sessionInfo() result. The datafile (GasketMelt1.csv) is attached. I updated both R and the epicalc packages and tried again before sending this request. sessionInfo result: R version 2.9.1 (2009-06-26) i386-pc-mingw32 locale: LC_COLLATE=English_United States.1252;LC_CTYPE=English_United States.1252;LC_MONETARY=English_United States.1252;LC_NUMERIC=C;LC_TIME=English_United States.1252 attached base packages: [1] splines stats graphics grDevices utils datasets methods [8] base other attached packages: [1] caret_4.19 lattice_0.17-25 epicalc_2.9.1.2 survival_2.35-4 [5] foreign_0.8-36 loaded via a namespace (and not attached): [1] grid_2.9.1 tools_2.9.1 Header information from package 'epicalc': Package:epicalc Version:2.9.1.2 Date: 2009-07-14 My code ... # # Logistic Regression (the model result is as expected) # dfile = 'GasketMelt1.csv' gmelt.df = read.csv(dfile, header = TRUE, as.is = TRUE) names(gmelt.df) gmelt.df$p = gmelt.df$Pass / gmelt.df$Total gmelt.glm = glm(p ~ Time + Temperature + Depth + Time*Temperature + Time*Depth + Temperature*Depth, family = binomial(link = logit), data=gmelt.df, weight=Total) summary(gmelt.glm) # # ROC # library(epicalc) lroc(gmelt.glm, graph = TRUE, line.col = red) The error message: lroc(gmelt.glm, graph = TRUE, line.col = red) Error in dimnames(x) - dn : length of 'dimnames' [2] not equal to array extent Have I overlooked something? Many thanks to anyone who might have a suggestion. Cliff __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] ROC curve from logistic regression
I know how to compute the ROC curve and the empirical AUC from the logistic regression after fitting the model. But here is my question, how can I compute the standard error for the AUC estimator resulting form logistic regression? The variance should be more complicated than AUC based on known test results. Does anybody know a reference on this problem? [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] ROC curve from logistic regression
gallon li wrote: I know how to compute the ROC curve and the empirical AUC from the logistic regression after fitting the model. But here is my question, how can I compute the standard error for the AUC estimator resulting form logistic regression? The variance should be more complicated than AUC based on known test results. Does anybody know a reference on this problem? The rcorr.cens function in the Hmisc package will compute the std. error of Somers' Dxy rank correlation. Dxy = 2*(C-.5) where C is the ROC area. This standard error does not include a variance component for the uncertainty in the model (e.g., it does not penalize for the estimation of the regression coefficients if you are estimating the coefficients and assessing ROC area on the same sample). The lrm function in the Design package fits binary and ordinal logistic regression models and reports C, Dxy, and other measures. I haven't seen an example where drawing the ROC curve provides useful information that leads to correct actions. Frank -- Frank E Harrell Jr Professor and Chair School of Medicine Department of Biostatistics Vanderbilt University __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] ROC curve from logistic regression
Hi
Try the following reference:
Comparison of Three Methods for Estimating the
Standard Error of the Area under the Curve in ROC
Analysis of Quantitative Data by Hajian-Tilaki and Hanley, Academic
Radiology, Vol 9, No 11, November 2002.
Below is a simple implementation that will return both the AUC and its
standard error (DeLong et al method).
Hope this helps...
Pedro
#Input: yreal [-1,1]
auc - function(yreal,forecasts){
sizeT -nrow(yreal)
pos - 0
for(i in 1:sizeT){
if(yreal[i]0) {pos - pos + 1}
}
neg - sizeT-pos
yrealpos - vector(length=pos)
yrealneg - vector(length=neg)
forepos - vector(length=pos)
foreneg - vector(length=neg)
controlpos - 1
controlneg - 1
for(i in 1:sizeT){
if(yreal[i]0) {
yrealpos[controlpos] - yreal[i]
forepos[controlpos] - forecasts[i]
controlpos - controlpos + 1
} else {
yrealneg[controlneg] - yreal[i]
foreneg[controlneg] - forecasts[i]
controlneg - controlneg + 1
}
}
oper - 0
for( i in 1:pos){
for(j in 1:neg){
if(forepos[i] foreneg[j]) {oper - oper + 1}
if(forepos[i]==foreneg[j]) {oper - oper + 0.50
}
}
}
area - oper/(pos*neg)
vpj - vector(length=pos)
vqk - vector(length=neg)
oper - 0
for(i in 1:pos){
for(j in 1:neg){
if(forepos[i] foreneg[j]) {oper -
oper + 1
} else {if(forepos[i]==foreneg[j]) {oper
- oper + 0.50 }}
}
division - oper/neg
resta - (division-area)^2
vpj[i] - resta
oper - 0
}
oper - 0
resta - 0
for(j in 1:neg){
for(i in 1:pos){
if(forepos[i] foreneg[j]) {oper -
oper + 1
} else {if(forepos[i]==foreneg[j]) {oper
- oper + 0.50 }}
}
division - oper/pos
resta - (division-area)^2
vqk[j] - resta
oper - 0
}
vpj - vpj/(pos*(pos-1))
vqk - vqk/(neg*(neg-1))
var - sum(vpj)+sum(vqk)
s - sqrt(var)
return(list(AUC=area, std=s))
}
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
On Behalf Of Frank E Harrell Jr
Sent: Monday, September 08, 2008 8:22 AM
To: gallon li
Cc: r-help
Subject: Re: [R] ROC curve from logistic regression
gallon li wrote:
I know how to compute the ROC curve and the empirical AUC from the
logistic
regression after fitting the model.
But here is my question, how can I compute the standard error for the
AUC
estimator resulting form logistic regression? The variance should be
more
complicated than AUC based on known test results. Does anybody know a
reference on this problem?
The rcorr.cens function in the Hmisc package will compute the std. error
of Somers' Dxy rank correlation. Dxy = 2*(C-.5) where C is the ROC
area. This standard error does not include a variance component for the
uncertainty in the model (e.g., it does not penalize for the estimation
of the regression coefficients if you are estimating the coefficients
and assessing ROC area on the same sample).
The lrm function in the Design package fits binary and ordinal logistic
regression models and reports C, Dxy, and other measures.
I haven't seen an example where drawing the ROC curve provides useful
information that leads to correct actions.
Frank
--
Frank E Harrell Jr Professor and Chair School of Medicine
Department of Biostatistics Vanderbilt
University
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] ROC curve from logistic regression
Hi there, See ?lroc in the epicalc package. HTH, Jorge On Mon, Sep 8, 2008 at 4:02 AM, gallon li [EMAIL PROTECTED] wrote: I know how to compute the ROC curve and the empirical AUC from the logistic regression after fitting the model. But here is my question, how can I compute the standard error for the AUC estimator resulting form logistic regression? The variance should be more complicated than AUC based on known test results. Does anybody know a reference on this problem? [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
