Hi, Raga, > 1. Just to make sure I understand correctly, using the .632+ bootstrap > method, the ACC_lower and ACC_upper are the lower and higher percentile of > the ACC_h,i distribution?
phew, I am actually not sure anymore … I think it’s the percentile of the ACC_boot distribution, similar to the “classic” bootstrap but where ACC_boot got computed from weighted ACC_h,i and ACC_r,i > 2. For regression algorithms, is there a recommended equation for the > no-information rate gamma? Sorry, can’t be of much help here; I am not sure what the equivalent of the no-information rate for regression would be ... > On Mar 1, 2017, at 5:39 PM, Raga Markely <raga.mark...@gmail.com> wrote: > > Thanks a lot, Sebastian! Very nicely written. > > I have a few follow-up questions: > 1. Just to make sure I understand correctly, using the .632+ bootstrap > method, the ACC_lower and ACC_upper are the lower and higher percentile of > the ACC_h,i distribution? > 2. For regression algorithms, is there a recommended equation for the > no-information rate gamma? > 3. I need to plot the confidence interval and prediction interval for my > Support Vector Regression prediction (just to clarify these intervals, please > see an analogy from linear model on slide 14: > http://www2.stat.duke.edu/~tjl13/s101/slides/unit6lec3H.pdf) - can I derive > the intervals from .632+ bootstrap method or is there a different way of > getting these intervals? > > Thank you! > Raga > > > On Wed, Mar 1, 2017 at 3:13 PM, Sebastian Raschka <se.rasc...@gmail.com> > wrote: > Hi, Raga, > I have a short section on this here > (https://sebastianraschka.com/blog/2016/model-evaluation-selection-part2.html#the-bootstrap-method-and-empirical-confidence-intervals) > if it helps. > > Best, > Sebastian > > > On Mar 1, 2017, at 3:07 PM, Raga Markely <raga.mark...@gmail.com> wrote: > > > > Hi everyone, > > > > I wonder if you could provide me with some suggestions on how to determine > > the confidence and prediction intervals of SVR? If you have suggestions for > > any machine learning algorithms in general, that would be fine too (doesn't > > have to be specific for SVR). > > > > So far, I have found: > > 1. Bootstrap: > > http://stats.stackexchange.com/questions/183230/bootstrapping-confidence-interval-from-a-regression-prediction > > 2. > > http://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0048723&type=printable > > 3. ftp://ftp.esat.kuleuven.ac.be/sista/suykens/reports/10_156_v0.pdf > > > > But, I don't fully understand the details in #2 and #3 to the point that I > > can write a step by step code. If I use bootstrap method, I can get the > > confidence interval as follows? > > a. Draw bootstrap sample of size n > > b. Fit the SVR model (with hyperparameters chosen during model selection > > with grid search cv) to this bootstrap sample > > c. Use this model to predict the output variable y* from input variable X* > > d. Repeat step a-c for, for instance, 100 times > > e. Order the 100 values of y*, and determine, for instance, the 10th > > percentile and 90th percentile (if we are looking for 0.8 confidence > > interval) > > f. Repeat a-e for different values of X* to plot the prediction with > > confidence interval > > > > But, I don't know how to get the prediction interval from here. > > > > Thank you very much, > > Raga > > _______________________________________________ > > scikit-learn mailing list > > scikit-learn@python.org > > https://mail.python.org/mailman/listinfo/scikit-learn > > _______________________________________________ > scikit-learn mailing list > scikit-learn@python.org > https://mail.python.org/mailman/listinfo/scikit-learn > > _______________________________________________ > scikit-learn mailing list > scikit-learn@python.org > https://mail.python.org/mailman/listinfo/scikit-learn _______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
