Hi, I stumbled upon the brief note about nested cross-validation in the online documentation at http://scikit-learn.org/stable/tutorial/statistical_inference/model_selection.html#grid-search ===================== Nested cross-validation >>> >>> cross_validation.cross_val_score(clf, X_digits, y_digits) ...
array([ 0.938..., 0.963..., 0.944...]) Two cross-validation loops are performed in parallel: one by the GridSearchCV estimator to set gamma and the other one bycross_val_score to measure the prediction performance of the estimator. The resulting scores are unbiased estimates of the prediction score on new data. ===================== I am wondering how to "use" or "interpret" those scores. For example, if the gamma parameters are set differently in the inner loops, we accumulate test scores from the outer loops that would correspond to different models, and calculating the average performance from those scores wouldn't be a good idea? So, if the estimated parameters are different for the different inner folds, I would say that my model is not "stable" and varies a lot with respect to the chosen training fold. In general, what would speak against an approach to just split the initial dataset into train/test (70/30), perform grid search (via k-fold CV) on the training set, and evaluate the model performance on the test dataset? Best, Sebastian ------------------------------------------------------------------------------ One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y _______________________________________________ Scikit-learn-general mailing list Scikit-learn-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
