hi sebastian,
sorry, maybe I was a little bit unclear, what I meant was the scenario 2)
> in contrast to 1) below:
>
> 1) perform k-fold cross-validation on the complete dataset for model
> selection and then report the score as estimate of the model's performance
> (not a good idea!)
>
if you mean that you choose the best-subset performance as the model, then
that's not a good idea.
2) split the dataset, only do cross-validation on the training set (which
> is then further subdivided into training and test folds), select the model
> based on the results, and then use the separate test set that the model
> hasn't seen before to estimate its performance to generalize
>
this is ok.
but cross_val_score with a GridSearchCV as clf, will in fact do 2) but for
cv different types of train and test sets. and as michael points out what
you are effectively validating is a model that also selects it's parameters
(the parameters given to grid search).
the model in this latter case is really gridsearchcv, not the specific clf
used inside it.
cheers,
satra
------------------------------------------------------------------------------
> One dashboard for servers and applications across Physical-Virtual-Cloud
> Widest out-of-the-box monitoring support with 50+ applications
> Performance metrics, stats and reports that give you Actionable Insights
> Deep dive visibility with transaction tracing using APM Insight.
> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
