hi sebastian,
I am wondering how to "use" or "interpret" those scores. For example, if
> the gamma parameters are set differently in the inner loops, we accumulate
> test scores from the outer loops that would correspond to different models,
> and calculating the average performance from those scores wouldn't be a
> good idea? So, if the estimated parameters are different for the different
> inner folds, I would say that my model is not "stable" and varies a lot
> with respect to the chosen training fold.
>
i think this speaks to the nature of the data more than the nature of
cross-validation. for cross-validation (or validation) the general
assumption is that samples are similarly distributed, such that models
built on a subset can generalize or extrapolate to out of sample data.
we could take an extreme artificial situation, where i have training data
from a group of individuals measured with one instrument, and my test data
are from a similar but different instrument that has a consistent bias. no
amount of model building on the training is going to prepare it for bias in
the second instrument.
thus, if the histogram of your model parameters from nested
cross-validation are quite different, i believe the key issue is that the
data being fed as training and test are quite different from each other.
with larger samples, this tends to even itself out.
In general, what would speak against an approach to just split the initial
> dataset into train/test (70/30), perform grid search (via k-fold CV) on the
> training set, and evaluate the model performance on the test dataset?
>
isn't this what the cross-val score really does? just keeps repeating for
several different outer splits? the reason outer splits are important is
again to account for distributional characteristics in smallish-samples.
cheers,
satra
------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
