> There have been previous attempts to incorporate training score, but
there's a general open question of how best to
> return Gird Search results: The current format cv_scores_ is not really
extensible, which seems to have stalled many of
> these issues. Input on this issue is welcome. Otherwise, for the moment,
you will have to roll your own implementation
> (and I should note that _fit_and_score is a fairly recent invention).
What about adding more fields to the _CVScoreTuple namedtuple
(GridSearchCV.grid_scores_ is a list of these namedtuples)? If things are
added at the end of the list, it should have a pretty small chance of
breaking backward compatibility. The current field names (`parameters`,
`mean_validation_score`, `cv_validation_scores`) are quite specific, so
for example adding `cv_train_scores` could be an option.
I'm not too aware of the history of the project or what has been tried
previously on this issue, so appologies if this is obviously incorrect.
FWIW, I put together the code + tests for this change:
https://github.com/rmcgibbo/scikit-learn/compare/scikit-learn:master...rmcgibbo:grid-search-train-error
Happy to file a PR if this is worthwhile for others.
-Robert
On Thu, May 1, 2014 at 10:10 PM, Joel Nothman <joel.noth...@gmail.com>wrote:
> There have been previous attempts to incorporate training score, but
> there's a general open question of how best to return Gird Search results:
> The current format cv_scores_ is not really extensible, which seems to have
> stalled many of these issues. Input on this issue is welcome. Otherwise,
> for the moment, you will have to roll your own implementation (and I should
> note that _fit_and_score is a fairly recent invention).
>
>
> On 2 May 2014 13:34, Robert McGibbon <rmcgi...@gmail.com> wrote:
>
>> Hi all,
>>
>> Is there any to get the score on the training data for each parameter set
>> (and each fold) when running GridSearchCV? While I haven't looked too
>> closely at the code, it appears that
>> BaseSearchCV<https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/grid_search.py#L378>uses
>> the
>> _fit_and_score<https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/cross_validation.py#L1118>
>> method,
>> which does have the ability to calculated and return scores on the training
>> data, but that this functionality isn't exposed in GridSearchCV.
>>
>> The use case for this would to compare training and test error (ala the
>> classic training error and test error vs. model complexity
>> plot<http://link.springer.com/protocol/10.1007%2F978-1-60327-429-6_15/fulltext.html#Fig3_15>
>> )
>>
>> -Robert
>>
>>
>> ------------------------------------------------------------------------------
>> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
>> Instantly run your Selenium tests across 300+ browser/OS combos. Get
>> unparalleled scalability from the best Selenium testing platform
>> available.
>> Simple to use. Nothing to install. Get started now for free."
>> http://p.sf.net/sfu/SauceLabs
>> _______________________________________________
>> Scikit-learn-general mailing list
>> Scikit-learn-general@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>
>>
>
>
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos. Get
> unparalleled scalability from the best Selenium testing platform available.
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos. Get
unparalleled scalability from the best Selenium testing platform available.
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
