Hi Faraz,
Not supporting multiple scores, in grid search even, is a known problem. We
haven't quite found a good solution.
Maybe supporting arbitrary scores from `cross_val_score` is a simple
quick-fix for some of that...
On 29 January 2014 08:59, Faraz Mirzaei <fmmirz...@gmail.com> wrote:
> Hello,
>
> I had used scikit-learn's cross correlation in regression test
> and performance validation for a software release. Unfortunately,
> scikit-learn was inadvertently updated to 0.14.1, breaking a large number
> of our regression and performance validation tests.
>
> I had used cross_val_score to measure "per class" recalls. I've been
> passing the following recall function to cross_val_score:
>
> def getRecalls(y_true, y_pred):
>
> p, r, f, s = precision_recall_fscore_support(y_true, y_pred)
>
> return r
>
>
> Note that r is a vector of recalls, with one element corresponding to each
> class. In the latest release (0.14.1), cross_val_score is changed to ensure
> that only a scalar is returned out of the scoring function. Specifically,
> in lines 1065-1067 of cross_validation.py, the following exception is
> raised for non-scalar output of scoring function:
>
>
> if not isinstance(score, numbers.Number):
>
> raise ValueError("scoring must return a number, got %s (%s)"
>
> " instead." % (str(score), type(score)))
>
>
>
> While I understand that for parameter optimization and grid search, one
> needs to use a scalar score function, I don't think cross_val_score is the
> right place to force or check that. Obviously, there are many other use
> cases for cross_val_score other than parameter optimization (namely for
> performance validation and reg test), where non-scalar metric makes sense.
> In our specific application, per class recalls are important and an average
> recall (over all classes) is not sufficient.
>
>
> Thanks,
>
>
> Faraz
>
>
>
> ------------------------------------------------------------------------------
> WatchGuard Dimension instantly turns raw network data into actionable
> security intelligence. It gives you real-time visual feedback on key
> security issues and trends. Skip the complicated setup - simply import
> a virtual appliance and go from zero to informed in seconds.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
------------------------------------------------------------------------------
WatchGuard Dimension instantly turns raw network data into actionable
security intelligence. It gives you real-time visual feedback on key
security issues and trends. Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
