It seems that removing lines 1065-1067 of cross_validation.py solves the
problem for now:
1065: if not isinstance(score, numbers.Number):
1066: raise ValueError("scoring must return a number, got %s (%s)"
1067 " instead." % (str(score), type(score)))
Can we patch the next release to reflect this?
Thanks,
Faraz
On Tue, Jan 28, 2014 at 2:11 PM, Joel Nothman <joel.noth...@gmail.com>wrote:
> Hi Faraz,
>
> Not supporting multiple scores, in grid search even, is a known problem.
> We haven't quite found a good solution.
>
> Maybe supporting arbitrary scores from `cross_val_score` is a simple
> quick-fix for some of that...
>
>
> On 29 January 2014 08:59, Faraz Mirzaei <fmmirz...@gmail.com> wrote:
>
>> Hello,
>>
>> I had used scikit-learn's cross correlation in regression test
>> and performance validation for a software release. Unfortunately,
>> scikit-learn was inadvertently updated to 0.14.1, breaking a large number
>> of our regression and performance validation tests.
>>
>> I had used cross_val_score to measure "per class" recalls. I've been
>> passing the following recall function to cross_val_score:
>>
>> def getRecalls(y_true, y_pred):
>>
>> p, r, f, s = precision_recall_fscore_support(y_true, y_pred)
>>
>> return r
>>
>>
>> Note that r is a vector of recalls, with one element corresponding to
>> each class. In the latest release (0.14.1), cross_val_score is changed to
>> ensure that only a scalar is returned out of the scoring function.
>> Specifically, in lines 1065-1067 of cross_validation.py, the following
>> exception is raised for non-scalar output of scoring function:
>>
>>
>> if not isinstance(score, numbers.Number):
>>
>> raise ValueError("scoring must return a number, got %s (%s)"
>>
>> " instead." % (str(score), type(score)))
>>
>>
>>
>> While I understand that for parameter optimization and grid search, one
>> needs to use a scalar score function, I don't think cross_val_score is the
>> right place to force or check that. Obviously, there are many other use
>> cases for cross_val_score other than parameter optimization (namely for
>> performance validation and reg test), where non-scalar metric makes sense.
>> In our specific application, per class recalls are important and an average
>> recall (over all classes) is not sufficient.
>>
>>
>> Thanks,
>>
>>
>> Faraz
>>
>>
>>
>> ------------------------------------------------------------------------------
>> WatchGuard Dimension instantly turns raw network data into actionable
>> security intelligence. It gives you real-time visual feedback on key
>> security issues and trends. Skip the complicated setup - simply import
>> a virtual appliance and go from zero to informed in seconds.
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Scikit-learn-general mailing list
>> Scikit-learn-general@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>
>>
>
>
> ------------------------------------------------------------------------------
> WatchGuard Dimension instantly turns raw network data into actionable
> security intelligence. It gives you real-time visual feedback on key
> security issues and trends. Skip the complicated setup - simply import
> a virtual appliance and go from zero to informed in seconds.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
------------------------------------------------------------------------------
WatchGuard Dimension instantly turns raw network data into actionable
security intelligence. It gives you real-time visual feedback on key
security issues and trends. Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
