Submit a PR and see what people think!
On 31 January 2014 09:52, Faraz Mirzaei <fmmirz...@gmail.com> wrote:
> It seems that removing lines 1065-1067 of cross_validation.py solves the
> problem for now:
>
>
> 1065: if not isinstance(score, numbers.Number):
>
> 1066: raise ValueError("scoring must return a number, got %s (%s)"
>
> 1067 " instead." % (str(score), type(score)))
>
>
> Can we patch the next release to reflect this?
>
>
> Thanks,
>
>
> Faraz
>
>
> On Tue, Jan 28, 2014 at 2:11 PM, Joel Nothman <joel.noth...@gmail.com>wrote:
>
>> Hi Faraz,
>>
>> Not supporting multiple scores, in grid search even, is a known problem.
>> We haven't quite found a good solution.
>>
>> Maybe supporting arbitrary scores from `cross_val_score` is a simple
>> quick-fix for some of that...
>>
>>
>> On 29 January 2014 08:59, Faraz Mirzaei <fmmirz...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I had used scikit-learn's cross correlation in regression test
>>> and performance validation for a software release. Unfortunately,
>>> scikit-learn was inadvertently updated to 0.14.1, breaking a large number
>>> of our regression and performance validation tests.
>>>
>>> I had used cross_val_score to measure "per class" recalls. I've been
>>> passing the following recall function to cross_val_score:
>>>
>>> def getRecalls(y_true, y_pred):
>>>
>>> p, r, f, s = precision_recall_fscore_support(y_true, y_pred)
>>>
>>> return r
>>>
>>>
>>> Note that r is a vector of recalls, with one element corresponding to
>>> each class. In the latest release (0.14.1), cross_val_score is changed to
>>> ensure that only a scalar is returned out of the scoring function.
>>> Specifically, in lines 1065-1067 of cross_validation.py, the following
>>> exception is raised for non-scalar output of scoring function:
>>>
>>>
>>> if not isinstance(score, numbers.Number):
>>>
>>> raise ValueError("scoring must return a number, got %s (%s)"
>>>
>>> " instead." % (str(score), type(score)))
>>>
>>>
>>>
>>> While I understand that for parameter optimization and grid search, one
>>> needs to use a scalar score function, I don't think cross_val_score is the
>>> right place to force or check that. Obviously, there are many other use
>>> cases for cross_val_score other than parameter optimization (namely for
>>> performance validation and reg test), where non-scalar metric makes sense.
>>> In our specific application, per class recalls are important and an average
>>> recall (over all classes) is not sufficient.
>>>
>>>
>>> Thanks,
>>>
>>>
>>> Faraz
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> WatchGuard Dimension instantly turns raw network data into actionable
>>> security intelligence. It gives you real-time visual feedback on key
>>> security issues and trends. Skip the complicated setup - simply import
>>> a virtual appliance and go from zero to informed in seconds.
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Scikit-learn-general mailing list
>>> Scikit-learn-general@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> WatchGuard Dimension instantly turns raw network data into actionable
>> security intelligence. It gives you real-time visual feedback on key
>> security issues and trends. Skip the complicated setup - simply import
>> a virtual appliance and go from zero to informed in seconds.
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Scikit-learn-general mailing list
>> Scikit-learn-general@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>
>>
>
>
> ------------------------------------------------------------------------------
> WatchGuard Dimension instantly turns raw network data into actionable
> security intelligence. It gives you real-time visual feedback on key
> security issues and trends. Skip the complicated setup - simply import
> a virtual appliance and go from zero to informed in seconds.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
------------------------------------------------------------------------------
WatchGuard Dimension instantly turns raw network data into actionable
security intelligence. It gives you real-time visual feedback on key
security issues and trends. Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
