Hello,
I had used scikit-learn's cross correlation in regression test
and performance validation for a software release. Unfortunately,
scikit-learn was inadvertently updated to 0.14.1, breaking a large number
of our regression and performance validation tests.
I had used cross_val_score to measure "per class" recalls. I've been
passing the following recall function to cross_val_score:
def getRecalls(y_true, y_pred):
p, r, f, s = precision_recall_fscore_support(y_true, y_pred)
return r
Note that r is a vector of recalls, with one element corresponding to each
class. In the latest release (0.14.1), cross_val_score is changed to ensure
that only a scalar is returned out of the scoring function. Specifically,
in lines 1065-1067 of cross_validation.py, the following exception is
raised for non-scalar output of scoring function:
if not isinstance(score, numbers.Number):
raise ValueError("scoring must return a number, got %s (%s)"
" instead." % (str(score), type(score)))
While I understand that for parameter optimization and grid search, one
needs to use a scalar score function, I don't think cross_val_score is the
right place to force or check that. Obviously, there are many other use
cases for cross_val_score other than parameter optimization (namely for
performance validation and reg test), where non-scalar metric makes sense.
In our specific application, per class recalls are important and an average
recall (over all classes) is not sufficient.
Thanks,
Faraz
------------------------------------------------------------------------------
WatchGuard Dimension instantly turns raw network data into actionable
security intelligence. It gives you real-time visual feedback on key
security issues and trends. Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
