Hi all,
I've had enough frustration at having to patch in things from a code fork
in order to merely get back precision and recall while optimising F1 in
grid search. This is something I need to do really frequently, as I'm sure
do others.
When I wrote and submitted PRs about this problem nine months ago, I
proposed relatively sophisticated solutions. Perhaps a simple, flexible
solution is appropriate:
GridSearchCV, RandomizedSearchCV, cross_val_score, and perhaps anything
else supporting 'scoring', should take an additional parameter, e.g.
'diagnostics', which is a callable with interface:
(estimator, X_train, y_train, X_test, y_test) -> object
The results of CV will include a params x folds array (or list of arrays)
to store each of these returned objects, whose dtype is automatically
detected, so that it may be compactly stored and easily accessed if desired.
So when scoring=f1, a diagnostic fn can be passed to calculate precision,
recall, etc., which means a bit of duplicated scoring work, but no
confusion of the existing scoring interface.
Scikit-learn may indeed provide ready-made diagnostic functions for certain
types of tasks. For example:
- a binary classification diagnostic might return P, R, F, AUC, AvgPrec;
- multiclass might add per-class performances, different averages and a
confusion matrix;
- a linear model diagnostic might measure model sparsity. (Perhaps the
parameter can take a sequence of callables to return a tuple of diagnostic
results per fold.)
As opposed to some of my more intricate proposals, this approach leaves it
to the user to do any averaging over folds etc.
*SearchCV should also store best_index_ more importantly than best_params_
so that this data can be cross-referenced. If the diagnostic output is a
list of arrays, rather than an array, the user can manually delete
information from the non-best trials, before saving the model to disk.
This also implies some refactoring of cross_val_score and fit_grid_point
that is overdue.
Does this seem the right level of complexity/flexibility? Please help me
and the many others who have requested it resolve this issue sooner rather
than later. I'd like to submit a PR towards this that actually gets
accepted, so some feedback is really welcome.
Cheers,
- Joel
------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today.
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
