Hello,
Recently, I had to learn a classifier with 3 hyperparameters (each taking
several possible values) and an exhaustive grid search was really
expensive. However, in some situations, some hyperparameters are more
important than others. For example, for a SVC with RBF kernel, my
experience is that choosing gamma (fixing C to 1.0) first, then C (fixing
gamma to the previous selected value) works pretty well. Would such a
non-exhaustive grid search be useful in scikit-learn?
In terms of API, I would add a "search_order" option to GridSearchCV. For
example, if param_grid is as follows:
param_grid = { "gamma": [0.01, 0.1, 1.0, 10.0], "C": [1, 0.1, 10, 100,
1000] }
then, search_order could be:
search_order = ["gamma", "C"].
The algorithm would first choose gamma given C=1.0 (as it is the first in
the list), then choose C given the best gamma. More generally.
len(search_order) doesn't need to be equal to len(param_grid) if only some
parameters must be fixed.
If I'm not mistaken, this reduces the number of grid points from \prod_i
n_values_i to \sum_i n_values_i.
I'm pretty sure this search procedure breaks in situations where
hyperparameters are very inter-dependent but I think it could still be
useful in some situations. Anyway, we really need more alternatives to
exhaustive grid search (c.f., James Bergstra's work).
Mathieu
------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
