I've hacked together something like this on top of a refactored
BaseSearchCV (building on my PR#1787's param_search_struct_arrays branch
rather than master):
https://github.com/jnothman/scikit-learn/tree/param_search_callback.
Unlike the current parameter searches require the entire sequence of
parameters to be pre-determined, this allows the generation of parameters
in response to the evaluation of previous parameters. In the case of
Scipy's solvers, this is by way of callback.
Since Scipy generally expects linear, real-valued parameter spaces, the
Scipy parameters often need to be transformed into sklearn parameter spaces
that may not be linear. Hence parameters are specified using the
param_transforms argument (assuming for now that initially all parameters
are 0), which makes the interface a bit unfriendly:
>>> from sklearn import svm, grid_search, datasets
>>> iris = datasets.load_iris()
>>> svr = svm.SVC()
>>> clf = grid_search.ScipyMinimizeCV(svr, {'C': lambda x: 10 ** (x /
0.0001) + 1e-25})
>>> clf = clf.fit(iris.data, iris.target)
>>> clf.grid_results_
array([({'C': 1.0}, 0.9666666666666667), ({'C': 316.22776601683796}, 0.94),
({'C': 0.0031622776601683794}, 0.32),
({'C': 17.782794100389228}, 0.96),
({'C': 0.056234132519034911}, 0.9066666666666666),
({'C': 4.2169650342858223}, 0.9533333333333334),
({'C': 4.2169650342858223}, 0.9533333333333334),
({'C': 0.23713737056616552}, 0.9333333333333333),
({'C': 2.0535250264571463}, 0.96),
({'C': 0.48696752516586311}, 0.94),
({'C': 1.4330125702369627}, 0.98),
({'C': 2.0535250264571463}, 0.96),
({'C': 1.1970850304957299}, 0.9666666666666667),
({'C': 1.1970850304957299}, 0.9666666666666667),
({'C': 1.7154378963428789}, 0.96),
({'C': 1.3097472643005899}, 0.9733333333333334),
({'C': 1.5678788438269704}, 0.9733333333333334),
({'C': 1.3699942677165546}, 0.98)],
dtype=[('parameters', 'O'), ('test_score', '<f8')])
>>> clf.best_index_, clf.best_params_, clf.best_score_
(10, {'C': 1.4330125702369627}, 0.97999999999999998)
In any case, these general enhancements to the parameter search framework
mean more sophisticated searches may be implemented, not necessarily with
this transformation overhead.
I might clean this up and turn it into a PR if it seems worthwhile.
- Joel
On Mon, Apr 8, 2013 at 11:27 AM, Joel Nothman
<[email protected]>wrote:
> Currently BaseSearchCV expects a predetermined sequence of parameter
> settings, which is unideal for some cases. SciPy opts for a callback
> approach. I've not used that interface, but I gather something like this
> might work:
>
> class MinimizeCV(BaseEstimator):
> def __init__(self, minimiser, clf, param_init, scoring, cv,
> minimise_kwargs={}):
> self.clf = clf
> self.param_init = param_init
> self.scoring = scoring
> self.cv = cv
> self.minimiser = minimiser
> self.minimise_kwargs = minimise_kwargs
>
> def fit(self, X, y=None):
> def objective(param_values):
> """"""
> # TODO: parallelise fold fitting
> return aggregate(
> fit_grid_point(X, y, self.clf, dict(zip(param_list,
> param_values)), self.scoring, ...)
> for train, test in self.cv)
> res = self.minimiser(objective, X0=[v for k, v in
> sorted(self.param_init.iteritems())], **self.minimise_kwargs)
> # TODO: store results and perhaps search history
>
> I think a variant could be implemented that shares most of its code with
> the existing BaseSearchCV.
>
> I haven't looked at hyperopt's interface yet.
>
> - Joel
>
> > On Sun, Apr 7, 2013 at 6:35 PM, Roman Sinayev <[email protected]> wrote:
>
>> >
>> > It seems like brute force Grid Search takes forever when attempting to
>> > determine best parameters with many classifiers. Let's say the
>> > parameter space looks something like this
>> > http://i.imgur.com/AiBl8Wt.png . Why not use the SciPy simulated
>> > annealing or some simple genetic algorithm instead of searching
>> > through all the possible parameter space of every classifier?
>>
>
------------------------------------------------------------------------------
Minimize network downtime and maximize team effectiveness.
Reduce network management and security costs.Learn how to hire
the most talented Cisco Certified professionals. Visit the
Employer Resources Portal
http://www.cisco.com/web/learning/employer_resources/index.html
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
