Interesting to see this issue get some attention again.
I'm hoping to develop sklearn wrappers for hyperopt this summer. If anyone
wants to be part of that development process, by all means let me know.
I'll keep this list posted on progress and listen to feedback on usability.
The quick wrapper I posted a few months ago for the Perceptron, for
example, generated great quality feedback. More interactions like that
would be ideal.
Hyperopt is currently relatively limited in terms of algorithms, but I feel
like the more natural next step is to produce sklearn wrappers. Once those
are sketched in, then it makes sense to do sklearn-specific
hyperparameter-optimization algorithm comparison. There are lots of
more/less complicated approaches to try.
In the meantime, keep an eye on https://github.com/jaberg/hyperopt-sklearn,
hopefully good stuff will be showing up there in the coming months.
If you want to hurry the process along, let me me know. Developers and
money are always welcome :)
- James
On Mon, Apr 8, 2013 at 8:46 AM, Joel Nothman
<[email protected]>wrote:
> I've hacked together something like this on top of a refactored
> BaseSearchCV (building on my PR#1787's param_search_struct_arrays branch
> rather than master):
> https://github.com/jnothman/scikit-learn/tree/param_search_callback.
>
> Unlike the current parameter searches require the entire sequence of
> parameters to be pre-determined, this allows the generation of parameters
> in response to the evaluation of previous parameters. In the case of
> Scipy's solvers, this is by way of callback.
>
> Since Scipy generally expects linear, real-valued parameter spaces, the
> Scipy parameters often need to be transformed into sklearn parameter spaces
> that may not be linear. Hence parameters are specified using the
> param_transforms argument (assuming for now that initially all parameters
> are 0), which makes the interface a bit unfriendly:
>
> >>> from sklearn import svm, grid_search, datasets
> >>> iris = datasets.load_iris()
> >>> svr = svm.SVC()
> >>> clf = grid_search.ScipyMinimizeCV(svr, {'C': lambda x: 10 ** (x /
> 0.0001) + 1e-25})
> >>> clf = clf.fit(iris.data, iris.target)
> >>> clf.grid_results_
> array([({'C': 1.0}, 0.9666666666666667), ({'C': 316.22776601683796}, 0.94),
> ({'C': 0.0031622776601683794}, 0.32),
> ({'C': 17.782794100389228}, 0.96),
> ({'C': 0.056234132519034911}, 0.9066666666666666),
> ({'C': 4.2169650342858223}, 0.9533333333333334),
> ({'C': 4.2169650342858223}, 0.9533333333333334),
> ({'C': 0.23713737056616552}, 0.9333333333333333),
> ({'C': 2.0535250264571463}, 0.96),
> ({'C': 0.48696752516586311}, 0.94),
> ({'C': 1.4330125702369627}, 0.98),
> ({'C': 2.0535250264571463}, 0.96),
> ({'C': 1.1970850304957299}, 0.9666666666666667),
> ({'C': 1.1970850304957299}, 0.9666666666666667),
> ({'C': 1.7154378963428789}, 0.96),
> ({'C': 1.3097472643005899}, 0.9733333333333334),
> ({'C': 1.5678788438269704}, 0.9733333333333334),
> ({'C': 1.3699942677165546}, 0.98)],
> dtype=[('parameters', 'O'), ('test_score', '<f8')])
> >>> clf.best_index_, clf.best_params_, clf.best_score_
> (10, {'C': 1.4330125702369627}, 0.97999999999999998)
>
> In any case, these general enhancements to the parameter search framework
> mean more sophisticated searches may be implemented, not necessarily with
> this transformation overhead.
>
> I might clean this up and turn it into a PR if it seems worthwhile.
>
> - Joel
>
>
> On Mon, Apr 8, 2013 at 11:27 AM, Joel Nothman <
> [email protected]> wrote:
>
>> Currently BaseSearchCV expects a predetermined sequence of parameter
>> settings, which is unideal for some cases. SciPy opts for a callback
>> approach. I've not used that interface, but I gather something like this
>> might work:
>>
>> class MinimizeCV(BaseEstimator):
>> def __init__(self, minimiser, clf, param_init, scoring, cv,
>> minimise_kwargs={}):
>> self.clf = clf
>> self.param_init = param_init
>> self.scoring = scoring
>> self.cv = cv
>> self.minimiser = minimiser
>> self.minimise_kwargs = minimise_kwargs
>>
>> def fit(self, X, y=None):
>> def objective(param_values):
>> """"""
>> # TODO: parallelise fold fitting
>> return aggregate(
>> fit_grid_point(X, y, self.clf, dict(zip(param_list,
>> param_values)), self.scoring, ...)
>> for train, test in self.cv)
>> res = self.minimiser(objective, X0=[v for k, v in
>> sorted(self.param_init.iteritems())], **self.minimise_kwargs)
>> # TODO: store results and perhaps search history
>>
>> I think a variant could be implemented that shares most of its code with
>> the existing BaseSearchCV.
>>
>> I haven't looked at hyperopt's interface yet.
>>
>> - Joel
>>
>> > On Sun, Apr 7, 2013 at 6:35 PM, Roman Sinayev <[email protected]> wrote:
>>
>>> >
>>> > It seems like brute force Grid Search takes forever when attempting to
>>> > determine best parameters with many classifiers. Let's say the
>>> > parameter space looks something like this
>>> > http://i.imgur.com/AiBl8Wt.png . Why not use the SciPy simulated
>>> > annealing or some simple genetic algorithm instead of searching
>>> > through all the possible parameter space of every classifier?
>>>
>>
>
>
> ------------------------------------------------------------------------------
> Minimize network downtime and maximize team effectiveness.
> Reduce network management and security costs.Learn how to hire
> the most talented Cisco Certified professionals. Visit the
> Employer Resources Portal
> http://www.cisco.com/web/learning/employer_resources/index.html
> _______________________________________________
> Scikit-learn-general mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
