Re: [Scikit-learn-general] Bayesian optimization for hyperparameter tuning

On Sun, Feb 2, 2014 at 5:26 PM, Joel Nothman wrote: > Nice. I've taken a look at what you've got there... > > So for example, to draw a randomised SVC instance: > > >>> from hpsklearn.components import svc > >>> from hyperopt.pyll.stochastic import sample > >>> sample(svc(str)) > SVC(C=0.47173606

Re: [Scikit-learn-general] Bayesian optimization for hyperparameter tuning

Nice. I've taken a look at what you've got there... So for example, to draw a randomised SVC instance: >>> from hpsklearn.components import svc >>> from hyperopt.pyll.stochastic import sample >>> sample(svc(str)) SVC(C=0.471736065582, cache_size=100.0, class_weight=None, coef0=-0.0579424882785,

Re: [Scikit-learn-general] Bayesian optimization for hyperparameter tuning

(Sorry about the comment about a revived thread, I was thinking of another one!) On Sun, Feb 2, 2014 at 10:43 AM, James Bergstra wrote: > Glad to see this thread revived! > > Sklearn-users who are interested in this stuff should check out Hyperopt's > sklearn interface: > > https://github.com/hy

Re: [Scikit-learn-general] Bayesian optimization for hyperparameter tuning

Glad to see this thread revived! Sklearn-users who are interested in this stuff should check out Hyperopt's sklearn interface: https://github.com/hyperopt/hyperopt-sklearn It's very much a work-in-progress. We're in the process of putting together some examples / tutorial, and a tech report that

Re: [Scikit-learn-general] Bayesian optimization for hyperparameter tuning

thanks. Fred On Thu, Jan 30, 2014 at 8:28 PM, Patrick Mineault wrote: > Sure you can: > > http://www.cs.toronto.edu/~jasper/bayesopt.pdf > > And some python code: > > https://github.com/JasperSnoek/spearmint > > > On Thu, Jan 30, 2014 at 7:53 PM, Frédéric Bastien wrote: >> >> I have a question

Re: [Scikit-learn-general] Bayesian optimization for hyperparameter tuning

On Thu, Jan 30, 2014 at 07:53:16PM -0500, Frédéric Bastien wrote: > I have a question on those type of algo for hyper parameter > optimization. With a grid search, we can run all jobs in parallel. But > I have the impression that those algo remove that possibility. Is > there there way to sample ma

Re: [Scikit-learn-general] Bayesian optimization for hyperparameter tuning

On Thu, Jan 30, 2014 at 5:21 PM, Sturla Molden wrote: > As I understand it fro reading about this a LONG time ago (apologies if my > memory is rusty), "Bayesian optimization" means maximizing the > log-likelihood using the Newton-Raphson method. Probably that was how the term was typically used

Re: [Scikit-learn-general] Bayesian optimization for hyperparameter tuning

> With a grid search, we can run all jobs in parallel. But I have the impression that those algo remove that possibility. ... You can still run all folds in, say 10-fold cross-validation in parallel. > But the most interresting question, if we start many jobs in parallel, if the jobs don't finis

Re: [Scikit-learn-general] Bayesian optimization for hyperparameter tuning

Sure you can: http://www.cs.toronto.edu/~jasper/*bayes*opt.pdf And some python code: https://github.com/JasperSnoek/spearmint On Thu, Jan 30, 2014 at 7:53 PM, Frédéric Bastien wrote: > I have a question on those type of algo for hyper parameter > optimization. With a grid search, we can run

Re: [Scikit-learn-general] Bayesian optimization for hyperparameter tuning

I have a question on those type of algo for hyper parameter optimization. With a grid search, we can run all jobs in parallel. But I have the impression that those algo remove that possibility. Is there there way to sample many starting configuration with those algo? But the most interresting quest

Re: [Scikit-learn-general] Bayesian optimization for hyperparameter tuning

On Thu, Jan 30, 2014 at 11:23:28AM -0800, James Jensen wrote: > Bayesian optimization is an efficient method used especially for > functions that are expensive to evaluate. The basic idea is to fit the > function using Gaussian processes, using a surrogate function that > determines where to eva

Re: [Scikit-learn-general] Bayesian optimization for hyperparameter tuning

As I understand it fro reading about this a LONG time ago (apologies if my memory is rusty), "Bayesian optimization" means maximizing the log-likelihood using the Newton-Raphson method. The word "Bayesian" comes from an obfuscated explanation of what really happens: If we assume a flat or Gaussian

Re: [Scikit-learn-general] Bayesian optimization for hyperparameter tuning

Hi, Had, It's true that I'd have limited time (working on a PhD). I imagine most possible contributors are also quite busy. Mainly, I lack the expertise necessary to do this properly; I understand Bayesian optimization at a high level but don't have much of a foundation in the underlying math,

Re: [Scikit-learn-general] Bayesian optimization for hyperparameter tuning

It seems that with GridSearchCV and RandomizedSearchCV both already included in scikit-learn, it would make sense to also include other common, more efficient hyperparameter searchers as well. zach On Thu, Jan 30, 2014 at 3:11 PM, Hadayat Seddiqi wrote: > Hi, > > So I was the one who volunteer

Re: [Scikit-learn-general] Bayesian optimization for hyperparameter tuning

Hi, So I was the one who volunteered to do contribute my GP code for a revamp of scikits module. I'm far from an expert, and I can't say I understand how this would fit off the top of my head, but if someone is knowledgeable and willing to work on this then I'd be more than happy to lend a hand as

Re: [Scikit-learn-general] Bayesian optimization for hyperparameter tuning

Actually, I wanted to create exactly this myself. I was then discouraged by the fact that Scikit-learn did not pull from a guy who implemented Multi-Armed Banditon the reason that Scikit-learn doesn't do reinforcement learning. I'm new here (ev

[Scikit-learn-general] Bayesian optimization for hyperparameter tuning

I usually hesitate to suggest a new feature in a library like this unless I am in a position to work on it myself. However, given the number of people who seem eager to find something to contribute, and given the recent discussion about improving the Gaussian process module, I thought I'd ventu