Hi Andy,
On 24.03.2015 21:00, Andy wrote:
Hi Matthias.
I think that is an interesting direction to go into, and I actually
thought a bit about if and how we could add something like that to
scikit-learn.
Is there online documentation for paramsklearn?
I just compiled the current state of the documentation and put it on my
website here:
http://aad.informatik.uni-freiburg.de/~feurerm/paramsklearn/index.html.
Currently, the documentation only shows how to do random search, but I
hope that I can add an example for pySMAC
<https://github.com/sfalkner/pysmac/> soon.
It is a bit hard to say what are good defaults, I think, and it often
encodes intuition about the problem.
Hm, yes indeed. You could choose between three approaches here:
* Use defaults which work well across a lot of problem domains.
* Use defaults which are calculated based on dataset properties (like in
some scikit-learn models).
* Use techniques like Meta-Learning or Algorithm Selection to do this
job for you.
Because I am working on the last of the three approaches, the defaults
in the SVC example are more or less the scikit-learn defaults.
Furthermore, the default is an optional argument. Thus, a hyperparameter
optimization algorithm is not obliged to make use of it.
The parameter spaces that you want to search are probably different
between GridSearchCV and a model-based approach, too.
Probably. After some more thinking about this, you have to design a grid
depending on the computing time available, right? This would make it
hard to provide a configuration space for GridSearchCV at all.
Do you have any examples or benchmarks available online?
There is nothing besides the random search example. The only benchmark I
can provide at this moment is that the ParamSklearn approach (together
with an ensemble post-processing technique) placed third in the manual
track and first in the auto track of the Chalearn AutoML competition
<https://sites.google.com/a/chalearn.org/automl/>. If you have a
specific dataset/benchmark in mind, I can configure ParamSklearn with
SMAC and tell you about the results.
Best regards,
Matthias
Cheers,
Andy
On 03/24/2015 03:50 PM, Matthias Feurer wrote:
Dear scikit-learn team,
After reading the proposal of Christoph Angermüller wanting to
enhance scikit-learn with Bayesian optimization
(http://sourceforge.net/p/scikit-learn/mailman/message/33630274/) as
a GSoC project, you might also want to think again about the
integration of a hyperparameter concept into scikit-learn.
Our group built a framework called ParamSklearn
(https://bitbucket.org/mfeurer/paramsklearn/overview), which provides
hyperparameter definitions for a subset of classifiers, regressors
and preprocessors in scikit-learn. The result is something similar
like what James Bergstra did in hpsklearn
(https://github.com/hyperopt/hyperopt-sklearn) and a post from 2010
(http://sourceforge.net/p/scikit-learn/mailman/scikit-learn-general/thread/aanlktilvznvavqr-sbiixcguwyuf6jyq_ijvytdx7...@mail.gmail.com/?page=0).
In the end you get a configuration space which can then be read by a
Sequential Model-based Optimization package. For example, we used
this module for our AutoSklearn entry in the first automated machine
learning competition: https://sites.google.com/a/chalearn.org/automl/
Optimizing hyperparameters is a challenge itself, but defining
relevant ranges is also a difficult task for non-experts. Thus, it
would be nice to find a way to integrate the hyperparameter
definitions into scikit-learn (see bottom of this e-mail for a
suggestion) such that they can be used either by the not-yet-existing
GPSearchCV, the already existing RandomizedSearchCV or the
GridSearchCV, but also by external tools like our ParamSklearn. The
hyperparameter definitions would leave a user with only two mandatory
choices: number of evaluations/runtime and the estimator to use.
What do you think?
Best regards,
Matthias Feurer
Currently, we define the hyperparameters with a package called
HPOlibConfigSpace (https://github.com/automl/HPOlibConfigSpace). For
the SVC it looks like this:
C = UniformFloatHyperparameter("C", 0.03125, 32768, log=True,
default=1.0)
kernel = CategoricalHyperparameter(name="kernel",
choices=["rbf", "poly", "sigmoid"], default="rbf")
degree = UniformIntegerHyperparameter("degree", 1, 5, default=3)
gamma = UniformFloatHyperparameter("gamma", 3.0517578125e-05, 8,
log=True, default=0.1)
coef0 = UniformFloatHyperparameter("coef0", -1, 1, default=0)
shrinking = CategoricalHyperparameter("shrinking", ["True", "False"],
default="True")
tol = UniformFloatHyperparameter("tol", 1e-5, 1e-1, default=1e-4,
log=True)
class_weight = CategoricalHyperparameter("class_weight",
["None", "auto"],default="None")
max_iter = UnParametrizedHyperparameter("max_iter", -1)
cs = ConfigurationSpace()
cs.add_hyperparameter(C)
cs.add_hyperparameter(kernel)
cs.add_hyperparameter(degree)
cs.add_hyperparameter(gamma)
cs.add_hyperparameter(coef0)
cs.add_hyperparameter(shrinking)
cs.add_hyperparameter(tol)
cs.add_hyperparameter(class_weight)
cs.add_hyperparameter(max_iter)
degree_depends_on_poly = EqualsCondition(degree, kernel, "poly")
coef0_condition = InCondition(coef0, kernel, ["poly", "sigmoid"])
cs.add_condition(degree_depends_on_poly)
cs.add_condition(coef0_condition)
The code is more verbose than it has to be, but we are working on
this. The ConfigurationSpace object can then be accessed by a
@staticmethod and be used as a parameter description object inside
*SearchCV. We can provide a stripped-down version of the
HPOlibConfigSpace for integration in sklearn.external, as well as the
hyperparameter definitions we have so far.
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now.http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
