On Wed, Apr 1, 2015 at 4:05 AM, Vlad Niculae <zephy...@gmail.com> wrote:
> Hi Gael,
>
> > On 31 Mar 2015, at 14:01, Gael Varoquaux <gael.varoqu...@normalesup.org>
> wrote:
> >
> >> Why do you think the GP route is easier?
> >
> > Because we already have GPs.
>
We have a GP implementation but it's being rewritten...
> Well, we already have random forests too.
>
SMAC needs the variance of predictions so we'll need to get this PR merged
https://github.com/scikit-learn/scikit-learn/pull/3645
I would really like #3645 to get merged in any case.
>
> Both cases would need quite a bit of machinery on top, and I don’t know
> the extent of it, but I thought it would be interesting to have a chance to
> explore both alleys, since Cristof’s proposal seems to cover Spearmint’s
> approach thoroughly. By reading [1], it also seemed like SMAC can be a
> good fit regarding discrete parameters and cross-validation support (though
> supporting this might just be an implementation detail vs. a limitation of
> the algorithm, I am not sure.)
>
In order to support discrete parameters, our tree implementation would need
to support categorical variables though.
I personally like the simplicity of SMAC. It could potentially be better
than GPs for highly non-smooth hyper-parameter surfaces.
Mathieu
> [1] Towards an empirical foundation for assessing bayesian optimization of
> hyperparameters https://www.cs.ubc.ca/~hoos/Publ/EggEtAl13.pdf
>
>
> > Also, I am worried that the description in
> > the SMAC paper is quite vague, and that we will have to dig in referenced
> > papers and code to understand what is really being done. Maybe that
> > simply reflects my lack of understanding of the field :).
> >
> > Gaël
> >
> >> On 03/28/2015 01:29 PM, Gael Varoquaux wrote:
> >>> Sorry for the slow reply,
> >
> >>> On Fri, Mar 27, 2015 at 11:49:46AM -0400, hamzeh alsalhi wrote:
> >>>> I have revised my proposal to focus only on SMAC and to prioritize
> SMAC RF
> >>>> because it can be worked on independently GP.
> >>> I actually believed that GP were an easier route forward.
> >
> >>> The way I would have tackled such a project would have been:
> >
> >>> 1. Improve GPs
> >>> 2. In parallel, built a Bayesian optimization example (example not
> >>> module)
> >>> 3. Use the work above to make a hyper-parameter selection object.
> >
> >>> In addition to giving us 3, 1 and 2 would be good to have.
> >
> >>> Cheers,
> >
> >>> Gaël
> >
> >>>
> ------------------------------------------------------------------------------
> >>> Dive into the World of Parallel Programming The Go Parallel Website,
> sponsored
> >>> by Intel and developed in partnership with Slashdot Media, is your hub
> for all
> >>> things parallel software development, from weekly thought leadership
> blogs to
> >>> news, videos, case studies, tutorials and more. Take a look and join
> the
> >>> conversation now. http://goparallel.sourceforge.net/
> >>> _______________________________________________
> >>> Scikit-learn-general mailing list
> >>> Scikit-learn-general@lists.sourceforge.net
> >>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
> >
> >
> >>
> ------------------------------------------------------------------------------
> >> Dive into the World of Parallel Programming The Go Parallel Website,
> sponsored
> >> by Intel and developed in partnership with Slashdot Media, is your hub
> for all
> >> things parallel software development, from weekly thought leadership
> blogs to
> >> news, videos, case studies, tutorials and more. Take a look and join the
> >> conversation now. http://goparallel.sourceforge.net/
> >> _______________________________________________
> >> Scikit-learn-general mailing list
> >> Scikit-learn-general@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
> >
> > --
> > Gael Varoquaux
> > Researcher, INRIA Parietal
> > Laboratoire de Neuro-Imagerie Assistee par Ordinateur
> > NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France
> > Phone: ++ 33-1-69-08-79-68
> > http://gael-varoquaux.info
> http://twitter.com/GaelVaroquaux
> >
> >
> ------------------------------------------------------------------------------
> > Dive into the World of Parallel Programming The Go Parallel Website,
> sponsored
> > by Intel and developed in partnership with Slashdot Media, is your hub
> for all
> > things parallel software development, from weekly thought leadership
> blogs to
> > news, videos, case studies, tutorials and more. Take a look and join the
> > conversation now. http://goparallel.sourceforge.net/
> > _______________________________________________
> > Scikit-learn-general mailing list
> > Scikit-learn-general@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website,
> sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for
> all
> things parallel software development, from weekly thought leadership blogs
> to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
