I think you could bench on other problems, but maybe focus on the ones in scikit-learn. Deep learning people might be happy with using external tools for optimizing. I'd also recommend benchmarking just the global optimization part on global optimization datasets as they were used in Jasper's work.
On 03/24/2015 06:01 PM, Christof Angermueller wrote: > Don't you think that I could also benchmark models that are not > implemented in sklearn? For instance, I could write a wrapper > DeepNet(...) with fit() and predict(), and which uses internally theano > to build a ANN? In this way, I could benchmark complex deep networks > beyond what will be possible with the new sklearn ANN module. This might > be interesting for the deep learning community. > > Obvious sklearn modules to benchmark are: > * RandomForestClassifier > * SVC > * GaussianProcess > * Perceptron > > As benchmark data sets, I would use those that were used before (see > Snoek at al 2012, Bergstra et at 2011) to evaluate optimizer like > spearmint. For classification, I candidates are > * MNIST > * CIFAR-10 > > and for regression: > * Bosting housing precises > > @Andy, @Kyle, and @Matthias: thanks for your references! I will have a > closer look at them tomorrow! > > Christof > > On 20150324 21:25, Andy wrote: >> One thing that might also be interesting is "Bootstrapping" (in the >> compiler sense, not the statistics sense) the optimizer. >> The latest Jasper Snoek paper http://arxiv.org/abs/1502.05700 they used >> a hyper-parameter optimizer to optimize the parameter >> of a hyper-parameter optimizer on a set of optimization tasks. >> >> https://www.youtube.com/watch?v=BIizqZ0mvIo >> >> So we could try to optimize the parameters of the GP using the GP :) >> >> ------------------------------------------------------------------------------ >> Dive into the World of Parallel Programming The Go Parallel Website, >> sponsored >> by Intel and developed in partnership with Slashdot Media, is your hub for >> all >> things parallel software development, from weekly thought leadership blogs to >> news, videos, case studies, tutorials and more. Take a look and join the >> conversation now. http://goparallel.sourceforge.net/ >> _______________________________________________ >> Scikit-learn-general mailing list >> Scikit-learn-general@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general ------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ Scikit-learn-general mailing list Scikit-learn-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
