Don't you think that I could also benchmark models that are not implemented in sklearn? For instance, I could write a wrapper DeepNet(...) with fit() and predict(), and which uses internally theano to build a ANN? In this way, I could benchmark complex deep networks beyond what will be possible with the new sklearn ANN module. This might be interesting for the deep learning community.
Obvious sklearn modules to benchmark are: * RandomForestClassifier * SVC * GaussianProcess * Perceptron As benchmark data sets, I would use those that were used before (see Snoek at al 2012, Bergstra et at 2011) to evaluate optimizer like spearmint. For classification, I candidates are * MNIST * CIFAR-10 and for regression: * Bosting housing precises @Andy, @Kyle, and @Matthias: thanks for your references! I will have a closer look at them tomorrow! Christof On 20150324 21:25, Andy wrote: > One thing that might also be interesting is "Bootstrapping" (in the > compiler sense, not the statistics sense) the optimizer. > The latest Jasper Snoek paper http://arxiv.org/abs/1502.05700 they used > a hyper-parameter optimizer to optimize the parameter > of a hyper-parameter optimizer on a set of optimization tasks. > > https://www.youtube.com/watch?v=BIizqZ0mvIo > > So we could try to optimize the parameters of the GP using the GP :) > > ------------------------------------------------------------------------------ > Dive into the World of Parallel Programming The Go Parallel Website, sponsored > by Intel and developed in partnership with Slashdot Media, is your hub for all > things parallel software development, from weekly thought leadership blogs to > news, videos, case studies, tutorials and more. Take a look and join the > conversation now. http://goparallel.sourceforge.net/ > _______________________________________________ > Scikit-learn-general mailing list > Scikit-learn-general@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general -- Christof Angermueller cangermuel...@gmail.com http://cangermueller.com ------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ Scikit-learn-general mailing list Scikit-learn-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
