Hi Cristoph, Gael, hi everyone, > On 24 Mar 2015, at 18:09, Gael Varoquaux <gael.varoqu...@normalesup.org> > wrote: > >> Don't you think that I could also benchmark models that are not >> implemented in sklearn? […] > > I am personally less interested in that. We have already a lot in > scikit-learn and more than enough to test the model selection code.
On top of this, people have already been using dedicated hyperparameter optimizer toolkits for Theano deep nets. I don’t think we should aim to compete with hyperopt/spearmint from day 0 (or ever), but, just like Gael said, > The focus should be on providing code that is readily-usable. As for your proposal, I have a few comments. 1. in 3.1 you say “It will have same interfaces as GridSearchCV and RandomizedSearchCV”. Even the use of plural “interfaces” here points at a problem: those two object do not have identical interfaces. Which interface will GPSearchCV have? Will it take (prior) distributions over hyperparameters? (In the same format as RandomizedSearchCV?) Ranges and assume a fixed prior? I think a more detailed discussion of the user-facing API would be useful. 2. Ideally this module would fully reuse the GP module. We should have no code redundancy, but the way your proposal is written, it does not focus much on the interaction of your changes with the GP module. (For example, will slice sampling be a contribution to the GP module?) Change sets that reach deeper will take longer to review and merge. 3. Your point in 4.4 about optimizing improvement *per second* seems desirable, where does it fit in the timeline? Will everything be done with this in mind from the start? 4. Parallelization is interesting and seems non-trivial. I’m a bit dense but I managed to understand Gael’s seeding suggestion earlier. The paragraph in your proposal confused me though, especially the part “I will use all completed, and integrate over pending evaluations to approximate the expected acquisition function.” Could you clarify? 5. (Timeline stuff.) I’m not sure what the relationship between “Build optimizing pipeline from parts implemented so far” and “First working prototype” is. Testing features shouldn’t come so late, it should be done at the same time. In general, the timeline would benefit from a slight shift of perspective: when would you like to have the PR on X functionally complete (this includes tests)? Overall complete (includes docs, examples and at least some review)? Hope my comments are helpful, Yours, Vlad > > I am worried that such task will be very time consuming and will not move > us much closer to code that improves model selection in scikit-learn. > > Gaël > > > ------------------------------------------------------------------------------ > Dive into the World of Parallel Programming The Go Parallel Website, sponsored > by Intel and developed in partnership with Slashdot Media, is your hub for all > things parallel software development, from weekly thought leadership blogs to > news, videos, case studies, tutorials and more. Take a look and join the > conversation now. http://goparallel.sourceforge.net/ > _______________________________________________ > Scikit-learn-general mailing list > Scikit-learn-general@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general ------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ Scikit-learn-general mailing list Scikit-learn-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
