See figure 5 of this paper: http://www.cs.ubc.ca/~hutter/papers/ICML14-HyperparameterAssessment.pdf for an example.
There is a better paper that exclusively tackles this but I cannot find it at the moment. I was referring to the optimizer preferring algorithms which are both fast and give good performance - EI per S tackles this, and was what I was referring to in my earlier email, though Hutter et. al. may have had an alternate metric. On Wed, Mar 25, 2015 at 4:22 PM, Christof Angermueller <c.angermuel...@gmail.com> wrote: > To which SMAC paper are you referring to? > What do you mean about optimizing runtime/training time? The optimizer > should find good parameters with in a short time. Do you mean comparing > the best result in a predefined time frame? For this, the 'expected > improvement per second' acquisition function, which is mentioned in my > proposal, might achieve good results. > > Christof > > On 20150324 21:01, Kyle Kastner wrote: >> It might be nice to talk about optimizing runtime and/or training time >> like SMAC did in their paper. I don't see any reason we couldn't do >> this in sklearn, and it might be of value to users since we don't >> really do deep learning as Andy said. >> >> On Tue, Mar 24, 2015 at 4:52 PM, Andy <t3k...@gmail.com> wrote: >>> On 03/24/2015 04:38 PM, Christof Angermueller wrote: >>>> Thanks Andy! I replied to your comments: >>>> https://docs.google.com/document/d/1bAWdiu6hZ6-FhSOlhgH-7x3weTluxRfouw9op9bHBxs/edit?usp=sharing. >>>> >>>> I summary, >>>> * I will not mentioned parallelization as an extended features, >>>> * suggest concrete data sets for benchmarking, >>>> * mentioned tasks for which I expect an improvement. >>> It is also important to have algorithms for which we expect improvements. >>> I'm not sure how much we want to focus on deep learning, as the MLP is >>> not merged. >>> >>>> Any further ideas? >>>> Where can I find the PR for gaussian_processes? I would like to know >>>> about what will be implemented and to which extend I can contribute. >>>> >>> As much as you want ;) >>> >>> ------------------------------------------------------------------------------ >>> Dive into the World of Parallel Programming The Go Parallel Website, >>> sponsored >>> by Intel and developed in partnership with Slashdot Media, is your hub for >>> all >>> things parallel software development, from weekly thought leadership blogs >>> to >>> news, videos, case studies, tutorials and more. Take a look and join the >>> conversation now. http://goparallel.sourceforge.net/ >>> _______________________________________________ >>> Scikit-learn-general mailing list >>> Scikit-learn-general@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general >> ------------------------------------------------------------------------------ >> Dive into the World of Parallel Programming The Go Parallel Website, >> sponsored >> by Intel and developed in partnership with Slashdot Media, is your hub for >> all >> things parallel software development, from weekly thought leadership blogs to >> news, videos, case studies, tutorials and more. Take a look and join the >> conversation now. http://goparallel.sourceforge.net/ >> _______________________________________________ >> Scikit-learn-general mailing list >> Scikit-learn-general@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general > > -- > Christof Angermueller > cangermuel...@gmail.com > http://cangermueller.com > > > ------------------------------------------------------------------------------ > Dive into the World of Parallel Programming The Go Parallel Website, sponsored > by Intel and developed in partnership with Slashdot Media, is your hub for all > things parallel software development, from weekly thought leadership blogs to > news, videos, case studies, tutorials and more. Take a look and join the > conversation now. http://goparallel.sourceforge.net/ > _______________________________________________ > Scikit-learn-general mailing list > Scikit-learn-general@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general ------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ Scikit-learn-general mailing list Scikit-learn-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
