Hi Niketan and Mike, As we are trying to implement this Bayesian Optimization, should we take input from more committers as well as this optimizer approach seems to have a couple of ways to implement. We may need to find out which suits us the best.
Thanks, Janardhan On Sat, Jul 22, 2017 at 3:41 PM, Janardhan Pulivarthi < [email protected]> wrote: > Dear committers, > > We will be planning to add bayesian optimizer support for both the ML and > Deep learning tasks for the SystemML. Relevant jira link: > https://issues.apache.org/jira/browse/SYSTEMML-979 > > The following is a simple outline of how we are going implement it. Please > feel free to make any kind of changes. In this google docs link: > http://bit.do/systemml-bayesian > > Description: > > Bayesian optimization is a sequential design strategy for global > optimization of black-box functions that doesn’t require derivatives. > > Process: > > 1. > > First we select a point that will be the best as far as the no. of > iterations that has happened. > 2. > > Candidate point selection with sampling from Sobol quasirandom > sequence generator the space. > 3. > > Gaussian process hyperparameter sampling with surrogate slice sampling > method. > > > Components: > > 1. > > Selecting the next point to Evaluate. > > [image: nextpoint.PNG] > > We specify a uniform prior for the mean, m, and width 2 top-hat priors for > each of the D length scale parameters. As we expect the observation noise > generally to be close to or exactly zero, v(nu) is given a horseshoe > prior. The covariance amplitude theta0 is given a zero mean, unit variance > lognormal prior, theta0 ~ ln N (0, 1). > > > > 1. > > Generation of QuasiRandom Sobol Sequence. > > Which kind of sobol patterns are needed? > > [image: sobol patterns.PNG] > > How many dimensions do we need? > > This paper argues that its generation target dimension is 21201. [pdf link > <https://researchcommons.waikato.ac.nz/bitstream/handle/10289/967/Joe%20constructing.pdf> > ] > > > > 1. > > Surrogate Slice Sampling. > > [image: surrogate data sampling.PNG] > > > References: > > 1. For the next point to evaluate: > > https://papers.nips.cc/paper/4522-practical-bayesian- > optimization-of-machine-learning-algorithms.pdf > > http://www.dmi.usherb.ca/~larocheh/publications/gpopt_nips_appendix.pdf > > > 2. QuasiRandom Sobol Sequence Generator: > > https://researchcommons.waikato.ac.nz/bitstream/handle/10289/967/Joe% > 20constructing.pdf > > > 3. Surrogate Slice Sampling: > > http://homepages.inf.ed.ac.uk/imurray2/pub/10hypers/hypers.pdf > > > > Thank you so much, > > Janardhan > > > >
