A few things I'd add to Olivier's reply: First, it's not quite accurate to call it "layered RBMs". The RBM interpretation, and the CD-1 approximate training procedure, really only make sense in the context of a single layer/unsupervised training, but we then take the weights and biases and shove them into a deterministic feed-forward network. It just so happens that these serve as a nice initialization for either
a) a Deep Belief Network, a probabilistic model with stochastic binary units and an undirected memory similar to an RBM making up the two top layers, but directed connections downward from layer N-1 to 1, OR b) a deterministic deep autoencoder that computes a set of real values at each layer as 1/(1 + exp(-Wx - b)), except for the middle, if you want unrestricted real values instead of a quasi-binary code. This is neither a Deep Belief Network nor a stack of RBMs, just a regular feed forward neural network that has a particularly well chosen set of initial weights. This is what's described in the Science paper. Second, a note about Theano and why you want to use it. If you were to do the implementation "by hand", you'd basically write out the expression for the error function (probably squared error or something like that) in terms of the output units, figure out the gradients with respect to the outputs, then apply the chain rule (repeatedly) to obtain the partial derivatives with respect to each parameter in the network (weights and biases). This is both time consuming and error prone, and even if you did have a mathematically correct expression for a given weight update (which can be tricky, especially if you're not used to such derivations -- symbolic math packages can help but they don't tend to deal well with all the sums and crap), it may not be a very numerically stable form. Bugs creep in in the implementation phase, and it can take a lot of work in order to make the code fast by making sure you don't compute a subexpression more than once in two places. Basically, Theano can do the gradients for you, corrects many numerical stability problems (especially ones that crop up in neural networks, as that's what it was mainly developed for), and optimizes the computational graph so as to make the best possible use of information that's already been computed. I think that's enough of an ad, [email protected] if you need more info. :) On Mon, Nov 28, 2011 at 03:22:19AM +0100, Olivier Grisel wrote: > You should definitely have a look at theano that will probably run > much faster than pure numpy for this kind of models (esp. if you have > access to a GPU with the CUDA runtime). > > http://deeplearning.net/software/theano/ > > The deep learning tutorial [1] have a section on backpropagation [2] > and also on RBMs and DBN. You should also have a look at the Efficient > Backprop paper by Lecun et al. [5] > > [1] http://www.deeplearning.net/tutorial/ > [2] http://www.deeplearning.net/tutorial/mlp.html#mlp > [3] http://www.deeplearning.net/tutorial/rbm.html#rbm > [4] http://www.deeplearning.net/tutorial/DBN.html#dbn > [5] http://yann.lecun.com/exdb/publis/#lecun-98b > > Best, > > -- > Olivier > > ------------------------------------------------------------------------------ > All the data continuously generated in your IT infrastructure > contains a definitive record of customers, application performance, > security threats, fraudulent activity, and more. Splunk takes this > data and makes sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunk-novd2d > _______________________________________________ > Scikit-learn-general mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-novd2d _______________________________________________ Scikit-learn-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
