RBMs are a factorization of a generally intractable problem - as you mention it is still O(n**2) but much better than the combinatorial brute force thing that the RBM factorization replaces. There might be faster RBM algorithms around but I don't know of any faster implementations that don't use GPU code. There might be specific RBMs for sparse data, but in general RBMs are designed for latent factor discovery in dense, low-ish dimensional (1000 - 10000 features) input data.
The current sklearn code for RBMs is just binary-binary, as you mention. The Gaussian version (both binary-Gaussian and Gaussian-Gaussian) exists but is not implemented in the library. I have personally had a harder time training real-valued latent variable models, compared to binarized versions - if you can "binarize" your problem it is worth trying that as a first shot. One hack I have tried on other tasks is to use KMeans clustering to get binary codes (by mapping data points to the nearest cluster, then representing this with one-hot / LabelBinarizer format). Then the RBM will give cluster indices, which can be mapped back to cluster centers or made into "stochastic" units by sampling from a Gaussian / RBF centered at the cluster center, with some fixed variance you choose. This is kind of weird but worked as well as could be expected for my task. RBMs are still used to form latent variable models such as the RNN-RBM for timeseries modeling (here http://deeplearning.net/tutorial/rnnrbm.html), or in the spike and slab RBM / DBN for texture modeling (here https://ift6266h15.files.wordpress.com/2015/04/20_vae.pdf). On Mon, Jul 27, 2015 at 5:17 AM, Mika S <siddhupi...@gmail.com> wrote: > i am using scikit learn's RBM implementation. There are two problems: > > 1. > > The running time is O(d^2) where d is the number of features. This > becomes a problem in using high dimensionality sparse features. Consider > features that come from feature hashing for instance. > 2. > > It only allows for binary visible features. Do I have to change the > sklearn code to have non binary units or there is some trick that I am > unaware of? > > I am expecting RBMs with 4 features to have a better fit than a mixture of > 2 gaussians (that has a similar number of parameters). Has anyone seen any > experiments done on RBMs for unsupervised modeling other than pretraining? > > > > ------------------------------------------------------------------------------ > > _______________________________________________ > Scikit-learn-general mailing list > Scikit-learn-general@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general > >
------------------------------------------------------------------------------
_______________________________________________ Scikit-learn-general mailing list Scikit-learn-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
