2013/2/5 David Lambert <caliband...@gmail.com>: > Olivier Grisel wrote: >> Am I right to assume that the main reference for ELM is >> http://www.ntu.edu.sg/home/egbhuang/? > > Absolutely. > > Lars Buitinck wrote: >> I've tried ELMs once. Apart from the hyped-up name, they're neural >> nets with random input-to-hidden weights (a very old idea) and >> least-squares fitting of the hidden-to-output weights. I could get >> some interesting results on the digits dataset >> (https://gist.github.com/larsmans/2493300) and they have the benefit >> of having two hyperparameters and very fast training, but in my experience >> the performance is very sensitive to the random seed. > > It was your gist that I used as the starting point, thanks :-) They are > extremely > sensitive to the random seed. > > Olivier Grisel wrote: >> Sounds like a good opportunity for ensembling then :) > > This is exactly why I started my implementation - I wanted them as > candidates > in an implementation of Rich Caruana's ensemble selection method > (http://www.cs.cornell.edu/~caruana/ctp/ct.papers/caruana.icml04.icdm06long.pdf) > that I was writing around scikit-learn.
I have read that exact same paper recently and indeed it would be very interesting to try to prototype this in a gist or branch with the sklearn API and base estimator implementations (independently of the ELM stuff). > There are a couple of reasons that I find them interesting - they're fast, > they can use > non-differentiable heterogeneous activation functions, the solution of the > hidden-output > weights need not be simple least squares fit (although something fancier > would take away > some of the speed), and the random encoding of the inputs by the hidden > layer reminds > me of the success of random patches in the Coates & Ng work on unsupervised > feature > learning in single layer networks. > > Olivier Grisel wrote: >> I guess that we won't integrate any new work that is a randomized >> version of MLP before this PR is first done implementing the >> traditional MLP learning. Actually after reading @larsmans implementation I think we could indeed start to investigate independently in pure python code and later think whether it's worth to make it more interoperable with the cython code of the MLP pull request. If you want to start working on a PR, please read this first: http://scikit-learn.org/dev/developers/index.html#contributing Also have a look at how other developers are working together on existing PRs to get a feel of the process: https://github.com/scikit-learn/scikit-learn/pulls Beware, this is a long and painful process but everybody will learn during the process (both contributors and reviewers) ;) -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ------------------------------------------------------------------------------ Free Next-Gen Firewall Hardware Offer Buy your Sophos next-gen firewall before the end March 2013 and get the hardware for free! Learn more. http://p.sf.net/sfu/sophos-d2d-feb _______________________________________________ Scikit-learn-general mailing list Scikit-learn-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
