On 3/21/2014 4:25 PM, James Bergstra wrote:
The proposal looks good to me! A few small comments:1. I'm confused by the paragraph on regularized ELMs: I think you mean that in cases where the hidden weights (the classifier?) are *underdetermined* because there are far more *unknowns* then *samples* then you need to regularize somehow. (Right!?)
I meant the opposite :) - there are usually far more "samples" than "unknowns". The unknowns depend on the number of hidden neurons and output neurons which is usually small.
Typically the hidden weights matrix (the weights going out of the hidden neurons to the output neuron) is a 150x1 matrix. In other words there are 150 hidden neurons and 1 output neuron. This means there are 150 unknown variables . Since least-square solutions can be considered as systems of linear equations, solving for 150 unknown variables is possible with 150 samples. But datasets usually are as large as 10, 000 samples, meaning the number of unique solutions are very large as well, hence overdetermined (http://en.wikipedia.org/wiki/Overdetermined_system).
Therefore, regularization would constrain the amount of solutions by making sure they satisfy a meaningful constraint - like SVM's maximization of the margins between classes.
Sorry that this wasn't clear in the proposal.
2. Testing: no mention of how you will test any of this work. It's hard to know when an ML algorithm is implemented well. How will you know? Usually reproducing published results is a good bar to aim for, which ones do you have in mind? E.g. if there are some results in your PhD thesis that you want to reproduce, then mention that. How long does it take to train such things, do you need access to big computers?
That's the main motivation of using Extreme Learning Machines; they take seconds to train ;). The only obstacle is memory, because it processes the matrices all at once; however, this is where Sequential ELMs come in :).
I will add another section explaining the evaluation of the algorithms. It would include, solving systems of linear equation by hand and comparing it with the algorithm's output; how does that sound? Obviously, this is besides testing for coding issues like checking whether the control flow works as intended.
A bit cheesy, but I intend to cross-check the algorithms' outputs with that of the MATLAB's versions of the implementations, and theano's implementation of deep networks. :)
3. If you are just now completing your Masters degree on such models, you might want to mention that in your proposal's "Past Work" section :)
Sure thing :).
On Fri, Mar 21, 2014 at 7:54 AM, Issam <issamo...@gmail.com <mailto:issamo...@gmail.com>> wrote:Hi all, I updated the Neural Network proposal in melange, http://www.google-melange.com/gsoc/proposal/public/google/gsoc2014/issamou/5668600916475904 Thank you. ~Issam ------------------------------------------------------------------------------ Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/13534_NeoTech _______________________________________________ Scikit-learn-general mailing list Scikit-learn-general@lists.sourceforge.net <mailto:Scikit-learn-general@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general ------------------------------------------------------------------------------ Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/13534_NeoTech _______________________________________________ Scikit-learn-general mailing list Scikit-learn-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------ Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/13534_NeoTech
_______________________________________________ Scikit-learn-general mailing list Scikit-learn-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
