Hey David. Olivier dug up this paper by LeCun's group: http://users.ics.aalto.fi/kcho/papers/icml11.pdf I think this might be quite interesting for the MLP.
It is probably also interesting for the linear SGD. I'm surprised that they didn't compare against diagonal stochastic Levenberg-Marquardt with constant learning rate... Cheers, Andy On 05/15/2012 12:12 AM, David Marek wrote: > Hi, > > I have worked on multilayer perceptron and I've got a basic > implementation working. You can see it at > https://github.com/davidmarek/scikit-learn/tree/gsoc_mlp The most > important part is the sgd implementation, which can be found here > https://github.com/davidmarek/scikit-learn/blob/gsoc_mlp/sklearn/mlp/mlp_fast.pyx > > I have encountered a few problems and I would like to know your opinion. > > 1) There are classes like SequentialDataset and WeightVector which are > used in sgd for linear_model, but I am not sure if I should use them > here as well. I have to do more with samples and weights than just > multiply and add them together. I wouldn't be able to use numpy > functions like tanh and do batch updates, would I? What do you think? > Am I missing something that would help me do everything I need with > SequentialDataset? I implemented my own LossFunction because I need a > vectorized version, I think that is the same problem. > > 2) I used Andreas' implementation as an inspiration and I am not sure > I understand some parts of it: > * Shouldn't the bias vector be initialized with ones instead of > zeros? I guess there is no difference. > * I am not sure why is the bias updated with: > bias_output += lr * np.mean(delta_o, axis=0) > shouldn't it be: > bias_output += lr / batch_size * np.mean(delta_o, axis=0)? > * Shouldn't the backward step for computing delta_h be: > delta_h[:] = np.dot(delta_o, weights_output.T) * hidden.doutput(x_hidden) > where hidden.doutput is a derivation of the activation function for > hidden layer? > > I hope my questions are not too stupid. Thank you. > > David > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Scikit-learn-general mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Scikit-learn-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
