Thanks Olivier! I really appreciate your updates on these issues! As far as I can tell Averaged SGD is similar to the Averaged Perceptron in that you simply average the weight vectors after each iteration (i.e. training sample). Of course, this can be done very efficiently in constant time and memory. AFAIK you cannot use this strategy for L1 regularization, tough.
The results in [Xu 2011] are pretty impressive given the simplicity of the algorithm - we should definitely give it a try. Unfortunately, the algorithm shares some of the undesirable properties of SGD: you need a number of heuristics to make it work (e.g. learning rate schedule, averaging start point t_0) best, Peter [Xu 2011] http://arxiv.org/pdf/1107.2490v1 2011/10/11 Olivier Grisel <[email protected]>: > I think people here (e.g. @pprett) might be interested in the > following new release of Léon Bottou's influential project: > > http://leon.bottou.org/projects/sgd > > It did not know about Averaged SGD. Will have to read the cited references. > > I wonder if those results are transposable on online clustering and / > or online matrix factorization. > > -- > Olivier > http://twitter.com/ogrisel - http://github.com/ogrisel > > ------------------------------------------------------------------------------ > All the data continuously generated in your IT infrastructure contains a > definitive record of customers, application performance, security > threats, fraudulent activity and more. Splunk takes this data and makes > sense of it. Business sense. IT sense. Common sense. > http://p.sf.net/sfu/splunk-d2d-oct > _______________________________________________ > Scikit-learn-general mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general > -- Peter Prettenhofer ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2d-oct _______________________________________________ Scikit-learn-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
