2011/10/31 Olivier Grisel <[email protected]>:
> 2011/10/31 Fabian Pedregosa <[email protected]>:
>> Thanks for the info, see you there!,
>
> Great: it's time to read or re-read:
>
> http://hal.archives-ouvertes.fr/hal-00608041/en
> http://arxiv.org/abs/1107.2490
I quickly scanned through both papers again (while skipping all proofs
as usual :P) and I came up with a list of questions I would like to
ask Francis Bach (if not answered directly during his talk):
1- results hold with truncated gradient and l1 penalty and elastic net
since sub-gradient is bounded, right?
2- do those results can be generalized to l1-ball projection for fitting
linear models with a such a sparsity inducing regularizer?
3- do those results can be generalized to positive cone projections and l2 unit
ball projection so as to implement online NMF using ASGD?
4- Would any of your convergence bound no long hold if using Wei Xu
& Leon Bottou learning rate schedule?:
gamma = eta0 * (1 + lambda * eta0 * n)^-alpha (with lambda == 1 / C)
instead of:
gamma = C * n^-alpha
6- would Exponentially Weighted Averaging help move faster to the "ASGD
better than SGD" regime (by 'forgetting' the initial bad values of the
theta_n sequence) while preserving the convergence bounds of
the standard Polyak-Rupper-style averaging?
7- can the efficiency of minibatch k-means over both SGD k-means and batch
k-means be somehow related to Polyak-Ruppert averaging (despite
objective function being non convex at all)?
8- can those results be used as theoretical ground for finding convergence
bounds for parallel versions of SGD (each thread see the data in
different random orders and synchronize its weights from time to
time with the weights of other thread using averaging) and parallel
Averaged SGD?
Please feel free to comment, complement or give answers if you already
have them at hand.
--
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
------------------------------------------------------------------------------
RSA® Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
