eric-haibin-lin commented on a change in pull request #7720: [sparse] add ftrl
optimizer for sparse
URL: https://github.com/apache/incubator-mxnet/pull/7720#discussion_r137708002
##########
File path: python/mxnet/optimizer.py
##########
@@ -790,6 +789,29 @@ class Ftrl(Optimizer):
Referenced from *Ad Click Prediction: a View from the Trenches*, available
at
http://dl.acm.org/citation.cfm?id=2488200.
+ The optimizer updates the weight by::
+
+ rescaled_grad = clip(grad * rescale_grad, clip_gradient)
+ z += rescaled_grad - (sqrt(n + rescaled_grad**2) - sqrt(n)) * weight /
learning_rate
+ n += rescaled_grad**2
+ w = (sign(z) * lamda1 - z) / ((beta + sqrt(n)) / learning_rate + wd) *
(abs(z) > lamda1)
+
+ If the storage types of weight, state and grad are all ``row_sparse``, \
+ sparse updates are applied by::
+
+ for row in grad.indices:
+ rescaled_grad[row] = clip(grad[row] * rescale_grad, clip_gradient)
+ z[row] += rescaled_grad[row] - (sqrt(n[row] +
rescaled_grad[row]**2) - \
Review comment:
backslash produces a problem displaying these in the documentation page on
mxnet.io. Please keep it in one line and add ` # pylint: disable=line-too-long`
before this class and ` # pylint: disable=line-too-long` after the class
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
