[
https://issues.apache.org/jira/browse/SPARK-1503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596991#comment-14596991
]
Aaron Staple commented on SPARK-1503:
-------------------------------------
[~josephkb] and [~lewuathe] Sure, happy to coordinate. So far I have just been
duplicating the the convergence tolerance check in TFOCS, the matlab package on
which the accelerated gradient descent implementation is based. TFOCS also
tests for convergence by checking if the relative change in the weight vector
is below a specified threshold. But there are some differences from the
SPARK-3382 implementation. For example in TFOCS the relative difference between
new and old weight vectors is measured with respect to the new weight vector
instead of the old. And if the new weight vector is smaller than the unit
vector the convergence test is changed to be an absolute rather than relative
difference between successive weight vectors. I am just describing the
implementation here, happy to discuss further and potentially look at making
changes.
Here is the relevant code if you are interested (there is also a separate
condition when the weight vector does not change between iterations):
https://github.com/cvxr/TFOCS/blob/e34c0daeb136935d23b8df506de8b7b191f6b0a3/private/tfocs_iterate.m#L19-L24
> Implement Nesterov's accelerated first-order method
> ---------------------------------------------------
>
> Key: SPARK-1503
> URL: https://issues.apache.org/jira/browse/SPARK-1503
> Project: Spark
> Issue Type: New Feature
> Components: MLlib
> Reporter: Xiangrui Meng
> Assignee: Aaron Staple
> Attachments: linear.png, linear_l1.png, logistic.png, logistic_l2.png
>
>
> Nesterov's accelerated first-order method is a drop-in replacement for
> steepest descent but it converges much faster. We should implement this
> method and compare its performance with existing algorithms, including SGD
> and L-BFGS.
> TFOCS (http://cvxr.com/tfocs/) is a reference implementation of Nesterov's
> method and its variants on composite objectives.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
