[
https://issues.apache.org/jira/browse/SINGA-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15551041#comment-15551041
]
ASF subversion and git services commented on SINGA-254:
-------------------------------------------------------
Commit 5716105be00b260b57fd6a35c72d065eccea5544 in incubator-singa's branch
refs/heads/master from [~flytosky]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-singa.git;h=5716105 ]
SINGA-254 Implement Adam for V1
Implemented Adam for pysinga.
Tested Adam for alexnet on cifar10, and the accuracy was 0.8 (SGD is
0.82); It was also not as good as SGD on VGG net (0.92); May need to
turn the learning rate.
Add one more argument 'step' into most functions in optimizer module for
the iteration ID within one epoch. Some optimization algorithms would
use it, e.g. Adam.
Update the batchnorm layer's backward function to return empty tensors
for mean and variance variables. The optimizers have to skip the
updating procedure if the grad tensor is empty.
> Implement Adam for V1
> ---------------------
>
> Key: SINGA-254
> URL: https://issues.apache.org/jira/browse/SINGA-254
> Project: Singa
> Issue Type: New Feature
> Reporter: wangwei
>
> This ticket implements the Adam algorithm in V1 for updating parameters.
> http://arxiv.org/abs/1412.6980v8
> It has been implemented in v0.3 SINGA-145
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
