sxjscience commented on a change in pull request #16715: Lamb optimizer update
URL: https://github.com/apache/incubator-mxnet/pull/16715#discussion_r349945634
##########
File path: src/operator/optimizer_op-inl.h
##########
@@ -1563,6 +1563,192 @@ inline void AdamUpdateEx(const nnvm::NodeAttrs& attrs,
}
}
+struct LambUpdatePhaseOneParam : public
dmlc::Parameter<LambUpdatePhaseOneParam> {
+ float beta1;
+ float beta2;
+ float epsilon;
+ float t;
Review comment:
@eric-haibin-lin @access2rohit I find this issue when reading the code.
Here, the `t` should be the number of updates and should not be stored as
`float`, which will lose the precision. I think we need to store it as index_t.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
