sxjscience opened a new issue #15759: [Optimizer] Gradient is mutated in the Adam optimizer URL: https://github.com/apache/incubator-mxnet/issues/15759 In the implementation of Adam: grad, mean and var are all mutated (See https://github.com/apache/incubator-mxnet/blob/master/src/operator/optimizer_op-inl.h#L1307-L1315). However, the `FMutateInput` flag is only set to {2, 3}, which should be {1, 2, 3}. (See https://github.com/apache/incubator-mxnet/blob/master/src/operator/optimizer_op.cc#L699) To reproduce the bug, you may check the value of the gradient before/after `adam_update` https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/optimizer/optimizer.py#L1226-L1227 . We can add the following into `optimizer.py` ```python grad1 = grad.asnumpy() adam_update(weight, grad, mean, var, out=weight, ...) grad2 = grad.asnumpy() import numpy as np np.testing.assert_allclose(grad1, grad2) ``` Create an adam optimizer with `wd=1E-3` and train any model. You will find that the gradient has been mutated.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
