MoisesHer opened a new pull request #19894: URL: https://github.com/apache/incubator-mxnet/pull/19894
## Description ## Use multi-tensor kernel strategy for resetting gradients. Undo [https://github.com/apache/incubator-mxnet/pull/16716](https://github.com/apache/incubator-mxnet/pull/16716) ## Checklist ## ### Essentials ### - [x] Changes are complete (i.e. I finished coding on this PR) - [x] All changes have test coverage ([https://github.com/sxjscience/mxnet/blob/master/tests/python/unittest/test_numpy_gluon.py#L185](https://github.com/sxjscience/mxnet/blob/master/tests/python/unittest/test_numpy_gluon.py#L185)) - [x] Code is well-documented ### Changes ### - [x] in zero_grad method, use ndarray.reset_arrays for resetting multiple arrays within same kernel ## Comments ## Performance observed training BERT-large on single V100 GPU: ........................................................................**Throughput samples / s** **BatchSize - BatchAccumulation............Pre-Change............This-PR (multi-tensor )............ Improvement(%)** ................4 - 2 .............................................. 29.82 ..........................36.24.............................................. 21.5 ................4 - 4............................................... 36.89............................41.45..............................................12.3 ................8 - 4............................................... 45.25............................48.41.............................................. 6.9 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
