ptrendx edited a comment on issue #15589: [Discussion] 1.6.0 Roadmap URL: https://github.com/apache/incubator-mxnet/issues/15589#issuecomment-526373840 We have multiple improvements to BERT inference and training speed that we would like to be part of 1.6 release: - [x] Softmax optimizations (#15545 ) - [ ] Pointwise fusion for GPU (#15167 ) - [ ] Eliminate common expressions (#15657 ) - [ ] Bias speed improvements (#16039 ) - [ ] Aggregated AdamW optimizer (not yet PR'ed) - [ ] Aggregated zeroing of the gradients (not yet PR'ed) - [ ] Aggregated sum of squares operator (also used in LARS, #16122) - [ ] Embedding gradient optimization (not yet PR'ed) - [ ] Faster multihead attention operator (not yet PR'ed)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
