ptrendx edited a comment on issue #15589: [Discussion] 1.6.0 Roadmap URL: https://github.com/apache/incubator-mxnet/issues/15589#issuecomment-526373840 We have multiple improvements to BERT inference and training speed that we would like to be part of 1.6 release: - [x] Softmax optimizations (#15545 ) - [x] Pointwise fusion for GPU (#15167 ) - [x] Eliminate common expressions (#15657 ) - [x] Bias speed improvements (#16039 ) - [x] Aggregated AdamW optimizer (#16398) - [x] Aggregated zeroing of the gradients (#16446) - [x] Aggregated sum of squares operator (also used in LARS, #16122) - [x] Embedding gradient optimization (#16355) - [ ] Faster multihead attention operator (#16408)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
