MoisesHer opened a new pull request #19703: URL: https://github.com/apache/incubator-mxnet/pull/19703
## Description ## This PR sets -INF as returned value in FWD for masked values in log-softmax. log(0) = -INF ## Checklist ## ### Essentials ### - [x] Changes are complete (i.e. I finished coding on this PR) - [x] All changes have test coverage - [x] Code is well-documented ### Changes ### - [x] Change returned values in FWD log-softmax for masked values. Before returning 0, now returning -INF - [x] Remove duplicated feature: scale_factor & temperature ## Comments ## In Backwards, the gradient for -INF elements should be +INF, but this would introduce numerical issues. Instead, I propose to set those with zero. Similar discussion for PyTorch can be found [here](https://github.com/pytorch/pytorch/issues/31829) @sxjscience ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
