haojin2 commented on issue #14830: Use env var to enforce safe accumulation in ReduceAxesCompute URL: https://github.com/apache/incubator-mxnet/pull/14830#issuecomment-493199318 @anirudh2290 Firstly, the "accurate" in the doc is describing the accumulation, NO claims where made about its influence on final accuracy/performance of any models built with MXNet with this env var turned on. Please make sure you do not factor in any assumed contexts that a random MXNet user may not have when reviewing this piece of doc. Thanks for pointing out that the softmax is still always done in the safe mode at this moment, which led me to a revisit to the original issue. Here's a table that lists out all the final test losses got from running this training script: | Version | Test Loss | Safe Softmax? | |-----------------|------|-----| | 1.5.0b20190220 | 4.79 | No | | 1.5.0b20190221 | 5.18 | Yes | | 1.5.0b20190313 | 4.90 | Yes | | This PR | 4.74 | Yes | We can see that: 1) The final test loss would have some variance even with the softmax's behavior unchanged. 2) The final test loss could vary to a higher or lower value when you change the behavior of softmax. From the 2 facts above I think maybe the increase in loss observed in the original issue is just some random variance, which does not necessarily need a "fix". I can definitely add this extra line or be more specific that the accurate is describing the accumulation, not any models built with MXNet if that's what you're looking for to get this PR merged ASAP for the 1.5 code freeze, but still, my original doc is not making any claims about its influence on the final accuracy of models in the first place so I would still think such disclaimers as nice-to-have but not really necessary.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
