nswamy commented on a change in pull request #14830: [WIP] Use env var to enforce safe accumulation in ReduceAxesCompute URL: https://github.com/apache/incubator-mxnet/pull/14830#discussion_r281790114
########## File path: docs/faq/env_var.md ########## @@ -280,6 +280,11 @@ When USE_PROFILER is enabled in Makefile or CMake, the following environments ca - Values: Int ```(default=4)``` - This variable controls how many CuDNN dropout state resources to create for each GPU context for use in operator. +* MXNET_ENFORCE_SAFE_ACCUMULATION + - Values: Values: 0(false) or 1(true) ```(default=0)``` + - If this variable is set, the accumulation will enter the safe mode, meaning accumulation is done in a data type of higher precision than + the input data type, leading to more accurate results with a possible performance loss and backward compatibility loss. Review comment: Also I am concerned about float32-> float64 casting as described in https://github.com/apache/incubator-mxnet/issues/14722. In our offline discussion @haojin2 and @eric-haibin-lin seem to allude that somehow now with float64 casting for accumulation it is exposing a higher loss which is the correct value and the earlier model which yields lower loss is actually wrong. I am not sure I understand this despite our long conversation (sorry!). Also I found another [old PR](https://github.com/dmlc/mshadow/pull/236) which casted the Accumulation variable to higher type, however [comment] (https://github.com/dmlc/mshadow/pull/236#discussion_r113569026) seem to suggest float32->float32 is sufficient(I checked with @cjolivier01 for the reasoning he does not remember unfortunately). My suggestion is to retain float32->float32 unless you have a convincing argument. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
