eric-haibin-lin commented on a change in pull request #14830: [WIP] Use env var to enforce safe accumulation in ReduceAxesCompute URL: https://github.com/apache/incubator-mxnet/pull/14830#discussion_r281808682
########## File path: docs/faq/env_var.md ########## @@ -280,6 +280,11 @@ When USE_PROFILER is enabled in Makefile or CMake, the following environments ca - Values: Int ```(default=4)``` - This variable controls how many CuDNN dropout state resources to create for each GPU context for use in operator. +* MXNET_ENFORCE_SAFE_ACCUMULATION + - Values: Values: 0(false) or 1(true) ```(default=0)``` + - If this variable is set, the accumulation will enter the safe mode, meaning accumulation is done in a data type of higher precision than + the input data type, leading to more accurate results with a possible performance loss and backward compatibility loss. Review comment: @nswamy I don't see a **concrete** reason why you want fp32 accumulation type for fp32 inputs if user set MXNET_ENFORCE_SAFE_ACCUMULATION=1. The only reason I see from the mshadow PR is to keep backward compatibility. If you use fp32 accumulation type for fp32 inputs, it is NOT safe accumulation. One simple test case where fp32 accumulation is not sufficient for safe accumulation: ``` # fp32 accumulation >>> import numpy as np >>> val1 = np.float32(3e38) >>> val2 = np.float32(-3e38) >>> sum = np.float32(3e38) >>> sum + val1 + val2 inf >>> sum64 = np.float64(3e38) >>> sum64 + val1 + val2 3e+38 >>> np.float32(sum64 + val1 + val2) 3e+38 ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
