qingqing01 opened a new issue #13399: One question about batch norm code when setting use_global_stats True. URL: https://github.com/apache/incubator-mxnet/issues/13399 In the cuDNN impl of batch norm, the code in `src/operator/nn/cudnn/cudnn_batch_norm-inl.h ` is: ``` CUDNN_CALL(cudnnBatchNormalizationBackward( s->dnn_handle_, mode, &a, &b, &a, req[cudnnbatchnorm::kGamma] == kWriteTo ? &b: &b_add, io_desc_, x.dptr_, io_desc_, dy.dptr_, io_desc_, dx.dptr_, mean_desc_, gamma.dptr_, dgamma.dptr_, dbeta.dptr_, param_.eps, global_stats ? nullptr : save_mean.dptr_, global_stats ? nullptr : save_inv_var.dptr_)); ``` In the cuDNN impl of batch norm, why the savedMean and savedInvVariance are nullptr when `global_stats` is True? From the cuDNN doc: > savedMean, savedInvVariance Inputs. Optional cache parameters containing saved intermediate results computed during the forward pass. For this to work correctly, the layer's x and bnScale, bnBias data has to remain unchanged until the backward function is called. Note that both of these parameters can be NULL but only at the same time. It is recommended to use this cache since the memory overhead is relatively small. https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#cudnnBatchNormalizationBackward it seems that when these two arguments are nullptr, the `cudnnBatchNormalizationBackward` will calculate mean/var by using the input X inner this API. I think it is not equivalent to the `real` backward (see following format ) when `global_stats` is True and using global mean/variance in forward. When using global mean/variance, the forward is: y = scale * ( (x-global_mean) / sqrt(global_var) ) + bias the dx is: dx = scale * dy / sqrt(global_var) So, I'm a little confused.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
