Verified that the non-determinism is brought in by cuDNN backward algorithms for Conv and Pooling operators. Results are consistent on CPU and GPU without cuDNN. This is expected behavior as per current MXNet implementation. See here for an open issue where there is discussion on providing an option for determinism - https://github.com/apache/incubator-mxnet/issues/11341
Please note that, enabling deterministic cuDNN algorithm significantly slows the computation - https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#reproducibility Resolving the issue here in favor of new feature request tracked at https://github.com/apache/incubator-mxnet/issues/11341 Please reopen if closed in error. [ Full content available at: https://github.com/apache/incubator-mxnet/issues/9410 ] This message was relayed via gitbox.apache.org for [email protected]
