perdasilva edited a comment on issue #16966: [CD] dynamic libmxet pipeline fix URL: https://github.com/apache/incubator-mxnet/pull/16966#issuecomment-562069469 @DickJC123, I'm trying to fix CD and I think it's been failing since the [fuse op PR](https://github.com/apache/incubator-mxnet/pull/15167). Do you have any idea why it could be failing for the cuda 9.0 builds? ``` ====================================================================== ERROR: test_operator_gpu.test_batchnorm_training ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/local/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc return func(*arg, **kw) File "/work/mxnet/tests/python/gpu/../unittest/common.py", line 177, in test_new orig_test(*args, **kwargs) File "/work/mxnet/tests/python/gpu/../unittest/test_operator.py", line 1830, in test_batchnorm_training check_batchnorm_training('default') File "/work/mxnet/tests/python/gpu/../unittest/test_operator.py", line 1769, in check_batchnorm_training check_numeric_gradient(test, in_location, mean_std, numeric_eps=1e-2, rtol=0.16, atol=1e-2) File "/work/mxnet/python/mxnet/test_utils.py", line 1101, in check_numeric_gradient symbolic_grads = {k:executor.grad_dict[k].asnumpy() for k in grad_nodes} File "/work/mxnet/python/mxnet/test_utils.py", line 1101, in <dictcomp> symbolic_grads = {k:executor.grad_dict[k].asnumpy() for k in grad_nodes} File "/work/mxnet/python/mxnet/ndarray/ndarray.py", line 2532, in asnumpy ctypes.c_size_t(data.size))) File "/work/mxnet/python/mxnet/base.py", line 255, in check_call raise MXNetError(py_str(_LIB.MXGetLastError())) MXNetError: [21:10:06] src/operator/fusion/fused_op.cu:558: Check failed: compileResult == NVRTC_SUCCESS (6 vs. 0) : NVRTC Compilation failed. Please set environment variable MXNET_USE_FUSION to 0. ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
