eric-haibin-lin opened a new issue #11241: Conv1D throws CUDNN_STATUS_EXECUTION_FAILED URL: https://github.com/apache/incubator-mxnet/issues/11241 Setup: ``` $ pip install mxnet-cu90==1.1.0 $ nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2017 NVIDIA Corporation Built on Fri_Sep__1_21:08:03_CDT_2017 Cuda compilation tools, release 9.0, V9.0.176 $ ls /usr/local/cuda/lib64/libcudnn.so.7.0.3 /usr/local/cuda/lib64/libcudnn.so.7.0.3 ``` Run the following script: ``` W_REQ = 'add' shape = (1, 65536, 1) ctx = mx.gpu() kwargs = {'no_bias': True, 'kernel': (1,), 'num_filter': 1} x = mx.sym.var('x') w = mx.sym.var('w') x_grad = mx.nd.zeros(shape, ctx=ctx) w_grad = mx.nd.zeros(shape, ctx=ctx) args_grad = {'x': x_grad, 'w': w_grad} sym = mx.sym.Convolution(x, w, **kwargs) executor = sym.bind(ctx, grad_req={'x': 'null', 'w': W_REQ}, args={'x': mx.nd.ones(shape, ctx=ctx), 'w': mx.nd.ones(shape, ctx=ctx)}, args_grad=args_grad) executor.forward() executor.backward([mx.nd.ones((1,1,1), ctx=ctx)]) mx.nd.waitall() ``` Gives the following error: ``` [06:31:41] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable) terminate called after throwing an instance of 'dmlc::Error' what(): [06:31:41] src/engine/./threaded_engine.h:359: [06:31:41] src/operator/nn/./cudnn/cudnn_convolution-inl.h:242: Check failed: e == CUDNN_STATUS_SUCCESS (8 vs. 0) cuDNN: CUDNN_STATUS_EXECUTION_FAILED Stack trace returned 10 entries: [bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2a9e78) [0x7f705d9a3e78] [bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2aa288) [0x7f705d9a4288] [bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2a920d1) [0x7f706018c0d1] [bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x262f5e7) [0x7f705fd295e7] [bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x24570bb) [0x7f705fb510bb] [bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x245d7d4) [0x7f705fb577d4] [bt] (6) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x243e2ed) [0x7f705fb382ed] [bt] (7) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2442bdb) [0x7f705fb3cbdb] [bt] (8) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2442db6) [0x7f705fb3cdb6] [bt] (9) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x243f68b) [0x7f705fb3968b] A fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging. Stack trace returned 9 entries: [bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2a9e78) [0x7f705d9a3e78] [bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2aa288) [0x7f705d9a4288] [bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x243e594) [0x7f705fb38594] [bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2442bdb) [0x7f705fb3cbdb] [bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2442db6) [0x7f705fb3cdb6] [bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x243f68b) [0x7f705fb3968b] [bt] (6) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f70d400bc80] [bt] (7) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f70d52a66ba] [bt] (8) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f70d4fdc3dd] ``` Note that there's no error if `W_REQ` is changed to 'write'.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
