when train to 13 epoch for coco, another error occurs: [13:44:22] src/resource.cc:262: Ignore CUDA Error [13:44:22] src/storage/storage.cc:65: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading CUDA: initialization error
Stack trace returned 10 entries: [bt] (0) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x379e1a) [0x7fadcc375e1a] [bt] (1) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x37a451) [0x7fadcc376451] [bt] (2) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x3024ddd) [0x7fadcf020ddd] [bt] (3) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x302cab8) [0x7fadcf028ab8] [bt] (4) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x302ff2c) [0x7fadcf02bf2c] [bt] (5) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x297291d) [0x7fadce96e91d] [bt] (6) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x297c414) [0x7fadce978414] [bt] (7) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x2987585) [0x7fadce983585] [bt] (8) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x29731f8) [0x7fadce96f1f8] [bt] (9) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x2973d24) [0x7fadce96fd24] [13:44:22] src/engine/threaded_engine_perdevice.cc:99: Ignore CUDA Error [13:44:22] /home/travis/build/dmlc/mxnet-distro/mxnet-build/3rdparty/mshadow/mshadow/./tensor_gpu-inl.h:35: Check failed: e == cudaSuccess CUDA: initialization error Stack trace returned 10 entries: [bt] (0) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x379e1a) [0x7fadcc375e1a] [bt] (1) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x37a451) [0x7fadcc376451] [bt] (2) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x297aea8) [0x7fadce976ea8] [bt] (3) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x2987572) [0x7fadce983572] [bt] (4) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x29731f8) [0x7fadce96f1f8] [bt] (5) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x2973d24) [0x7fadce96fd24] [bt] (6) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x3030221) [0x7fadcf02c221] [bt] (7) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x30302e2) [0x7fadcf02c2e2] [bt] (8) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x37d45a) [0x7fadcc37945a] [bt] (9) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x30345c9) [0x7fadcf0305c9] [13:44:22] src/resource.cc:262: Ignore CUDA Error [13:44:22] src/storage/storage.cc:65: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading CUDA: initialization error Stack trace returned 10 entries: [bt] (0) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x379e1a) [0x7fadcc375e1a] [bt] (1) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x37a451) [0x7fadcc376451] [bt] (2) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x3024ddd) [0x7fadcf020ddd] [bt] (3) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x302cab8) [0x7fadcf028ab8] [bt] (4) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x302ff2c) [0x7fadcf02bf2c] [bt] (5) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x297291d) [0x7fadce96e91d] [bt] (6) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x297c414) [0x7fadce978414] [bt] (7) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x2987585) [0x7fadce983585] [bt] (8) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x29731f8) [0x7fadce96f1f8] [bt] (9) /home/liang/.local/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x2973d24) [0x7fadce96fd24] terminate called after throwing an instance of 'std::system_error' what(): Invalid argument Segmentation fault (core dumped) [ Full content available at: https://github.com/apache/incubator-mxnet/issues/12619 ] This message was relayed via gitbox.apache.org for [email protected]
