ballcue commented on issue #17126: Memory should be completely released after an OOM happens URL: https://github.com/apache/incubator-mxnet/issues/17126#issuecomment-579312728 I seem to be having the same problem. ``` File ..., line 225, in detect scores = net_out[idx].asnumpy() File "/opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/ndarray/ndarray.py", line 1996, in asnumpy ctypes.c_size_t(data.size))) File "/opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/base.py", line 253, in check_call raise MXNetError(py_str(_LIB.MXGetLastError())) mxnet.base.MXNetError: [05:22:25] /home/travis/build/dmlc/mxnet-distro/mxnet-build/3rdparty/mshadow/mshadow/././././cuda/tensor_gpu-inl.cuh:110: Check failed: err == cudaSuccess (2 vs. 0) : Name: MapPlanKernel ErrStr:out of memory Stack trace: [bt] (0) /opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x4b04cb) [0x7f8f0701e4cb] [bt] (1) /opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x2f59431) [0x7f8f09ac7431] [bt] (2) /opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x31b61ee) [0x7f8f09d241ee] [bt] (3) /opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x31b9a16) [0x7f8f09d27a16] [bt] (4) /opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x25db7a9) [0x7f8f091497a9] [bt] (5) /opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x25e1a1a) [0x7f8f0914fa1a] [bt] (6) /opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x25c1cd1) [0x7f8f0912fcd1] [bt] (7) /opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x25c51e0) [0x7f8f091331e0] [bt] (8) /opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x25c5476) [0x7f8f09133476] ``` @lorenzob were you able to solve the out of memory errors? @szha , if the problem is in the call to mshadow, is there a way of working around mshadow to avoid it? Thanks.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
