samskalicky commented on pull request #5986: URL: https://github.com/apache/incubator-tvm/pull/5986#issuecomment-674158779
Lots of testing over the past month, definitely reduced the occurrence of the problem by making the runtime not static. But still seeing intermittent failures (depending on model can be more prevalent) ``` Segmentation fault: 11 *** Error in `python': double free or corruption (!prev): 0x000055becd8c4460 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x777f5)[0x7fd5a64827f5] /lib/x86_64-linux-gnu/libc.so.6(+0x8038a)[0x7fd5a648b38a] /lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7fd5a648f58c] /lib/x86_64-linux-gnu/libc.so.6(+0x3a035)[0x7fd5a6445035] /lib/x86_64-linux-gnu/libc.so.6(+0x3a055)[0x7fd5a6445055] /home/ubuntu/anaconda3/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x7fc3125)[0x7fd5498ea125] /lib/x86_64-linux-gnu/libc.so.6(+0x354c0)[0x7fd5a64404c0] /usr/local/cuda/lib64/libcudart.so.10.0(+0x1d9fe)[0x7fd4fc1909fe] /usr/local/cuda/lib64/libcudart.so.10.0(+0x2296b)[0x7fd4fc19596b] /usr/local/cuda/lib64/libcudart.so.10.0(cudaSetDevice+0x47)[0x7fd4fc1bd087] /home/ubuntu/anaconda3/lib/python3.7/site-packages/neomxnet/libdlr.so(_ZN3tvm7runtime13CUDADeviceAPI13FreeDataSpaceE9DLContextPv+0x3a)[0x7fd4eda8652a] ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
