masahi commented on pull request #7123:
URL: https://github.com/apache/tvm/pull/7123#issuecomment-754891319
I can reproduce the issue by running ssd test in tensorflow/test_forward.py
with cuda target:
```
terminate called after throwing an instance of 'dmlc::Error'
what(): [05:42:13]
/home/masa/projects/dev/tvm/src/runtime/cuda/cuda_device_api.cc:126:
---------------------------------------------------------------
An internal invariant was violated during the execution of TVM.
Please read TVM's error reporting guidelines.
More details can be found here:
https://discuss.tvm.ai/t/error-reporting/7793.
---------------------------------------------------------------
Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading == false:
CUDA: an illegal memory access was encountered
Stack trace:
[bt] (0) /home/masa/projects/dev/tvm/build/libtvm.so(+0x14aa8e8)
[0x7f4fcb8ca8e8]
[bt] (1)
/home/masa/projects/dev/tvm/build/libtvm.so(tvm::runtime::CUDADeviceAPI::FreeDataSpace(DLContext,
void*)+0xe4) [0x7f4fcb8cabe4]
[bt] (2)
/home/masa/projects/dev/tvm/build/libtvm.so(tvm::runtime::NDArray::Internal::DefaultDeleter(tvm::runtime::Object*)+0x5b)
[0x7f4fcb8593fb]
[bt] (3)
/home/masa/projects/dev/tvm/build/libtvm.so(tvm::runtime::NDArray::CopyTo(DLContext
const&) const+0x325) [0x7f4fcb5e4915]
[bt] (4)
/home/masa/projects/dev/tvm/build/libtvm.so(tvm::runtime::vm::CopyTo(tvm::runtime::ObjectRef,
DLContext const&)+0x311) [0x7f4fcb884b11]
[bt] (5)
/home/masa/projects/dev/tvm/build/libtvm.so(tvm::runtime::vm::VirtualMachine::RunLoop()+0x2aee)
[0x7f4fcb880dde]
[bt] (6)
/home/masa/projects/dev/tvm/build/libtvm.so(tvm::runtime::vm::VirtualMachine::Invoke(tvm::runtime::vm::VMFunction
const&, std::vector<tvm::runtime::ObjectRef,
std::allocator<tvm::runtime::ObjectRef> > const&)+0x27) [0x7f4fcb881c17]
[bt] (7) /home/masa/projects/dev/tvm/build/libtvm.so(+0x14621f0)
[0x7f4fcb8821f0]
[bt] (8) /home/masa/projects/dev/tvm/build/libtvm.so(TVMFuncCall+0x63)
[0x7f4fcb835613]
```
@trevor-m Are you sure this is caused by `get_valid_counts` change? I've
also changed NMS in https://github.com/apache/tvm/pull/7172, I hope that
change is fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
