Lunderberg opened a new pull request, #16980: URL: https://github.com/apache/tvm/pull/16980
Prior to this commit, the RAII handler in `NDArray` would always attempt to free a cuda memory allocation on destruction. However, the call to `cudaFree` may throw an exception. If this happens during stack unwinding due to a previously-thrown exception, this causes the program to immediately terminate, making it difficult to identify the source of the original error. This can commonly occur if an async compute kernel performs an illegal memory access. An exception is thrown from the next cuda API call following the asynchronous error, causing the stack to unwind. If the stack contains any `NDArray` instances which reference cuda allocations, the destructor of these `NDArray` instances will attempt to free memory, triggering the segfault. This commit updates the `CUDADeviceAPI::FreeDataSpace` function to check if the program is currently unwinding the stack due to a thrown exception, while the cuda driver has been left in an unrecoverable state. If this occurs, no attempt to free memory is made, as all cuda API calls will result in an error, and the original exception is allowed to propagate. If the cuda driver is in an unrecoverable state, but no exception is currently unwinding the stack, then this may be the first cuda API call to occur after the asynchronous error. In this case, the `cudaFree` call is still performed, which throws the initial exception. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
