masahi commented on issue #13639: URL: https://github.com/apache/tvm/issues/13639#issuecomment-1374381932
It seems the error happens before or at the very start of `run` - I added a debug print to see at which kernel this error is raised, but nothing is logged before the error is emitted. I have some finding on this issue. I'm pretty sure it is related to CUDA driver version, and in particular, it is related to how PyTorch (used to generate the input models) is installed. If I serialize the created module once, and run the test without importing PT ever, the test runs successfully. I also get this warning from PT during tracing, which supports my hypothesis. @KangHe000 @zxybazh Do you get this warning as well? ``` /home/masa/miniconda3/lib/python3.9/site-packages/torch/jit/_trace.py:443: UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:109.) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
