aIbrahiim commented on issue #30644: URL: https://github.com/apache/beam/issues/30644#issuecomment-4281799789
Based on my investigation, I found that the failure is caused by GPU runtime availability on Dataflow workers and from Dataflow worker logs, the repeated error is: RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver The traceback shows failure during PyTorch CUDA initialization (torch._C._cuda_init()), inside Beam inference execution (apache_beam/ml/inference/pytorch_inference.py). I also observed quota related signals in the same run window (T4 GPU quota pressure in us-central1) CC: @damccorm @Abacn @Amar3tto -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
