akshayjadiyanv commented on issue #33078:
URL: https://github.com/apache/beam/issues/33078#issuecomment-5688732714
@tvalentyn
here is the full state and what is left. Two of the steps need write access
I do not have, and two are a PR I can send.
A serialized TRT engine can only be deserialized by the same TensorRT major
version and GPU architecture that built it. All three `.trt` files under
`gs://apache-beam-ml/models/` were built with TensorRT 8.x in 2022:
1. `single_tensor_features_engine.trt`
2. `multiple_tensor_features_engine.trt`
3. `ssd_mobilenet_v2_320x320_coco17_tpu-8.trt`
Since #39922 the handler requires TensorRT 10+ , none of these load.
Rebuilding them is the blocker.
Already done in #39922 :
`sdks/python/test-suites/containers/tensorrt_runinference/build_test_engines.py`
rebuilds all three from the ONNX sources **already staged in the same bucket**
— nothing new needs uploading as a source. It verifies each result by loading
it back through `TensorRTEngineHandlerNumPy`, and only uploads after
verification passes. `README.md` next to it has the commands.
I have run it on a Tesla T4 with TensorRT 11.0.0.114 in
`nvcr.io/nvidia/tensorrt:26.06-py3`. All three build and verify:
- `single_tensor_features_engine` → `[2.5, 10.5, -5.5, 20.5]`, exactly what
the unit test asserts
- `multiple_tensor_features_engine` → `[17.5, 36.5, -27.5, 3.0]`, likewise
- `ssd_mobilenet_v2_320x320_coco17_tpu-8` → confident detections on both
COCO images from `tensorrt_image_file_names.txt`, top scores 0.77 and 0.72
1. You can either run the script yourself on a T4:
```
sudo apt-get install -y docker.io
sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart
docker
sudo docker run --rm --gpus all -v "$PWD:/w" -w /w
nvcr.io/nvidia/tensorrt:26.06-py3 bash -c "\
pip install -q --break-system-packages 'apache-beam[gcp]' cuda-python
pillow && \
python3 build_test_engines.py --dest
gs://SOME_BUCKET_YOU_CAN_WRITE/models"
```
…or I can hand over the files I already built and verified, whichever you
prefer.
The script deliberately refuses `--dest gs://apache-beam-ml/models`, since
that bucket has no object versioning and an overwrite could not be undone.
**One thing to decide, since it is your bucket.** The script names its
output with a `_trt11` suffix, so the 2022 files survive. Can keep the same if
we are okay with overwrite.
2. Rebuild and push the test container image
`us.gcr.io/apache-beam-testing/python-postcommit-it/tensor_rt:latest`, from
the updated `tensor_rt.dockerfile` on master. It now uses
`nvcr.io/nvidia/tensorrt:26.06-py3` (TensorRT 11.0, CUDA 13.3) and
`apache/beam_python3.12_sdk`. The old pushed image is still TensorRT 8.6 /
Python 3.10, so the IT would fail on the image alone.
### What I will send as a PR once those land
**3. Bump the driver request.** `common.gradle:454` asks for
`install-nvidia-driver` with no version, while the vLLM ITs on 486/490/503 ask
for `install-nvidia-driver:5xx`. TRT 11 with CUDA 13.3 needs a 580+ driver - so
without this it fails at CUDA init rather than at engine load.
**4. Re-enable the task.** `tensorRTtests` is commented out in
`inferencePostCommitITPy312`. It already moved to the py312 suite in #39922
,since the new base image is Python 3.12.
Plus the path updates from step 1 if we go with the suffixed names.
### Not a blocker, just so it is not a surprise
1. The IT runs the example pipeline and passes if it completes - it does not
diff against a golden output, so nothing needs regenerating.
2. Also, `num_detections` is always 100 because the ONNX pins `EfficientNMS`
`keepTopK=100`; that was equally true of the 2022 engine, so it is not a
regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
