tvalentyn commented on code in PR #22131: URL: https://github.com/apache/beam/pull/22131#discussion_r929432277
########## sdks/python/apache_beam/ml/inference/tensorrt_inference.py: ########## @@ -0,0 +1,281 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# pytype: skip-file + +import logging +import sys +from typing import Any +from typing import Dict +from typing import Iterable +from typing import Optional +from typing import Sequence +from typing import Tuple + +import numpy as np + +import tensorrt as trt +from apache_beam.io.filesystems import FileSystems +from apache_beam.ml.inference.base import ModelHandler +from apache_beam.ml.inference.base import PredictionResult +from cuda import cuda + +TRT_LOGGER = trt.Logger(trt.Logger.INFO) + +logging.basicConfig(level=logging.INFO) +logging.getLogger("TensorRTEngineHandlerNumPy").setLevel(logging.INFO) +log = logging.getLogger("TensorRTEngineHandlerNumPy") + + +def _load_engine(engine_path): + file = FileSystems.open(engine_path, 'rb') + runtime = trt.Runtime(TRT_LOGGER) + engine = runtime.deserialize_cuda_engine(file.read()) + assert engine + return engine + + +def _load_onnx(onnx_path): + builder = trt.Builder(TRT_LOGGER) + network = builder.create_network( + flags=1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)) + parser = trt.OnnxParser(network, TRT_LOGGER) + with FileSystems.open(onnx_path) as f: + if not parser.parse(f.read()): + log.error("Failed to load ONNX file: %s", onnx_path) + for error in range(parser.num_errors): + log.error(parser.get_error(error)) + sys.exit(1) Review Comment: We should raise a ValueError instead of exiting the process. ########## sdks/python/apache_beam/ml/inference/tensorrt_inference.py: ########## @@ -0,0 +1,281 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# pytype: skip-file + +import logging +import sys +from typing import Any +from typing import Dict +from typing import Iterable +from typing import Optional +from typing import Sequence +from typing import Tuple + +import numpy as np + +import tensorrt as trt +from apache_beam.io.filesystems import FileSystems +from apache_beam.ml.inference.base import ModelHandler +from apache_beam.ml.inference.base import PredictionResult +from cuda import cuda + +TRT_LOGGER = trt.Logger(trt.Logger.INFO) + +logging.basicConfig(level=logging.INFO) +logging.getLogger("TensorRTEngineHandlerNumPy").setLevel(logging.INFO) +log = logging.getLogger("TensorRTEngineHandlerNumPy") + + +def _load_engine(engine_path): + file = FileSystems.open(engine_path, 'rb') + runtime = trt.Runtime(TRT_LOGGER) + engine = runtime.deserialize_cuda_engine(file.read()) + assert engine Review Comment: We should check how does this fail when GPUs are not provided (as in: is there a clear / actionable error). ########## sdks/python/apache_beam/examples/inference/README.md: ########## @@ -154,6 +169,52 @@ This writes the output to the `predictions.csv` with contents like: ``` Each line has data separated by a semicolon ";". The first item is the file name. The second item is a list of predicted instances. +--- +## Object Detection + +[`tensorrt_object_detection.py`](./tensorrt_object_detection.py) contains an implementation for a RunInference pipeline that performs object detection using [Tensorflow Object Detection's](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md) SSD MobileNet v2 320x320 architecture. + +The pipeline reads the images, performs basic preprocessing, passes them to the TensorRT implementation of RunInference, and then writes the predictions to a text file. + +### Dataset and model for image classification + +You will need to create or download images, and place them into your `IMAGES_DIR` directory. Popular dataset for such task is [COCO dataset](https://cocodataset.org/#home). COCO validation dataset can be obtained [here](http://images.cocodataset.org/zips/val2017.zip). +- **Required**: A path to a file called `IMAGE_FILE_NAMES` that contains the absolute paths of each of the images in `IMAGES_DIR` on which you want to run image segmentation. Paths can be different types of URIs such as your local file system, a AWS S3 bucket or GCP Cloud Storage bucket. For example: +``` +/absolute/path/to/000000000139.jpg +/absolute/path/to/000000289594.jpg +``` +- **Required**: A path to a file called `TRT_ENGINE` that contains the pre-built TensorRT engine from SSD MobileNet v2 320x320 model. You will need to [follow instructions](https://github.com/NVIDIA/TensorRT/tree/main/samples/python/tensorflow_object_detection_api) on how to download and convert this SSD model into TensorRT engine. At [Create ONNX Graph](https://github.com/NVIDIA/TensorRT/tree/main/samples/python/tensorflow_object_detection_api#create-onnx-graph) step, keep batch size at 1. As soon as you are done with [Build TensorRT Engine](https://github.com/NVIDIA/TensorRT/tree/main/samples/python/tensorflow_object_detection_api#build-tensorrt-engine) step. You can use resulted engine as `TRT_ENGINE` input. In addition, make sure that environment you use for TensorRT engine creation is the same environment you use to run TensorRT inference. It is related not only to TensorRT version, but also to a specific GPU used. Read more about it [here](https://docs.nvidia.com/deeplearning/ tensorrt/developer-guide/index.html#compatibility-serialized-engines). + +- **Required**: A path to a file called `OUTPUT`, to which the pipeline will write the predictions. +- **Optional**: `IMAGES_DIR`, which is the path to the directory where images are stored. Not required if image names in the input file `IMAGE_FILE_NAMES` have absolute paths. + +### Running `tensorrt_object_detection.py` + +To run the image classification pipeline locally, use the following command: Review Comment: 1. I assume one will need a NVIDIA GPU to successfully run this example. I think we should mention it? Or without GPUs one can still run the example, but with a degraded performance? 2. Related to 1: if we want to run the pipeline on the dataflow runner: will users still need a machine with GPU at _job submission time_? For example, to be able to install all the tensorrt dependencies and submit the pipeline for execution on Dataflow? ########## sdks/python/apache_beam/ml/inference/tensorrt_inference.py: ########## @@ -0,0 +1,281 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# pytype: skip-file + +import logging +import sys +from typing import Any +from typing import Dict +from typing import Iterable +from typing import Optional +from typing import Sequence +from typing import Tuple + +import numpy as np + +import tensorrt as trt +from apache_beam.io.filesystems import FileSystems +from apache_beam.ml.inference.base import ModelHandler +from apache_beam.ml.inference.base import PredictionResult +from cuda import cuda + +TRT_LOGGER = trt.Logger(trt.Logger.INFO) + +logging.basicConfig(level=logging.INFO) Review Comment: Are these logging changes here just for debugging purposes while we are iterating on the PR? See related: https://github.com/apache/beam/blob/01fa485b31bbaf649873eb27623539e2e2b27cf2/sdks/python/apache_beam/pipeline.py#L160 https://github.com/apache/beam/blob/01fa485b31bbaf649873eb27623539e2e2b27cf2/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L95 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
