rszper commented on code in PR #22069: URL: https://github.com/apache/beam/pull/22069#discussion_r922328517
########## sdks/python/apache_beam/examples/inference/README.md: ########## @@ -32,55 +32,70 @@ pip install apache-beam==2.40.0 ### PyTorch dependencies +The following installation requirements are for the files used in these examples. + The RunInference API supports the PyTorch framework. To use PyTorch locally, first install `torch`. ``` -pip install torch==1.11.0 +pip install torch==1.10.0 ``` -If you are using pretrained models from Pytorch's `torchvision.models` [subpackage](https://pytorch.org/vision/0.12/models.html#models-and-pre-trained-weights), you may also need to install `torchvision`. +If you are using pretrained models from Pytorch's `torchvision.models` [subpackage](https://pytorch.org/vision/0.12/models.html#models-and-pre-trained-weights), you also need to install `torchvision`. ``` pip install torchvision ``` -If you are using pretrained models from Hugging Face's `transformers` [package](https://huggingface.co/docs/transformers/index), you may also need to install `transformers`. +If you are using pretrained models from Hugging Face's `transformers` [package](https://huggingface.co/docs/transformers/index), you also need to install `transformers`. ``` pip install transformers ``` -For installation of the `torch` dependency on a distributed runner, like Dataflow, refer to these [instructions](https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#pypi-dependencies). +For installation of the `torch` dependency on a distributed runner such as Dataflow, refer to the +[PyPI dependency instructions](https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#pypi-dependencies). + +RunInference uses dynamic batching. However, the RunInference API cannot batch tensor elements of different sizes, because `torch.stack()` expects tensors of the same length. If you provide images of different sizes or word embeddings of different lengths, errors might occur. + +To avoid this issue: + +1. Either use elements that have the same size, or resize image inputs and word embeddings to make them +the same size. Depending on the language model and encoding technique, this option might not be available. +2. Disable batching by overriding the `batch_elements_kwargs` function in your ModelHandler and setting the maximum batch size (`max_batch_size`) to one: `max_batch_size=1`. For more information, see BatchElements PTransforms. Review Comment: I'm pretty sure I removed this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
