rszper commented on code in PR #29509: URL: https://github.com/apache/beam/pull/29509#discussion_r1401286815
########## website/www/site/content/en/documentation/ml/about-ml.md: ########## @@ -38,38 +40,48 @@ limitations under the License. You can use Apache Beam to: * Process large volumes of data, both for preprocessing and for inference. -* Experiment with your data during the exploration phase of your project and provides a seamless transition when - upscaling your data pipelines as part of your MLOps ecosystem in a production environment. +* Experiment with your data during the exploration phase of your project. +* Upscale your data pipelines as part of your ML ops ecosystem in a production environment. * Run your model in production on a varying data load, both in batch and streaming. ## AI/ML workloads -You can use Apache Beam for data validation, data preprocessing, model validation, and model deployment/inference. +You can use Apache Beam for data validation, data preprocessing, model validation, and model deployment and inference.  -1. Data ingestion: Incoming new data is stored in your file system or database, or it's published to a messaging queue. -2. **Data validation**: After you receieve your data, check the quality of your data. For example, you might want to detect outliers and calculate standard deviations and class distributions. -3. **Data preprocessing**: After you validate your data, transform the data so that it is ready to use to train your model. -4. Model training: When your data is ready, you can start training your AI/ML model. This step is typically repeated multiple times, depending on the quality of your trained model. -5. Model validation: Before you deploy your new model, validate its performance and accuracy. +1. Data ingestion: Incoming new data is either stored in your file system or database, or published to a messaging queue. +2. **Data validation**: After you receieve your data, check the quality of the data. For example, you might want to detect outliers and calculate standard deviations and class distributions. +3. **Data preprocessing**: After you validate your data, transform the data so that it's ready to use to train your model. +4. Model training: When your data is ready, train your AI/ML model. This step is typically repeated multiple times, depending on the quality of your trained model. +5. Model validation: Before you deploy your model, validate its performance and accuracy. 6. **Model deployment**: Deploy your model, using it to run inference on new or existing data. -To keep your model up to date and performing well as your data grows and evolves, run these steps multiple times. In addition, you can apply MLOps to your project to automate the AI/ML workflows throughout the model and data lifecycle. Use orchestrators to automate this flow and to handle the transition between the different building blocks in your project. +To keep your model up to date and performing well as your data grows and evolves, run these steps multiple times. In addition, you can apply ML ops to your project to automate the AI/ML workflows throughout the model and data lifecycle. Use orchestrators to automate this flow and to handle the transition between the different building blocks in your project. ## Use RunInference -The recommended way to implement inference in Apache Beam is by using the [RunInference API](/documentation/sdks/python-machine-learning/). RunInference takes advantage of existing Apache Beam concepts, such as the `BatchElements` transform and the `Shared` class, to enable you to use models in your pipelines to create transforms optimized for machine learning inferences. The ability to create arbitrarily complex workflow graphs also allows you to build multi-model pipelines. +The [RunInference API](/documentation/sdks/python-machine-learning/) is a `PTransform` optimized for machine learning inferences that lets you efficiently use ML models in your pipelines. The API includes the following features: -You can integrate your model in your pipeline by using the corresponding model handlers. A `ModelHandler` is an object that wraps the underlying model and allows you to configure its parameters. Model handlers are available for PyTorch, scikit-learn, and TensorFlow. Examples of how to use RunInference for PyTorch, scikit-learn, and TensorFlow are shown in this [notebook](https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_pytorch_tensorflow_sklearn.ipynb). +- To efficiently feed your model, dynamically batches inputs based on pipeline throughput. +- To optimize your pipeline for ML inference, takes advantage of existing Apache Beam concepts, such as the `BatchElements` transform and the `Shared` class. +- To balance memory and throughput usage, determines the optimal number of models to load using a central model manager. +- Ensures that your pipeline uses the most recently deployed version of your model with the [Automatic model refresh](#automatic-model-refresh) feature. +- Supports [multiple frameworks and model hubs](#use-pre-trained-models), including Tensorflow, Pytorch, Sklearn, XGBoost, Hugging Face, TensorFlow Hub, Vertex AI, TensorRT, and ONNX. +- Supports arbitrary frameworks using a [custom model handler](#use-custom-models). +- Supports [multi-model pipelines](#multi-model-pipelines). +- Lets you [use GPUs](/documentation/ml/runinference-metrics) to increase inference speed. Because GPUs can process multiple computations simultaneously, they are optimized for training artificial intelligence and deep learning models. Review Comment: I edited this and added a link to the Dataflow GPU docs. ########## website/www/site/content/en/documentation/ml/about-ml.md: ########## @@ -158,22 +370,108 @@ The RunInference API doesn't currently support making remote inference calls usi * Consider monitoring and measuring the performance of a pipeline when deploying, because monitoring can provide insight into the status and health of the application. -### Use custom models +## Multi-model pipelines -If you would like to use a model that isn't specified by one of the supported frameworks, the RunInference API is designed flexibly to allow you to use any custom machine learning models. -You only need to create your own `ModelHandler` or `KeyedModelHandler` with logic to load your model and use it to run the inference. +Use the RunInference transform to add multiple inference models to your pipeline. Multi-model pipelines can be useful for A/B testing or for building out cascade models made up of models that perform tokenization, sentence segmentation, part-of-speech tagging, named entity extraction, language detection, coreference resolution, and more. For more information, see [Multi-model pipelines](https://beam.apache.org/documentation/ml/multi-model-pipelines/). + +### A/B Pattern + +``` +with pipeline as p: + data = p | 'Read' >> beam.ReadFromSource('a_source') + model_a_predictions = data | RunInference(<model_handler_A>) + model_b_predictions = data | RunInference(<model_handler_B>) +``` + +Where `model_handler_A` and `model_handler_B` are the model handler setup code. + +### Cascade Pattern + +``` +with pipeline as p: + data = p | 'Read' >> beam.ReadFromSource('a_source') + model_a_predictions = data | RunInference(<model_handler_A>) + model_b_predictions = model_a_predictions | beam.Map(some_post_processing) | RunInference(<model_handler_B>) +``` + +Where `model_handler_A` and `model_handler_B` are the model handler setup code. + +### Use Resource Hints for Different Model Requirements + +When using multiple models in a single pipeline, different models may have different memory or worker SKU requirements. +Resource hints allow you to provide information to a runner about the compute resource requirements for each step in your +pipeline. + +For example, the following snippet extends the previous cascade pattern with hints for each RunInference call +to specify RAM and hardware accelerator requirements: + +``` +with pipeline as p: + data = p | 'Read' >> beam.ReadFromSource('a_source') + model_a_predictions = data | RunInference(<model_handler_A>).with_resource_hints(min_ram="20GB") + model_b_predictions = model_a_predictions + | beam.Map(some_post_processing) + | RunInference(<model_handler_B>).with_resource_hints( + min_ram="4GB", + accelerator="type:nvidia-tesla-k80;count:1;install-nvidia-driver") +``` + +For more information on resource hints, see [Resource hints](/documentation/runtime/resource-hints/). -A simple example can be found in [this notebook](https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_custom_inference.ipynb). -The `load_model` method shows how to load the model using a popular `spaCy` package while `run_inference` shows how to run the inference on a batch of examples. ## Model validation Model validation allows you to benchmark your model’s performance against a previously unseen dataset. You can extract chosen metrics, create visualizations, log metadata, and compare the performance of different models with the end goal of validating whether your model is ready to deploy. Beam provides support for running model evaluation on a TensorFlow model directly inside your pipeline. The [ML model evaluation](/documentation/ml/model-evaluation) page shows how to integrate model evaluation as part of your pipeline by using [TensorFlow Model Analysis (TFMA)](https://www.tensorflow.org/tfx/guide/tfma). +## Troubleshooting + +If you run into problems with your pipeline or job, this section lists issues that you might encounter and provides suggestions for how to fix them. + +### Unable to batch tensor elements + +RunInference uses dynamic batching. However, the RunInference API cannot batch tensor elements of different sizes, so samples passed to the RunInferene transform must be the same dimension or length. If you provide images of different sizes or word embeddings of different lengths, the following error might occur: Review Comment: Updated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
