damccorm commented on code in PR #24301: URL: https://github.com/apache/beam/pull/24301#discussion_r1029328038
########## website/www/site/content/en/documentation/ml/data-processing.md: ########## @@ -53,17 +53,17 @@ ib.collect(beam_df.describe()) ib.collect(beam_df.isnull()) ``` -For a full end-to-end example on how to implement data exploration and data preprocessing with Beam and the DataFrame API for your AI/ML project, you can follow the [Beam Dataframe API tutorial for AI/ML](https://github.com/apache/beam/tree/master/examples/notebooks/beam-ml/dataframe_api_preprocessing.ipynb). +For a full end-to-end example that implements data exploration and data preprocessing with Apache Beam and the DataFrame API for your AI/ML project, see the [Beam Dataframe API tutorial for AI/ML](https://github.com/apache/beam/tree/master/examples/notebooks/beam-ml/dataframe_api_preprocessing.ipynb). ## Data pipeline for ML A typical data preprocessing pipeline consists of the following steps: -1. Reading and writing data: read/write the data from your filesystem, database or messaging queue. Beam has a rich set of [IO connectors](https://beam.apache.org/documentation/io/built-in/) for ingesting and writing data. -2. Data cleaning: you typically want to filter and clean your data before using it for your ML model. Examples of this are to remove duplicate or irrelevant data, correct mistakes in your dataset, filter out unwanted outliers and handle missing data. -3. Data transformations: your data needs to fit the expected input your model needs to train. Examples of this are normalization, one-hot encoding, scaling and vectorizing. -4. Data enrichment: often you will want to enrich your data with external data sources to make your data more meaningful or more easy to interpret by an ML model. An example of this is to transform a city name or address into a coordinate. -5. Data validation & metrics: you also want to make sure your data adheres to a specific set of requirements that can be validated in your pipeline. And you might want to report some metrics from your data such as the class distributions. +1. Read and write data: Read and write the data from your file system, database, or messaging queue. Apache Beam has a rich set of [IO connectors](https://beam.apache.org/documentation/io/built-in/) for ingesting and writing data. +2. Data cleaning: Filter and clean your data before using it in your ML model. You might remove duplicate or irrelevant data, correct mistakes in your dataset, filter out unwanted outliers, or handle missing data. +3. Data transformations: Your data needs to fit the expected input your model needs to train. You might need to normalize, one-hot encode, scale, or vectorize your data. +4. Data enrichment: You might want to enrich your data with external data sources to make your data more meaningful or easier for an ML model to interpret. For example, you might want to transform a city name or address into a coordinate. Review Comment: ```suggestion 4. Data enrichment: You might want to enrich your data with external data sources to make your data more meaningful or easier for an ML model to interpret. For example, you might want to transform a city name or address into a set of coordinates. ``` (or other wording) ########## website/www/site/content/en/documentation/ml/online-clustering.md: ########## @@ -98,33 +98,33 @@ The file structure for clustering_pipeline is: ├── main.py └── setup.py -`pipeline/transformations.py` contains the code for different `beam.DoFn` that are used in the pipeline +`pipeline/transformations.py` contains the code for the different `beam.DoFn` that are used in the pipeline. -`pipeline/options.py` contains the pipeline options to configure the Dataflow pipeline +`pipeline/options.py` contains the pipeline options to configure the Dataflow pipeline. -`config.py` defines some variables like GCP PROJECT_ID, NUM_WORKERS that are used multiple times +`config.py` defines variables that are used multiple times, like Google Cloud PROJECT_ID and NUM_WORKERS. -`setup.py` defines the packages/requirements for the pipeline to run +`setup.py` defines the packages and requirements for the pipeline to run. -`main.py` contains the pipeline code and some additional function used for running the pipeline +`main.py` contains the pipeline code and some additional function used for running the pipeline. Review Comment: ```suggestion `main.py` contains the pipeline code and some additional functions used for running the pipeline. ``` ########## website/www/site/content/en/documentation/ml/overview.md: ########## @@ -17,65 +17,76 @@ limitations under the License. # AI/ML pipelines -Beam <3 Machine Learning. Being productive and successful as a machine learning practitioner is often dependent on your ability to efficiently leverage large volumes of data in a way that is uniquely tailored to your resources, requirements and budget. When starting your next AI/ML project or upscaling an existing one, a vital tool you should consider adding to your project is Beam. +Beam <3 machine learning. Being productive and successful as a machine learning practitioner is often dependent on your ability to efficiently leverage large volumes of data in a way that is uniquely tailored to your resources, requirements, and budget. Whether starting your next AI/ML project or upscaling an existing project, consider adding Apache Beam to your project. -Beam enables you to process large volumes of data, both for preprocessing and for inference. It allows you to experiment with your data during the exploration phase of your project, while providing a seamless transition to upscaling your data pipelines as part of your MLOps ecosystem in a production environment. It allows you to run your model in production on a varying data load, both in batch and streaming. +* Apache Beam enables you to process large volumes of data, both for preprocessing and for inference. +* It allows you to experiment with your data during the exploration phase of your project and provides a seamless transition when + upscaling your data pipelines as part of your MLOps ecosystem in a production environment. +* It enables you to run your model in production on a varying data load, both in batch and streaming. ## AI/ML workloads -Let’s take a look at the different building blocks that we need to build an end-to-end AI/ML use case, and where Beam will come in handy in building those blocks. +Let’s take a look at the different building blocks that we need to create an end-to-end AI/ML use case and where Apache Beam can help. - + -1. Data ingestion: incoming new data will be stored in your filesystem, database or published on a messaging queue. -2. **Data validation**: once you have received your data you need to check the quality of your data such as detecting outliers and reporting on standard deviations and class distributions. -3. **Data preprocessing**: after validating your data, you need to transform it so that it is ready to be used for training your model. -4. Model training: once your data is ready, you can start training your AI/ML model. This step will typically be repeated multiple times depending on the quality of your trained model. -5. Model validation: before deploying your new model you need to validate its performance and accuracy. -6. **Model deployment**: finally you can deploy your model, meaning it can run inference on any new or existing data. +1. Data ingestion: Incoming new data is stored in your file system or database, or it's published to a messaging queue. +2. **Data validation**: After you receieve your data, check the quality of your data. For example, you might want to detect outliers and calculate standard deviations and class distributions. +3. **Data preprocessing**: After you validate your data, transform the data so that it is ready to use to train your model. +4. Model training: When your data is ready, you can start training your AI/ML model. This step is typically repeated multiple times, depending on the quality of your trained model. +5. Model validation: Before you deploy your new model, validate its performance and accuracy. +6. **Model deployment**: Deploy your model, using it to run inference on new or existing data. -All of these steps can be executed multiple times, as your data might grow and evolve over time and you want your model to stay up to date and guarantee its best performance. This is why it is very important to apply MLOps to your project, meaning that you aim to automate the AI/ML workflows throughout the model and data lifecycle. This can be achieved by using orchestrators that automate this flow and handle the transition between the different building blocks in your project. +To keep your model up to date and performing well as your data grows and evolves, run these steps multiple times. In addition, you can apply MLOps to your project to automate the AI/ML workflows throughout the model and data lifecycle. Use orchestrators to automate this flow and to handle the transition between the different building blocks in your project. -Beam can be used for data validation, data preprocessing and model deployment/inference. We will now take a look at these different building blocks in more detail and at how they can be orchestrated. Finally, you can also find full examples of AI/ML pipelines in Beam. +You can use Apache Beam for data validation, data preprocessing, and model deployment/inference. The next section examines these building blocks in more detail and explores how they can be orchestrated. ## Data processing -Data validation and preprocessing can be done in Beam by setting up data pipelines that transform your data and output metrics computed from your data. Beam has a rich set of [IO connectors](https://beam.apache.org/documentation/io/built-in/) for ingesting and writing data, which means you can easily integrate it with your existing filesystem, database or messaging queue. When developing your ML model, you can also first explore your data with the [Beam DataFrame API](https://beam.apache.org/documentation/dsls/dataframes/overview/) so that you can identify and implement the required preprocessing steps allowing you to iterate faster towards production. Another common pattern is that the steps executed during preprocessing need to also be applied before running inference, in which case you can use the same Beam implementation twice. Lastly, if you need to do post-processing after running inference, this can also be done as part of your model inference pipeline. +You can use Apache Beam for data validation and preprocessing by setting up data pipelines that transform your data and output metrics computed from your data. Beam has a rich set of [I/O connectors](https://beam.apache.org/documentation/io/built-in/) for ingesting and writing data, which allows you to integrate it with your existing file system, database, or messaging queue. + +When developing your ML model, you can also first explore your data with the [Beam DataFrame API](https://beam.apache.org/documentation/dsls/dataframes/overview/). The DataFrom API lets you identify and implement the required preprocessing steps, making it easier for you to move your pipeline to production. + +Steps executed during preprocessing often also need to be applied before running inference, in which case you can use the same Beam implementation twice. Lastly, when you need to do postprocessing after running inference, Apache Beam allows you to incoporate the postprocessing into your model inference pipeline. Further reading: * [AI/ML pipelines in Beam: data processing](/documentation/ml/data-processing) ## Inference -Beam provides different ways of implementing inference as part of your pipeline. This way you can run your ML model directly in your pipeline and apply it on big scale datasets, both in batch and streaming pipelines. +Beam provides different ways to implement inference as part of your pipeline. You can run your ML model directly in your pipeline and apply it on big scale datasets, both in batch and streaming pipelines. ### RunInference + The recommended way to implement inference is by using the [RunInference API](https://beam.apache.org/documentation/sdks/python-machine-learning/). RunInference takes advantage of existing Apache Beam concepts, such as the `BatchElements` transform and the `Shared` class, to enable you to use models in your pipelines to create transforms optimized for machine learning inferences. The ability to create arbitrarily complex workflow graphs also allows you to build multi-model pipelines. -You can easily integrate your model in your pipeline by using the corresponding model handlers. A `ModelHandler` is an object that wraps the underlying model and allows you to configure its parameters. Model handlers are available for PyTorch, Scikit-learn and TensorFlow. Examples of how to use RunInference for PyTorch, Scikit-learn and TensorFlow are shown in this [notebook](https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_pytorch_tensorflow_sklearn.ipynb). +You can integrate your model in your pipeline by using the corresponding model handlers. A `ModelHandler` is an object that wraps the underlying model and allows you to configure its parameters. Model handlers are available for PyTorch, scikit-learn, and TensorFlow. Examples of how to use RunInference for PyTorch, scikit-learn, and TensorFlow are shown in this [notebook](https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_pytorch_tensorflow_sklearn.ipynb). -GPUs are optimized for training artificial intelligence and deep learning models as they can process multiple computations simultaneously. RunInference also allows you to use GPUs for significant inference speedup. An example of how to use RunInference with GPUs is demonstrated [here](/documentation/ml/runinference-metrics). +Because they can process multiple computations simultaneously, GPUs are optimized for training artificial intelligence and deep learning models. RunInference also allows you to use GPUs for significant inference speedup. An example of how to use RunInference with GPUs is demonstrated on the [RunInference metrics](/documentation/ml/runinference-metrics) page. ### Custom Inference -As of now, RunInference API doesn't support making remote inference calls (e.g. Natural Language API, Cloud Vision API and others). Therefore, in order to use these remote APIs with Beam, one needs to write custom inference call. The [notebook](https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/custom_remote_inference.ipynb) shows how you can implement such a custom remote inference call using `beam.DoFn`. While implementing such a remote inference for real life projects, you need to think about following: -* API quotas and the heavy load you might incur on your external API. For optimizing the calls to external API, you can confgure `PipelineOptions` to limit the parallel calls to the external remote API. +The RunInference API doesn't currently support making remote inference calls using, for example, the Natural Language API or the Cloud Vision API. Therefore, in order to use these remote APIs with Apache Beam, you need to write custom inference calls. The [Remote inference in Apache Beam notebook](https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/custom_remote_inference.ipynb) shows how to implement a custom remote inference call using `beam.DoFn`. When you implement a remote inference for real life projects, consider the following factors: -* You must be prepared to encounter, identify, and handle failure as gracefully as possible. We recommend using techniques like `Exponential backoff` and `Dead letter queues`. +* API quotas and the heavy load you might incur on your external API. To optimize the calls to an external API, you can confgure `PipelineOptions` to limit the parallel calls to the external remote API. -* When running inference with an external API, you should batch your input together to allow for more efficient execution. +* Be prepared to encounter, identify, and handle failure as gracefully as possible. Use techniques like `Exponential backoff` and `Dead letter queues`. Review Comment: Should exponential backoff and dead letter queues be capitalized here? ########## website/www/site/content/en/documentation/ml/online-clustering.md: ########## @@ -55,35 +55,35 @@ The file structure for ingestion pipeline is: ├── main.py └── setup.py -`pipeline/utils.py` contains the code for loading the emotion dataset and two `beam.DoFn` that are used for data transformation +`pipeline/utils.py` contains the code for loading the emotion dataset and two `beam.DoFn` that are used for data transformation. -`pipeline/options.py` contains the pipeline options to configure the Dataflow pipeline +`pipeline/options.py` contains the pipeline options to configure the Dataflow pipeline. -`config.py` defines some variables like GCP PROJECT_ID, NUM_WORKERS that are used multiple times +`config.py` defines some variables that are used multiple times, like GCP PROJECT_ID and NUM_WORKERS. -`setup.py` defines the packages/requirements for the pipeline to run +`setup.py` defines the packages and requirements for the pipeline to run. -`main.py` contains the pipeline code and some additional function used for running the pipeline +`main.py` contains the pipeline code and some additional function used for running the pipeline. -### How to Run the Pipeline ? -First, make sure you have installed the required packages. +## Run the Pipeline +First, install the required packages. 1. Locally on your machine: `python main.py` 2. On GCP for Dataflow: `python main.py --mode cloud` The `write_data_to_pubsub_pipeline` contains four different transforms: -1. Load emotion dataset using HuggingFace Datasets (We take samples from 3 classes instead of 6 for simplicity) -2. Associate each text with a unique identifier (UID) -3. Convert the text into a format PubSub is expecting -4. Write the formatted message to PubSub +1. Load the emotion dataset using Hugging Face datasets (for simplicity, we take samples from three classes instead of six). +2. Associate each piece of text with a unique identifier (UID). +3. Convert the text into the format that Pub/Sub expects. +4. Write the formatted message to Pub/Sub. ## Clustering on Streaming Data -After having the data ingested to PubSub, we can now look into the second pipeline, where we read the streaming message from PubSub, convert the text to a embedding using a language model and cluster them using BIRCH. +After ingeting the data to Pub/Sub, examine the second pipeline, where we read the streaming message from Pub/Sub, convert the text to a embedding using a language model, and cluster the embedding using BIRCH. Review Comment: ```suggestion After ingesting the data to Pub/Sub, examine the second pipeline, where we read the streaming message from Pub/Sub, convert the text to a embedding using a language model, and cluster the embedding using BIRCH. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
