This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 7449b67a6d1 Editing notebooks to prepare for DevSite import (#28949)
7449b67a6d1 is described below
commit 7449b67a6d1019c82d852a69ae8cb093199335e6
Author: Rebecca Szper <[email protected]>
AuthorDate: Thu Oct 12 06:54:39 2023 -0700
Editing notebooks to prepare for DevSite import (#28949)
---
.../notebooks/beam-ml/automatic_model_refresh.ipynb | 8 ++++----
examples/notebooks/beam-ml/mltransform_basic.ipynb | 6 ++----
examples/notebooks/beam-ml/per_key_models.ipynb | 18 +++++++++---------
3 files changed, 15 insertions(+), 17 deletions(-)
diff --git a/examples/notebooks/beam-ml/automatic_model_refresh.ipynb
b/examples/notebooks/beam-ml/automatic_model_refresh.ipynb
index 9cbab0a1417..3bafa4f0788 100644
--- a/examples/notebooks/beam-ml/automatic_model_refresh.ipynb
+++ b/examples/notebooks/beam-ml/automatic_model_refresh.ipynb
@@ -248,7 +248,7 @@
" This example uses `TFModelHandlerTensor` as the model handler and
the `resnet_101` model trained on [ImageNet](https://www.image-net.org/).\n",
"\n",
"\n",
- "For DataflowRunner, the model needs to be stored remote location
accessible by the Beam pipeline. So we will download `ResNet101` model and
upload it to the GCS location.\n"
+ "For the Dataflow runner, you need to store the model in a remote
location that the Apache Beam pipeline can access. For this example, download
the `ResNet101` model, and upload it to the Google Cloud Storage bucket.\n"
],
"metadata": {
"id": "_AUNH_GJk_NE"
@@ -392,7 +392,7 @@
"source": [
"2. To read and preprocess the images, use the `preprocess_image`
function. This example uses `Cat-with-beanie.jpg` for all inferences.\n",
"\n",
- " **Note**: Image used for prediction is licensed in CC-BY. The
creator is listed in the
[LICENSE.txt](https://storage.googleapis.com/apache-beam-samples/image_captioning/LICENSE.txt)
file."
+ " **Note**: The image used for prediction is licensed in CC-BY. The
creator is listed in the
[LICENSE.txt](https://storage.googleapis.com/apache-beam-samples/image_captioning/LICENSE.txt)
file."
],
"metadata": {
"id": "8-sal2rFAxP2"
@@ -424,7 +424,7 @@
"cell_type": "markdown",
"source": [
"3. Pass the images to the RunInference `PTransform`. RunInference
takes `model_handler` and `model_metadata_pcoll` as input parameters.\n",
- " * `model_metadata_pcoll` is a side input `PCollection` to the
RunInference `PTransform`. This side input is used to update the `model_uri` in
the `model_handler` without needing to stop the Apache Beam pipeline\n",
+ " * `model_metadata_pcoll` is a side input `PCollection` to the
RunInference `PTransform`. This side input updates the `model_uri` in the
`model_handler` while the Apache Beam pipeline runs.\n",
" * Use `WatchFilePattern` as side input to watch a `file_pattern`
matching `.keras` files. In this case, the `file_pattern` is
`'gs://BUCKET_NAME/dataflow/*keras'`.\n",
"\n"
],
@@ -483,7 +483,7 @@
"source": [
"### Watch for the model update\n",
"\n",
- "After the pipeline starts processing data and when you see output
emitted from the RunInference `PTransform`, upload a `resnet152` model saved in
`.keras` format to a Google Cloud Storage bucket location that matches the
`file_pattern` you defined earlier.\n"
+ "After the pipeline starts processing data, when you see output
emitted from the RunInference `PTransform`, upload a `resnet152` model saved in
the `.keras` format to a Google Cloud Storage bucket location that matches the
`file_pattern` you defined earlier.\n"
],
"metadata": {
"id": "wYp-mBHHjOjA"
diff --git a/examples/notebooks/beam-ml/mltransform_basic.ipynb
b/examples/notebooks/beam-ml/mltransform_basic.ipynb
index 820bc3400b5..fd305bddb3b 100644
--- a/examples/notebooks/beam-ml/mltransform_basic.ipynb
+++ b/examples/notebooks/beam-ml/mltransform_basic.ipynb
@@ -65,7 +65,7 @@
"id": "d3b81cf2-8603-42bd-995e-9e14631effd0"
},
"source": [
- "This notebook demonstrates how to use `MLTransform` to preprocess
your data for machine learning models. `MLTransform` is a `PTransform` that
wraps multiple Apache Beam data processing transforms. As a result,
`MLTransform` gives you the ability to preprocess different types of data in
multiple ways with one transform.\n",
+ "This notebook demonstrates how to use `MLTransform` to preprocess
your data for machine learning models. `MLTransform` is a `PTransform` that
wraps multiple Apache Beam data processing transforms. With `MLTransform`, you
can preprocess different types of data in multiple ways with one transform.\n",
"\n",
"This notebook uses data processing transforms defined in the
[apache_beam/ml/transforms/tft](https://beam.apache.org/releases/pydoc/current/apache_beam.ml.transforms.tft.html)
module."
]
@@ -423,8 +423,6 @@
"source": [
"### Scale the data by using the z-score\n",
"\n",
- "Scale to the data using the z-score\n",
- "\n",
"Similar to `ScaleTo01`, use
[ScaleToZScore](https://beam.apache.org/releases/pydoc/current/apache_beam.ml.transforms.tft.html#apache_beam.ml.transforms.tft.ScaleToZScore)
to scale the values by using the
[z-score]([z-score](https://www.tensorflow.org/tfx/transform/api_docs/python/tft/scale_to_z_score#:~:text=Scaling%20to%20z%2Dscore%20subtracts%20out%20the%20mean%20and%20divides%20by%20standard%20deviation.%20Note%20that%20the%20standard%20deviation%20computed%20here%20is%20base
[...]
],
"metadata": {
@@ -607,7 +605,7 @@
"\n",
"The previous examples show how to preprocess data for model training.
This example uses the same preprocessing steps on the inference data. By using
the same steps on the inference data, you can maintain consistent results.\n",
"\n",
- "Preprocess the data going into the inference by using the same
preprocessing steps used on the data prior to training. To do this with
`MLTransform`, pass the artifact location from the previous transforms to the
parameter `read_artifact_location`. `MLTransform` uses the values and artifacts
produced in the previous steps. You don't need to provide the transforms,
because they are saved with the artifacts in the artifact location.\n"
+ "Preprocess the data used by the inference by using the same
preprocessing steps that you used on the data prior to training. When using
`MLTransform`, pass the artifact location from the previous transforms to the
parameter `read_artifact_location`. `MLTransform` uses the values and artifacts
produced in the previous steps. You don't need to provide the transforms,
because they are saved with the artifacts in the artifact location.\n"
],
"metadata": {
"id": "kcnQSwkA-eSA"
diff --git a/examples/notebooks/beam-ml/per_key_models.ipynb
b/examples/notebooks/beam-ml/per_key_models.ipynb
index a5dfa0d54e3..53845c0b3e1 100644
--- a/examples/notebooks/beam-ml/per_key_models.ipynb
+++ b/examples/notebooks/beam-ml/per_key_models.ipynb
@@ -70,7 +70,7 @@
"\n",
"In Apache Beam, the recommended way to run inference is to use the
`RunInference` transform. By using a `KeyedModelHandler`, you can efficiently
run inference with O(100s) of models without having to manage memory
yourself.\n",
"\n",
- "This notebook demonstrates how to use a `KeyedModelHandler` to run
inference in an Apache Beam pipeline with multiple different models on a
per-key basis. This notebook uses pretrained pipelines from Hugging Face.
Before continuing with this notebook, it is recommended that you walk through
the [beginner RunInference
notebook](https://colab.sandbox.google.com/github/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_pytorch_tensorflow_sklearn.ipynb)."
+ "This notebook demonstrates how to use a `KeyedModelHandler` to run
inference in an Apache Beam pipeline with multiple different models on a
per-key basis. This notebook uses pretrained pipelines from Hugging Face.
Before continuing with this notebook, it is recommended that you walk through
the [Use RunInference in Apache
Beam](https://colab.sandbox.google.com/github/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_pytorch_tensorflow_sklearn.ipynb)
notebook."
],
"metadata": {
"id": "ZAVOrrW2An1n"
@@ -81,7 +81,7 @@
"source": [
"## Install dependencies\n",
"\n",
- "First, install both Apache Beam and the dependencies needed by
Hugging Face."
+ "Install both Apache Beam and the dependencies needed by Hugging Face."
],
"metadata": {
"id": "_fNyheQoDgGt"
@@ -144,7 +144,7 @@
"\n",
"A model handler is the Apache Beam method used to define the
configuration needed to load and invoke models. Because this example uses two
models, we define two model handlers, one for each model. Because both models
are incapsulated within Hugging Face pipelines, we use the model handler
`HuggingFacePipelineModelHandler`.\n",
"\n",
- "In this notebook, we load the models using Hugging Face and run them
against an example. The models produce different outputs."
+ "For this example, load the models using Hugging Face, and then run
them against an example. The models produce different outputs."
],
"metadata": {
"id": "uEqljVgCD7hx"
@@ -350,7 +350,7 @@
"source": [
"## Define the examples\n",
"\n",
- "Next, define examples to input into the pipeline. The examples
include their correct classifications."
+ "Define examples to input into the pipeline. The examples include the
correct classifications."
],
"metadata": {
"id": "yd92MC7YEsTf"
@@ -387,7 +387,7 @@
"class FormatExamples(beam.DoFn):\n",
" \"\"\"\n",
" Map each example to a tuple of ('<model_name>-<actual_sentiment>',
'example').\n",
- " We use these keys to map our elements to the correct models.\n",
+ " Use these keys to map our elements to the correct models.\n",
" \"\"\"\n",
" def process(self, element: Tuple[str, str]) -> Iterable[Tuple[str,
str]]:\n",
" yield (f'distilbert-{element[1]}', element[0])\n",
@@ -402,7 +402,7 @@
{
"cell_type": "markdown",
"source": [
- "Use the formatted keys to define a `KeyedModelHandler` that maps keys
to the `ModelHandler` used for those keys. The `KeyedModelHandler` method lets
you define an optional `max_models_per_worker_hint`, which limits the number of
models that can be held in a single worker process at one time. If you're
worried about your worker running out of memory, use this option. For more
information about managing memory, see [Use a keyed
ModelHandler](https://beam.apache.org/documentation/s [...]
+ "Use the formatted keys to define a `KeyedModelHandler` that maps keys
to the `ModelHandler` used for those keys. The `KeyedModelHandler` method lets
you define an optional `max_models_per_worker_hint`, which limits the number of
models that can be held in a single worker process at one time. If your worker
might run out of memory, use this option. For more information about managing
memory, see [Use a keyed
ModelHandler](https://beam.apache.org/documentation/sdks/python-machine- [...]
],
"metadata": {
"id": "IP65_5nNGIb8"
@@ -428,9 +428,9 @@
"source": [
"## Postprocess the results\n",
"\n",
- "The `RunInference` transform returns a Tuple containing:\n",
+ "The `RunInference` transform returns a tuple that contains the
following objects:\n",
"* the original key\n",
- "* a `PredictionResult` object containing the original example and the
inference.\n",
+ "* a `PredictionResult` object containing the original example and the
inference\n",
"Use those outputs to extract the relevant data. Then, to compare each
model's prediction, group this data by the original example."
],
"metadata": {
@@ -505,7 +505,7 @@
"source": [
"## Run the pipeline\n",
"\n",
- "Put together all of the pieces to run a single Apache Beam pipeline."
+ "To run a single Apache Beam pipeline, combine the previous steps."
],
"metadata": {
"id": "-LrpmM2PGAkf"
