tvalentyn commented on a change in pull request #16938:
URL: https://github.com/apache/beam/pull/16938#discussion_r832426684
##########
File path:
website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md
##########
@@ -123,3 +134,25 @@ If your pipeline uses non-Python packages (e.g. packages
that require installati
--setup_file /path/to/setup.py
**Note:** Because custom commands execute after the dependencies for your
workflow are installed (by `pip`), you should omit the PyPI package dependency
from the pipeline's `requirements.txt` file and from the `install_requires`
parameter in the `setuptools.setup()` call of your `setup.py` file.
+
+## Pre-building SDK container image
+
+In pipeline execution modes where a Beam runner launches SDK workers in Docker
containers, the additional pipeline dependencies (specified via
`--requirements_file` and other runtime options) are installed into the
containers at runtime. This can increase the worker startup time.
+ However, it may be possible to pre-build the SDK containers and perform the
dependency installation once before the workers start. To pre-build the
container image before pipeline submission, provide the pipeline options
mentioned below.
+1. Provide the container engine. Beam supports `local_docker`(requires local
installation of Docker) and `cloud_build`(requires a GCP project with Cloud
Build API enabled).
+
+ --prebuild_sdk_container_engine=<container_engine>
+2. To pass a base image for pre-building dependencies, provide
`--sdk_container_image`. If not, Apache beam's base
[image](https://hub.docker.com/search?q=apache%2Fbeam&type=image) would be used.
+
+ --sdk_container_image=<location_to_base_image>
+3. If using `local_docker` engine, provide a URL for the remote registry to
which the image will be pushed by passing
+
+ --docker_registry_push_url=<remote_registry_url>
+ # Example: --docker_registry_push_url=<registry_name>/beam
+ # pre-built image will be pushed to the
<registry_name>/beam/beam_python_prebuilt_sdk:<unique_image_tag>
+ # <unique_image_tag> tag is generated by Beam SDK.
+
+ **NOTE:** `docker_registry_push_url` must be a remote registry.
+> To use Docker, the `--sdk_container_image` should be compatible with Apache
Beam Runner. Please follow the
[instructions](https://beam.apache.org/documentation/runtime/environments/#building-and-pushing-custom-containers)
on how to build a base container image compatible with Apache Beam.
+
Review comment:
Suggestion to add to the notes:
The pre-building feature requires the Apache Beam SDK for Python, version
2.25.0 or later.
The container images created during prebuilding will persist beyond the
pipeline runtime.
Once your job is finished or stopped, you can remove the pre-built image
from the container registry.
If your pipeline is using a custom container image, most likely you will not
benefit from prebuilding step as extra dependencies can be preinstalled in the
custom image at build time. If you still would like to use prebuilding with
custom images, use Apache Beam SDK 2.38.0 or newer and supply your custom image
in via the `--sdk_container_image` pipeline option.
##########
File path:
website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md
##########
@@ -123,3 +134,25 @@ If your pipeline uses non-Python packages (e.g. packages
that require installati
--setup_file /path/to/setup.py
**Note:** Because custom commands execute after the dependencies for your
workflow are installed (by `pip`), you should omit the PyPI package dependency
from the pipeline's `requirements.txt` file and from the `install_requires`
parameter in the `setuptools.setup()` call of your `setup.py` file.
+
+## Pre-building SDK container image
+
+In pipeline execution modes where a Beam runner launches SDK workers in Docker
containers, the additional pipeline dependencies (specified via
`--requirements_file` and other runtime options) are installed into the
containers at runtime. This can increase the worker startup time.
+ However, it may be possible to pre-build the SDK containers and perform the
dependency installation once before the workers start. To pre-build the
container image before pipeline submission, provide the pipeline options
mentioned below.
+1. Provide the container engine. Beam supports `local_docker`(requires local
installation of Docker) and `cloud_build`(requires a GCP project with Cloud
Build API enabled).
+
+ --prebuild_sdk_container_engine=<container_engine>
+2. To pass a base image for pre-building dependencies, provide
`--sdk_container_image`. If not, Apache beam's base
[image](https://hub.docker.com/search?q=apache%2Fbeam&type=image) would be used.
Review comment:
As discussed offline, let's remove this and line 156.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
