This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new b8f68e3f4c1 Publishing website 2024/08/30 23:38:31 at commit 511f294
b8f68e3f4c1 is described below
commit b8f68e3f4c1ad5a246f0879f98bea4b229d3fb6d
Author: runner <runner@main-runner-xcffs-cclwh>
AuthorDate: Fri Aug 30 23:38:31 2024 +0000
Publishing website 2024/08/30 23:38:31 at commit 511f294
---
.../sdks/python-pipeline-dependencies/index.html | 45 ++++++++++------------
website/generated-content/sitemap.xml | 2 +-
2 files changed, 21 insertions(+), 26 deletions(-)
diff --git
a/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html
b/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html
index c277eb77353..4ca14b99453 100644
---
a/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html
+++
b/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html
@@ -54,34 +54,29 @@ RUN python -m pip install -r /tmp/requirements.txt
</code></pre><p>where package-name is the package’s tarball. You can
build the package tarball using a command line tool called <a
href=https://setuptools.pypa.io/en/latest/userguide/quickstart.html#install-build>build</a>.</p><pre><code>
# Install build using pip
pip install --upgrade build
python -m build --sdist
-</code></pre><p>See the <a
href=https://pypa-build.readthedocs.io/en/latest/index.html>build
documentation</a> for more details on this command.</p></li></ol></li></ol><h2
id=multiple-file-dependencies>Multiple File Dependencies</h2><p>Often, your
pipeline code spans multiple files. To run your project remotely, you must
group these files as a Python package and specify the package when you run your
pipeline. When the remote workers start, they will install your package. To
group your fi [...]
- requires = ["setuptools"]
- build-backend = "setuptools.build_meta"
+</code></pre><p>See the <a
href=https://pypa-build.readthedocs.io/en/latest/index.html>build
documentation</a> for more details on this command.</p></li></ol></li></ol><h2
id=multiple-file-dependencies>Multiple File Dependencies</h2><p>Often, your
pipeline code spans multiple files. To run your project remotely, you must
group these files as a Python package and specify the package when you run your
pipeline. When the remote workers start, they will install your package. To
group your fi [...]
- [project]
- name = "PACKAGE-NAME"
- version = "PACKAGE-VERSION"
- dependencies = [
- # List Python packages your pipeline depends on.
- ]
-</code></pre></li><li><p>If your package requires if some programmatic
configuration, or you need to use the <code>--setup_file</code> pipeline
option, create a setup.py file for your project.</p><pre><code> # Note that the
package can be completely defined by pyproject.toml.
- # This file is optional.
- import setuptools
- setuptools.setup()
-</code></pre></li><li><p>Structure your project so that the root directory
contains the <code>pyproject.toml</code>, the <code>setup.py</code> file, and a
<code>src/</code> directory with the rest of the files. For
example:</p><pre><code> root_dir/
- pyproject.toml
+ setuptools.setup(
+ name='PACKAGE-NAME',
+ version='PACKAGE-VERSION',
+ install_requires=[
+ # List Python packages your pipeline depends on.
+ ],
+ packages=setuptools.find_packages(),
+ )
+</code></pre></li><li><p>Structure your project so that the root directory
contains the <code>setup.py</code> file, the main workflow file, and a
directory with the rest of the files, for example:</p><pre><code> root_dir/
setup.py
- src/
- main.py
- my_package/
- my_pipeline_launcher.py
- my_custom_dofns_and_transforms.py
- other_utils_and_helpers.py
+ main.py
+ my_package/
+ my_pipeline_launcher.py
+ my_custom_dofns_and_transforms.py
+ other_utils_and_helpers.py
</code></pre><p>See <a
href=https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/complete/juliaset>Juliaset</a>
for an example that follows this project structure.</p></li><li><p>Install
your package in the submission environment, for example by using the following
command:</p><pre><code> pip install -e .
-</code></pre></li><li><p>If you use a <a href=#custom-containers>custom
container</a>, copy and install the package in the container as
well.</p></li><li><p>Run your pipeline with the following command-line
option:</p><pre><code> --setup_file /path/to/setup.py
-</code></pre></li></ol><p><strong>Note:</strong> It is not necessary to supply
the <code>--requirements_file</code> <a href=#pypi-dependencies>option</a> if
the dependencies of your package are defined in the
-<code>dependencies</code> field of the <code>pyproject.toml</code> file (see
step 1). However unlike with the <code>--requirements_file</code> option, when
you use the <code>--setup_file</code> option, Beam doesn’t stage the
dependent packages to the runner.
-Only the pipeline package is staged. If they aren’t already provided in
the runtime environment, the package dependencies are installed from PyPI at
runtime.</p><h2 id=nonpython>Non-Python Dependencies or PyPI Dependencies with
Non-Python Dependencies</h2><p>If your pipeline uses non-Python packages, such
as packages that require installation using the <code>apt install</code>
command, or uses a PyPI package that depends on non-Python dependencies during
package installation, we re [...]
+</code></pre></li><li><p>Run your pipeline with the following command-line
option:</p><pre><code> --setup_file /path/to/setup.py
+</code></pre></li></ol><p><strong>Note:</strong> It is not necessary to supply
the <code>--requirements_file</code> <a href=#pypi-dependencies>option</a> if
the dependencies of your package are defined in the
<code>install_requires</code> field of the <code>setup.py</code> file (see step
1).
+However unlike with the <code>--requirements_file</code> option, when you use
the <code>--setup_file</code> option, Beam doesn’t stage the dependent
packages to the runner.
+Only the pipeline package is staged. If they aren’t already provided in
the runtime environment,
+the package dependencies are installed from PyPI at runtime.</p><h2
id=nonpython>Non-Python Dependencies or PyPI Dependencies with Non-Python
Dependencies</h2><p>If your pipeline uses non-Python packages, such as packages
that require installation using the <code>apt install</code> command, or uses a
PyPI package that depends on non-Python dependencies during package
installation, we recommend installing them using a <a
href=#custom-containers>custom container</a>.
Otherwise, you must perform the following steps.</p><ol><li><p><a
href=#multiple-file-dependencies>Structure your pipeline as a
package</a>.</p></li><li><p>Add the required installation commands for the
non-Python dependencies, such as the <code>apt install</code> commands, to the
list of <code>CUSTOM_COMMANDS</code> in your <code>setup.py</code> file. See
the <a
href=https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/juliaset/setup.py>Juliaset
setup.py [...]
</code></pre></li></ol><p><strong>Note:</strong> Because custom commands
execute after the dependencies for your workflow are installed (by
<code>pip</code>), you should omit the PyPI package dependency from the
pipeline’s <code>requirements.txt</code> file and from the
<code>install_requires</code> parameter in the <code>setuptools.setup()</code>
call of your <code>setup.py</code> file.</p><h2
id=pre-building-sdk-container-image>Pre-building SDK Container Image</h2><p>In
pipeline [...]
However, it may be possible to pre-build the SDK containers and perform the
dependency installation once before the workers start with
<code>--prebuild_sdk_container_engine</code>. For instructions of how to use
pre-building with Google Cloud
diff --git a/website/generated-content/sitemap.xml
b/website/generated-content/sitemap.xml
index 4155b4faf36..e7fa96c61cc 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/blog/beam-2.58.1/</loc><lastmod>2024-08-30T11:03:03-04:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2024-08-30T11:03:03-04:00</lastmod></url><url><loc>/blog/</loc><lastmod>2024-08-30T11:03:03-04:00</lastmod></url><url><loc>/categories/</loc><lastmod>2024-08-30T11:03:03-04:00</lastmod></url><url><loc>/catego
[...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/blog/beam-2.58.1/</loc><lastmod>2024-08-30T11:42:49-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2024-08-30T11:42:49-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2024-08-30T11:42:49-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2024-08-30T11:42:49-07:00</lastmod></url><url><loc>/catego
[...]
\ No newline at end of file
