This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new a09a8ec Add verbose error messages to container-related scripts.
(#16056)
a09a8ec is described below
commit a09a8ecbcb971c6779b687a517128260471d3a87
Author: tvalentyn <[email protected]>
AuthorDate: Thu Dec 16 13:22:44 2021 -0800
Add verbose error messages to container-related scripts. (#16056)
---
sdks/python/container/Dockerfile | 2 +-
sdks/python/container/run_generate_requirements.sh | 19 +++++++++++++++----
.../www/site/content/en/contribute/release-guide.md | 19 ++++++-------------
3 files changed, 22 insertions(+), 18 deletions(-)
diff --git a/sdks/python/container/Dockerfile b/sdks/python/container/Dockerfile
index b97cb3a..182d7bd 100644
--- a/sdks/python/container/Dockerfile
+++ b/sdks/python/container/Dockerfile
@@ -69,7 +69,7 @@ RUN ccache --set-config=sloppiness=file_macro && ccache
--set-config=hash_dir=fa
####
COPY target/apache-beam.tar.gz /opt/apache/beam/tars/
RUN pip install --no-deps -v /opt/apache/beam/tars/apache-beam.tar.gz[gcp]
-RUN pip check
+RUN pip check || (echo "Container does not include required Beam dependencies
or has conflicting dependencies. If Beam dependencies have changed, you need to
regenerate base_image_requirements.txt files. See:
https://s.apache.org/beam-python-requirements-generate"; && exit 1)
# Log complete list of what exact packages and versions are installed.
RUN pip freeze --all
diff --git a/sdks/python/container/run_generate_requirements.sh
b/sdks/python/container/run_generate_requirements.sh
index 9b533b7..a3b8f61 100755
--- a/sdks/python/container/run_generate_requirements.sh
+++ b/sdks/python/container/run_generate_requirements.sh
@@ -31,15 +31,26 @@
# https://s.apache.org/beam-python-dev-wiki
if [[ $# != 2 ]]; then
- printf "Usage: \n$> ./sdks/python/container/run_generate_requirements.sh
<python_version> <sdk_tarball>"
- printf "\n\tpython_version: [required] Python version to generate
dependencies for."
- printf " Use 3.7 for Python3.7, 3.8 for Python3.8 etc."
+ printf "Example usage: \n$>
./sdks/python/container/run_generate_requirements.sh 3.8 <sdk_tarball>"
+ printf "\n\twhere 3.8 is the Python major.minor version."
+ exit 1
fi
-set -ex
PY_VERSION=$1
SDK_TARBALL=$2
+if ! python$PY_VERSION --version > /dev/null 2>&1 ; then
+ echo "Please install a python${PY_VERSION} interpreter. See
s.apache.org/beam-python-dev-wiki for Python installation tips."
+ exit 1
+fi
+
+if ! python$PY_VERSION -m venv --help > /dev/null 2>&1 ; then
+ echo "Your python${PY_VERSION} installation does not have a required venv
module. See s.apache.org/beam-python-dev-wiki for Python installation tips."
+ exit 1
+fi
+
+set -ex
+
ENV_PATH="$PWD/build/python${PY_VERSION/./}_requirements_gen"
rm -rf $ENV_PATH 2>/dev/null || true
python${PY_VERSION} -m venv $ENV_PATH
diff --git a/website/www/site/content/en/contribute/release-guide.md
b/website/www/site/content/en/contribute/release-guide.md
index 370a572..727c94b 100644
--- a/website/www/site/content/en/contribute/release-guide.md
+++ b/website/www/site/content/en/contribute/release-guide.md
@@ -233,17 +233,10 @@ If you are not a PMC, please ask for help in dev@ mailing
list.
## 3. Update base image dependencies for Python container images
-1. Check the versions specified in
sdks/python/container/base_image_requirements_manual.txt` and update them if
necessary.
-1. Regenerate full dependency list by running:
-`./gradlew :sdks:python:container:generatePythonRequirementsAll` and commiting
-the changes. Exectution takes about ~5 min per Python version and is somewhat
resource-demanding.
-You can also regenerate the dependencies indiviually per version with targets
like `./gradlew :sdks:python:container:py38:generatePythonRequirements`.
+See instructions at: https://s.apache.org/beam-python-requirements-generate
-
-Ideally, do this at least a week before the release cut, so that any issues
+Ideally, do the update at least a week before the release cut, so that any
issues
related to the update have time to surface.
-You will need Python intepreters for all versions supported by Beam, see:
-https://s.apache.org/beam-python-dev-wiki for tips how to install them.
## 4. Investigate performance regressions
@@ -589,12 +582,12 @@ See the source of the script for more details, or to run
commands manually in ca
Please note that dependencies for the SDKs with different Python
versions vary.
Need to verify all Python images by replacing `${ver}` with each
supported Python version `X.Y`.
```
- docker run -it --entrypoint=/bin/bash
apache/beam_python${ver}_sdk:${RELEASE}_rc{RC_NUM}
+ docker run --rm -it --entrypoint=/bin/bash
apache/beam_python${ver}_sdk:${RELEASE}_rc{RC_NUM}
ls -al /opt/apache/beam/third_party_licenses/ | wc -l
```
- For Java SDK images, there should be around 200 dependencies.
```
- docker run -it --entrypoint=/bin/bash
apache/beam_java${ver}_sdk:${RELEASE}_rc{RC_NUM}
+ docker run --rm -it --entrypoint=/bin/bash
apache/beam_java${ver}_sdk:${RELEASE}_rc{RC_NUM}
ls -al /opt/apache/beam/third_party_licenses/ | wc -l
```
1. Publish staging artifacts
@@ -785,8 +778,8 @@ Here’s an email template; please adjust as you see fit.
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "v1.2.3-RC3" [5],
* website pull request listing the release [6], the blog post [6], and
publishing the API reference manual [7].
- * Java artifacts were built with Maven MAVEN_VERSION and OpenJDK/Oracle
JDK JDK_VERSION.
- * Python artifacts are deployed along with the source release to the
dist.apache.org [2] and pypy[8].
+ * Java artifacts were built with Gradle GRADLE_VERSION and OpenJDK/Oracle
JDK JDK_VERSION.
+ * Python artifacts are deployed along with the source release to the
dist.apache.org [2] and PyPI[8].
* Validation sheet with a tab for 1.2.3 release to help with validation
[9].
* Docker images published to Docker Hub [10].
