This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a1e9d43f20f7b8f4734bf183519f4f4b6990a9f3 Author: Jarek Potiuk <[email protected]> AuthorDate: Mon Jun 28 14:35:12 2021 +0200 Add preparation of images as part of RC preparation process (#16674) (cherry picked from commit 98c12d49f37f6879e3e9fd926853f57a15ab761b) --- CI.rst | 16 ---------------- dev/README_RELEASE_AIRFLOW.md | 20 +++++++++++++++++--- scripts/ci/images/ci_build_dockerhub.sh | 5 ++++- scripts/ci/libraries/_build_images.sh | 4 ++-- 4 files changed, 23 insertions(+), 22 deletions(-) diff --git a/CI.rst b/CI.rst index 1ab37d1..4b92d2d 100644 --- a/CI.rst +++ b/CI.rst @@ -573,22 +573,6 @@ Those runs and their corresponding ``Build Images`` runs are only executed in ma repository, they are not executed in forks - we want to be nice to the contributors and not use their free build minutes on GitHub Actions. -Sometimes (bugs in DockerHub or prolonged periods when the scheduled builds are failing) -the automated build for nightly main is not executed for a long time. Such builds can be manually -prepared and pushed by a maintainer who has the rights to push images to DockerHub (committers need -to file JIRA ticket to Apache Infra in order to get an access). - -.. code-block:: bash - - export BRANCH=main - export DOCKER_REPO=docker.io/apache/airflow - for python_version in "3.6" "3.7" "3.8" - ( - export DOCKER_TAG=${BRANCH}-python${python_version} - ./scripts/ci/images/ci_build_dockerhub.sh - ) - - Workflows ========= diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md index 2d6a0b1..a281cda 100644 --- a/dev/README_RELEASE_AIRFLOW.md +++ b/dev/README_RELEASE_AIRFLOW.md @@ -22,6 +22,7 @@ - [Prepare the Apache Airflow Package RC](#prepare-the-apache-airflow-package-rc) - [Build RC artifacts](#build-rc-artifacts) + - [Manually prepare production Docker Image](#manually-prepare-production-docker-image) - [[\Optional\] Create new release branch](#%5Coptional%5C-create-new-release-branch) - [Prepare PyPI convenience "snapshot" packages](#prepare-pypi-convenience-snapshot-packages) - [Prepare production Docker Image](#prepare-production-docker-image) @@ -37,7 +38,7 @@ - [Publish release to SVN](#publish-release-to-svn) - [Prepare PyPI "release" packages](#prepare-pypi-release-packages) - [Update CHANGELOG.md](#update-changelogmd) - - [Manually prepare production Docker Image](#manually-prepare-production-docker-image) + - [Manually prepare production Docker Image](#manually-prepare-production-docker-image-1) - [Publish documentation](#publish-documentation) - [Notify developers of release](#notify-developers-of-release) - [Update Announcements page](#update-announcements-page) @@ -56,8 +57,9 @@ The Release Candidate artifacts we vote upon should be the exact ones we vote ag ```shell script # Set Version - export VERSION=2.0.2rc3 + export VERSION=2.1.2rc3 export VERSION_SUFFIX=rc3 + export VERSION_CONSTRAINT_BRANCH=2-1 export VERSION_WITHOUT_RC=${VERSION/rc?/} # Set AIRFLOW_REPO_ROOT to the path of your git repo @@ -105,7 +107,7 @@ The Release Candidate artifacts we vote upon should be the exact ones we vote ag - Tag & Push the latest constraints files. This pushes constraints with rc suffix (this is expected)! ```shell script - git checkout constraints-2-0 + git checkout constraints-${VERSION_CONSTRAINT_BRANCH} git tag -s "constraints-${VERSION}" git push origin "constraints-${VERSION}" ``` @@ -127,6 +129,18 @@ The Release Candidate artifacts we vote upon should be the exact ones we vote ag svn commit -m "Add artifacts for Airflow ${VERSION}" ``` + +## Manually prepare production Docker Image + + +```shell script +./scripts/ci/tools/prepare_prod_docker_images.sh ${VERSION} +``` + +This will wipe Breeze cache and docker-context-files in order to make sure the build is "clean". It +also performs image verification before pushing the images. + + ## [\Optional\] Create new release branch When you just released the `X.Y.0` version (first release of new minor version) you need to create release diff --git a/scripts/ci/images/ci_build_dockerhub.sh b/scripts/ci/images/ci_build_dockerhub.sh index 495295a..b464cf2 100755 --- a/scripts/ci/images/ci_build_dockerhub.sh +++ b/scripts/ci/images/ci_build_dockerhub.sh @@ -44,13 +44,16 @@ rm -rf "${AIRFLOW_SOURCES}/docker-context-files/*" build_images::prepare_prod_build build_images::build_prod_images verify_image::verify_prod_image "${AIRFLOW_PROD_IMAGE}" +echo +echo "Pushing airflow image as apache/airflow:${INSTALL_AIRFLOW_VERSION}-python${PYTHON_MAJOR_MINOR_VERSION}" +echo # Re-tag the image to be published in "apache/airflow" docker tag "apache/airflow-ci:${INSTALL_AIRFLOW_VERSION}-python${PYTHON_MAJOR_MINOR_VERSION}" \ "apache/airflow:${INSTALL_AIRFLOW_VERSION}-python${PYTHON_MAJOR_MINOR_VERSION}" docker push "apache/airflow:${INSTALL_AIRFLOW_VERSION}-python${PYTHON_MAJOR_MINOR_VERSION}" if [[ ${PYTHON_MAJOR_MINOR_VERSION} == "${DEFAULT_PYTHON_MAJOR_MINOR_VERSION}" ]]; then echo - echo "Pushing default airflow image" + echo "Pushing default airflow image as apache/airflow:${INSTALL_AIRFLOW_VERSION}" echo # In case of default Python version we also push ":version" tag docker tag "apache/airflow:${INSTALL_AIRFLOW_VERSION}-python${PYTHON_MAJOR_MINOR_VERSION}" \ diff --git a/scripts/ci/libraries/_build_images.sh b/scripts/ci/libraries/_build_images.sh index d9c77aa..0a7f6bf 100644 --- a/scripts/ci/libraries/_build_images.sh +++ b/scripts/ci/libraries/_build_images.sh @@ -743,8 +743,8 @@ function build_images::prepare_prod_build() { build_images::add_build_args_for_remote_install elif [[ -n "${INSTALL_AIRFLOW_VERSION=}" ]]; then # When --install-airflow-version is used then the image is build using released PIP package - # For PROD image only numeric versions are allowed - if [[ ! ${INSTALL_AIRFLOW_VERSION} =~ ^[0-9\.]*$ ]]; then + # For PROD image only numeric versions are allowed and RC candidates + if [[ ! ${INSTALL_AIRFLOW_VERSION} =~ ^[0-9\.]+(rc[0-9]+)?$ ]]; then echo echo "${COLOR_RED}ERROR: Bad value for install-airflow-version: '${INSTALL_AIRFLOW_VERSION}'. Only numerical versions allowed for PROD image here'!${COLOR_RESET}" echo
