This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 659779fbf0341be0481752eefb7f52fe45096791 Author: Kaxil Naik <[email protected]> AuthorDate: Fri Nov 27 13:35:28 2020 +0000 Add 1.10.13 to CI, Breeze and Docs (#12652) (cherry picked from commit 9a74ee5fff6922543b7a3086969ca578d05c7417) --- BREEZE.rst | 8 +++---- IMAGES.rst | 18 +++++++-------- breeze-complete | 1 + docs/installation.rst | 10 ++++----- docs/production-deployment.rst | 50 +++++++++++++++++++++--------------------- 5 files changed, 44 insertions(+), 43 deletions(-) diff --git a/BREEZE.rst b/BREEZE.rst index 2c4bffbe..f91b598 100644 --- a/BREEZE.rst +++ b/BREEZE.rst @@ -1187,8 +1187,8 @@ This is the current syntax for `./breeze <./breeze>`_: If specified, installs Airflow directly from PIP released version. This happens at image building time in production image and at container entering time for CI image. One of: - 1.10.12 1.10.11 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3 1.10.2 - wheel + 1.10.13 1.10.12 1.10.11 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3 + 1.10.2 wheel -t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE If specified, installs Airflow directly from reference in GitHub. This happens at @@ -2098,8 +2098,8 @@ This is the current syntax for `./breeze <./breeze>`_: If specified, installs Airflow directly from PIP released version. This happens at image building time in production image and at container entering time for CI image. One of: - 1.10.12 1.10.11 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3 1.10.2 - wheel + 1.10.13 1.10.12 1.10.11 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3 + 1.10.2 wheel -t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE If specified, installs Airflow directly from reference in GitHub. This happens at diff --git a/IMAGES.rst b/IMAGES.rst index 13ce935..724d73c 100644 --- a/IMAGES.rst +++ b/IMAGES.rst @@ -39,7 +39,7 @@ The images are named as follows: where: -* ``BRANCH_OR_TAG`` - branch or tag used when creating the image. Examples: ``master``, ``v1-10-test``, ``1.10.12`` +* ``BRANCH_OR_TAG`` - branch or tag used when creating the image. Examples: ``master``, ``v1-10-test``, ``1.10.13`` The ``master`` and ``v1-10-test`` labels are built from branches so they change over time. The ``1.10.*`` and in the future ``2.*`` labels are build from git tags and they are "fixed" once built. * ``PYTHON_MAJOR_MINOR_VERSION`` - version of python used to build the image. Examples: ``3.5``, ``3.7`` @@ -115,15 +115,15 @@ parameter to Breeze: .. code-block:: bash ./breeze build-image --python 3.7 --additional-extras=presto \ - --production-image --install-airflow-version=1.10.12 + --production-image --install-airflow-version=1.10.13 This will build the image using command similar to: .. code-block:: bash pip install \ - apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv,presto]==1.10.12 \ - --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.6.txt"; + apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv,presto]==1.10.13 \ + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.13/constraints-3.6.txt"; You can also build production images from specific Git version via providing ``--install-airflow-reference`` parameter to Breeze (this time constraints are taken from the ``constraints-master`` branch which is the @@ -210,8 +210,8 @@ For example: apache/airflow:master-python3.6 - production "latest" image from current master apache/airflow:master-python3.6-ci - CI "latest" image from current master apache/airflow:v1-10-test-python2.7-ci - CI "latest" image from current v1-10-test branch - apache/airflow:1.10.12-python3.6 - production image for 1.10.12 release - apache/airflow:1.10.12-1-python3.6 - production image for 1.10.12 with some patches applied + apache/airflow:1.10.13-python3.6 - production image for 1.10.13 release + apache/airflow:1.10.13-1-python3.6 - production image for 1.10.13 with some patches applied You can see DockerHub images at `<https://hub.docker.com/repository/docker/apache/airflow>`_ @@ -292,7 +292,7 @@ additional apt dev and runtime dependencies. --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \ - --build-arg AIRFLOW_INSTALL_VERSION="==1.10.12" \ + --build-arg AIRFLOW_INSTALL_VERSION="==1.10.13" \ --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \ --build-arg AIRFLOW_SOURCES_FROM="empty" \ --build-arg AIRFLOW_SOURCES_TO="/empty" \ @@ -308,7 +308,7 @@ the same image can be built using ``breeze`` (it supports auto-completion of the .. code-block:: bash ./breeze build-image -f Dockerfile.ci \ - --production-image --python 3.7 --install-airflow-version=1.10.12 \ + --production-image --python 3.7 --install-airflow-version=1.10.13 \ --additional-extras=jdbc --additional-python-deps="pandas" \ --additional-dev-apt-deps="gcc g++" --additional-runtime-apt-deps="default-jre-headless" You can build the default production image with standard ``docker build`` command but they will only build @@ -326,7 +326,7 @@ based on example in `this comment <https://github.com/apache/airflow/issues/8605 --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \ - --build-arg AIRFLOW_INSTALL_VERSION="==1.10.12" \ + --build-arg AIRFLOW_INSTALL_VERSION="==1.10.13" \ --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \ --build-arg AIRFLOW_SOURCES_FROM="empty" \ --build-arg AIRFLOW_SOURCES_TO="/empty" \ diff --git a/breeze-complete b/breeze-complete index 94854ad..505c6bc 100644 --- a/breeze-complete +++ b/breeze-complete @@ -49,6 +49,7 @@ _breeze_allowed_test_types="All Core Integration Heisentests Postgres MySQL Helm } _breeze_allowed_install_airflow_versions=$(cat <<-EOF +1.10.13 1.10.12 1.10.11 1.10.10 diff --git a/docs/installation.rst b/docs/installation.rst index de1985c..12ce19e 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -31,7 +31,7 @@ if needed. This means that from time to time plain ``pip install apache-airflow` produce unusable Airflow installation. In order to have repeatable installation, however, starting from **Airflow 1.10.10** and updated in -**Airflow 1.10.12** we also keep a set of "known-to-be-working" constraint files in the +**Airflow 1.10.13** we also keep a set of "known-to-be-working" constraint files in the ``constraints-master`` and ``constraints-1-10`` orphan branches. Those "known-to-be-working" constraints are per major/minor python version. You can use them as constraint files when installing Airflow from PyPI. Note that you have to specify correct Airflow version @@ -47,22 +47,22 @@ and python versions in the URL. sudo apt-get install build-essential -1. Installing just airflow +1. Installing just Airflow .. code-block:: bash - AIRFLOW_VERSION=1.10.12 + AIRFLOW_VERSION=1.10.13 PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)" # For example: 3.6 CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"; - # For example: https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.6.txt + # For example: https://raw.githubusercontent.com/apache/airflow/constraints-1.10.13/constraints-3.6.txt pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}" 2. Installing with extras (for example postgres, google) .. code-block:: bash - AIRFLOW_VERSION=1.10.12 + AIRFLOW_VERSION=1.10.13 PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)" CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"; pip install "apache-airflow[postgres,google]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}" diff --git a/docs/production-deployment.rst b/docs/production-deployment.rst index 4bfaabb..3edddb8 100644 --- a/docs/production-deployment.rst +++ b/docs/production-deployment.rst @@ -64,7 +64,7 @@ You should be aware, about a few things: .. code-block:: dockerfile - FROM: apache/airflow:1.10.12 + FROM: apache/airflow:1.10.13 USER root RUN apt-get update \ && apt-get install -y --no-install-recommends \ @@ -81,7 +81,7 @@ You should be aware, about a few things: .. code-block:: dockerfile - FROM: apache/airflow:1.10.12 + FROM: apache/airflow:1.10.13 RUN pip install --no-cache-dir --user my-awesome-pip-dependency-to-add @@ -92,7 +92,7 @@ You should be aware, about a few things: .. code-block:: dockerfile - FROM: apache/airflow:1.10.12 + FROM: apache/airflow:1.10.13 USER root RUN apt-get update \ && apt-get install -y --no-install-recommends \ @@ -125,7 +125,7 @@ in the `<#production-image-build-arguments>`_ chapter below. Here just a few examples are presented which should give you general understanding of what you can customize. -This builds the production image in version 3.7 with additional airflow extras from 1.10.10 Pypi package and +This builds the production image in version 3.7 with additional airflow extras from 1.10.13 PyPI package and additional apt dev and runtime dependencies. .. code-block:: bash @@ -134,7 +134,7 @@ additional apt dev and runtime dependencies. --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \ - --build-arg AIRFLOW_INSTALL_VERSION="==1.10.12" \ + --build-arg AIRFLOW_INSTALL_VERSION="==1.10.13" \ --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \ --build-arg AIRFLOW_SOURCES_FROM="empty" \ --build-arg AIRFLOW_SOURCES_TO="/empty" \ @@ -150,7 +150,7 @@ the same image can be built using ``breeze`` (it supports auto-completion of the .. code-block:: bash ./breeze build-image \ - --production-image --python 3.7 --install-airflow-version=1.10.12 \ + --production-image --python 3.7 --install-airflow-version=1.10.13 \ --additional-extras=jdbc --additional-python-deps="pandas" \ --additional-dev-apt-deps="gcc g++" --additional-runtime-apt-deps="default-jre-headless" @@ -166,7 +166,7 @@ based on example in `this comment <https://github.com/apache/airflow/issues/8605 --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \ - --build-arg AIRFLOW_INSTALL_VERSION="==1.10.12" \ + --build-arg AIRFLOW_INSTALL_VERSION="==1.10.13" \ --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \ --build-arg AIRFLOW_SOURCES_FROM="empty" \ --build-arg AIRFLOW_SOURCES_TO="/empty" \ @@ -225,7 +225,7 @@ Preparing the constraint files and wheel files: pip download --dest docker-context-files \ --constraint docker-context-files/constraints-1-10.txt \ - apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv]==1.10.12 + apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv]==1.10.13 Building the image (after copying the files downloaded to the "docker-context-files" directory: @@ -233,7 +233,7 @@ Building the image (after copying the files downloaded to the "docker-context-fi .. code-block:: bash ./breeze build-image \ - --production-image --python 3.7 --install-airflow-version=1.10.12 \ + --production-image --python 3.7 --install-airflow-version=1.10.13 \ --disable-mysql-client-installation --disable-pip-cache --add-local-pip-wheels \ --constraints-location="/docker-context-files/constraints-1-10.txt" @@ -245,7 +245,7 @@ or --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \ - --build-arg AIRFLOW_INSTALL_VERSION="==1.10.12" \ + --build-arg AIRFLOW_INSTALL_VERSION="==1.10.13" \ --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \ --build-arg AIRFLOW_SOURCES_FROM="empty" \ --build-arg AIRFLOW_SOURCES_TO="/empty" \ @@ -392,7 +392,7 @@ The following build arguments (``--build-arg`` in docker build command) can be u | | | ``constraints-master`` but can be | | | | ``constraints-1-10`` for 1.10.* versions | | | | or it could point to specific version | -| | | for example ``constraints-1.10.12`` | +| | | for example ``constraints-1.10.13`` | +------------------------------------------+------------------------------------------+------------------------------------------+ | ``AIRFLOW_EXTRAS`` | (see Dockerfile) | Default extras with which airflow is | | | | installed | @@ -503,7 +503,7 @@ production image. There are three types of build: | ``AIRFLOW_INSTALL_VERSION`` | Optional - might be used for | | | package installation case to | | | set Airflow version for example | -| | "==1.10.12" | +| | "==1.10.13" | +-----------------------------------+-----------------------------------+ | ``AIRFLOW_CONSTRAINTS_REFERENCE`` | reference (branch or tag) from | | | GitHub where constraints file | @@ -512,7 +512,7 @@ production image. There are three types of build: | | ``constraints-1-10`` for 1.10.* | | | constraint or if you want to | | | point to specific version | -| | might be ``constraints-1.10.12`` | +| | might be ``constraints-1.10.13`` | +-----------------------------------+-----------------------------------+ | ``SLUGIFY_USES_TEXT_UNIDECODE`` | In case of of installing airflow | | | 1.10.2 or 1.10.1 you need to | @@ -546,7 +546,7 @@ of 2.0 currently): docker build . -This builds the production image in version 3.7 with default extras from 1.10.12 tag and +This builds the production image in version 3.7 with default extras from 1.10.13 tag and constraints taken from constraints-1-10-12 branch in GitHub. .. code-block:: bash @@ -554,14 +554,14 @@ constraints taken from constraints-1-10-12 branch in GitHub. docker build . \ --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ - --build-arg AIRFLOW_INSTALL_SOURCES="https://github.com/apache/airflow/archive/1.10.12.tar.gz#egg=apache-airflow"; \ + --build-arg AIRFLOW_INSTALL_SOURCES="https://github.com/apache/airflow/archive/1.10.13.tar.gz#egg=apache-airflow"; \ --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \ --build-arg AIRFLOW_BRANCH="v1-10-test" \ --build-arg AIRFLOW_SOURCES_FROM="empty" \ --build-arg AIRFLOW_SOURCES_TO="/empty" -This builds the production image in version 3.7 with default extras from 1.10.12 Pypi package and -constraints taken from 1.10.12 tag in GitHub and pre-installed pip dependencies from the top +This builds the production image in version 3.7 with default extras from 1.10.13 PyPI package and +constraints taken from 1.10.13 tag in GitHub and pre-installed pip dependencies from the top of v1-10-test branch. .. code-block:: bash @@ -570,14 +570,14 @@ of v1-10-test branch. --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \ - --build-arg AIRFLOW_INSTALL_VERSION="==1.10.12" \ + --build-arg AIRFLOW_INSTALL_VERSION="==1.10.13" \ --build-arg AIRFLOW_BRANCH="v1-10-test" \ - --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1.10.12" \ + --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1.10.13" \ --build-arg AIRFLOW_SOURCES_FROM="empty" \ --build-arg AIRFLOW_SOURCES_TO="/empty" -This builds the production image in version 3.7 with additional airflow extras from 1.10.12 Pypi package and -additional python dependencies and pre-installed pip dependencies from 1.10.12 tagged constraints. +This builds the production image in version 3.7 with additional airflow extras from 1.10.13 PyPI package and +additional python dependencies and pre-installed pip dependencies from 1.10.13 tagged constraints. .. code-block:: bash @@ -585,15 +585,15 @@ additional python dependencies and pre-installed pip dependencies from 1.10.12 t --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \ - --build-arg AIRFLOW_INSTALL_VERSION="==1.10.12" \ + --build-arg AIRFLOW_INSTALL_VERSION="==1.10.13" \ --build-arg AIRFLOW_BRANCH="v1-10-test" \ - --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1.10.12" \ + --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1.10.13" \ --build-arg AIRFLOW_SOURCES_FROM="empty" \ --build-arg AIRFLOW_SOURCES_TO="/empty" \ --build-arg ADDITIONAL_AIRFLOW_EXTRAS="mssql,hdfs" --build-arg ADDITIONAL_PYTHON_DEPS="sshtunnel oauth2client" -This builds the production image in version 3.7 with additional airflow extras from 1.10.12 Pypi package and +This builds the production image in version 3.7 with additional airflow extras from 1.10.13 PyPI package and additional apt dev and runtime dependencies. .. code-block:: bash @@ -602,7 +602,7 @@ additional apt dev and runtime dependencies. --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \ --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \ - --build-arg AIRFLOW_INSTALL_VERSION="==1.10.12" \ + --build-arg AIRFLOW_INSTALL_VERSION="==1.10.13" \ --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \ --build-arg AIRFLOW_SOURCES_FROM="empty" \ --build-arg AIRFLOW_SOURCES_TO="/empty" \
