This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 91b32ad2fd163f586e925e71e2f2be07eeb4a08e Author: Kaxil Naik <[email protected]> AuthorDate: Thu Dec 10 20:52:29 2020 +0000 Add changes from 1.10.14 (#12993) This commit adds Changelog, Updating.md and replaces 1.10.13 to 1.10.14 across the codebase (cherry picked from commit 969d3ea4f86a39807d696ca6f47c7cdf23526333) --- .github/workflows/ci.yml | 1 + BREEZE.rst | 117 ++++++++++++++++++++++++++++++++++++++++++++--- CONTRIBUTING.rst | 12 ----- IMAGES.rst | 67 +++++++++++++++++---------- README.md | 27 +++++++---- breeze | 3 +- breeze-complete | 14 +++--- 7 files changed, 181 insertions(+), 60 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ccfd50f..74670d3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -289,6 +289,7 @@ jobs: with: name: airflow-documentation path: "./files/documentation" + retention-days: 7 tests-helm: timeout-minutes: 20 diff --git a/BREEZE.rst b/BREEZE.rst index 43705f9..1ed3cfe 100644 --- a/BREEZE.rst +++ b/BREEZE.rst @@ -541,6 +541,74 @@ dependencies). If you work offline and do not want to rebuild the images when ne ``FORCE_ANSWER_TO_QUESTIONS`` variable to ``no`` as described in the `Setting default behaviour for user interaction <#setting-default-behaviour-for-user-interaction>`_ section. +Preparing packages +------------------ + +Breeze can also be used to prepare airflow packages - both "apache-airflow" main package and +provider packages. + +You can read more about testing provider packages in +`TESTING.rst <TESTING.rst#running-tests-with-packages>`_ + +There are several commands that you can run in Breeze to manage and build packages: + +* preparing Provider Readme files +* preparing Airflow packages +* preparing Provider packages + +Preparing provider readme files is part of the release procedure by the release managers +and it is described in detail in `dev <dev/README.md>`_ . + +You can prepare provider packages - by default regular provider packages are prepared, but with +``--backport`` flag you can prepare backport packages. + +The packages are prepared in ``dist`` folder. Note, that this command cleans up the ``dist`` folder +before running, so you should run it before generating airflow package below as it will be removed. + +The below example builds provider packages in the wheel format. + +.. code-block:: bash + + ./breeze prepare-provider-packages + +If you run this command without packages, you will prepare all packages, you can however specify +providers that you would like to build. By default only ``wheel`` packages are prepared, +but you can change it providing optional --package-format flag. + + +.. code-block:: bash + + ./breeze prepare-provider-packages --package-format=both google amazon + +You can also prepare backport provider packages, if you specify ``--backport`` flag. You can read more +about backport packages in `dev <dev/README.md>`_ + +.. code-block:: bash + + ./breeze prepare-provider-packages --backports --package-format=both google amazon + +You can see all providers available by running this command: + +.. code-block:: bash + + ./breeze prepare-provider-packages -- --help + + +You can also prepare airflow packages using breeze: + +.. code-block:: bash + + ./breeze prepare-airflow-packages + +This prepares airflow .whl package in the dist folder. + +Again, you can specify optional ``--package-format`` flag to build airflow packages. + +.. code-block:: bash + + ./breeze prepare-airflow-packages --package-format=bot + + Building Production images -------------------------- @@ -1067,6 +1135,7 @@ This is the current syntax for `./breeze <./breeze>`_: generate-constraints Generates pinned constraint files push-image Pushes images to registry initialize-local-virtualenv Initializes local virtualenv + prepare-airflow-packages Prepares airflow packages setup-autocomplete Sets up autocomplete for breeze start-airflow Starts Scheduler and Webserver and enters the shell stop Stops the docker-compose environment @@ -1142,7 +1211,7 @@ This is the current syntax for `./breeze <./breeze>`_: Detailed usage for command: build-docs - breeze build-docs [-- <EXTRA_ARGS>] + breeze build-docs Builds Airflow documentation. The documentation is build inside docker container - to maintain the same build environment for everyone. Appropriate sources are mapped from @@ -1150,8 +1219,6 @@ This is the current syntax for `./breeze <./breeze>`_: is generated ('docs/_build') are also mounted to the container - this way results of the documentation build is available in the host. - The possible extra args are: --docs-only, --spellcheck-only, --help - #################################################################################################### @@ -1187,8 +1254,7 @@ This is the current syntax for `./breeze <./breeze>`_: If specified, installs Airflow directly from PIP released version. This happens at image building time in production image and at container entering time for CI image. One of: - 1.10.13 1.10.12 1.10.11 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3 - 1.10.2 wheel none + 1.10.14 1.10.13 1.10.12 1.10.11 1.10.10 1.10.9 wheel none When 'none' is used, you can install airflow from local packages. When building image, airflow package should be added to 'docker-context-files' and @@ -1553,6 +1619,44 @@ This is the current syntax for `./breeze <./breeze>`_: #################################################################################################### + Detailed usage for command: prepare-airflow-packages + + + breeze prepare-airflow-packages [FLAGS] + + Prepares airflow packages (sdist and wheel) in dist folder. Note that + prepare-provider-packages command cleans up the dist folder, so if you want also + to generate provider packages, make sure you run prepare-provider-packages first, + and prepare-airflow-packages second. + + General form: + + 'breeze prepare-airflow-packages + + Flags: + + --package-format PACKAGE_FORMAT + + Chooses format of packages to prepare. + + One of: + + wheel,sdist,both + + Default: + + -v, --verbose + Show verbose information about executed docker, kind, kubectl, helm commands. Useful for + debugging - when you run breeze with --verbose flags you will be able to see the commands + executed under the hood and copy&paste them to your terminal to debug them more easily. + + Note that you can further increase verbosity and see all the commands executed by breeze + by running 'export VERBOSE_COMMANDS="true"' before running breeze. + + + #################################################################################################### + + Detailed usage for command: setup-autocomplete @@ -2105,8 +2209,7 @@ This is the current syntax for `./breeze <./breeze>`_: If specified, installs Airflow directly from PIP released version. This happens at image building time in production image and at container entering time for CI image. One of: - 1.10.13 1.10.12 1.10.11 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3 - 1.10.2 wheel none + 1.10.14 1.10.13 1.10.12 1.10.11 1.10.10 1.10.9 wheel none When 'none' is used, you can install airflow from local packages. When building image, airflow package should be added to 'docker-context-files' and diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index 8c4bf35..bc5661c 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -818,18 +818,6 @@ You can only run one of the steps via ``--spellcheck-only`` or ``--docs-only``. ./breeze build-docs -or just to run spell-check - -.. code-block:: bash - - ./breeze build-docs -- --spellcheck-only - -or just to run documentation building - -.. code-block:: bash - - ./breeze build-docs - Also documentation is available as downloadable artifact in GitHub Actions after the CI builds your PR. **Known issues:** diff --git a/IMAGES.rst b/IMAGES.rst index 513f8cf..94ad6fd 100644 --- a/IMAGES.rst +++ b/IMAGES.rst @@ -24,8 +24,8 @@ Airflow has two images (build from Dockerfiles): * Production image (Dockerfile) - that can be used to build your own production-ready Airflow installation You can read more about building and using the production image in the - `Production Deployments <docs/production-deployment.rst>`_ document. The image is built using - `Dockerfile <Dockerfile>`_ + `Production Deployments <https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html>`_ document. + The image is built using `Dockerfile <Dockerfile>`_ * CI image (Dockerfile.ci) - used for running tests and local development. The image is built using `Dockerfile.ci <Dockerfile.ci>`_ @@ -64,7 +64,7 @@ The easiest way to build those images is to use `<BREEZE.rst>`_. Note! Breeze by default builds production image from local sources. You can change it's behaviour by providing ``--install-airflow-version`` parameter, where you can specify the -tag/branch used to download Airflow package from in github repository. You can +tag/branch used to download Airflow package from in GitHub repository. You can also change the repository itself by adding ``--dockerhub-user`` and ``--dockerhub-repo`` flag values. You can build the CI image using this command: @@ -117,6 +117,16 @@ parameter to Breeze: ./breeze build-image --python 3.7 --additional-extras=presto \ --production-image --install-airflow-version=1.10.14 + +.. note:: + + On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver + does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice + of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4 + ``pip upgrade --pip==20.2.4`` or, in case you use Pip 20.3, you need to add option + ``--use-deprecated legacy-resolver`` to your pip install command. + + This will build the image using command similar to: .. code-block:: bash @@ -125,14 +135,6 @@ This will build the image using command similar to: apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv,presto]==1.10.14 \ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt"; -.. note:: - On 30th of November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. - This resolver does not yet work with Apache Airflow and might leads to errors in installation - - depends on your choice of extras. In order to install Airflow you need to either downgrade - pip to version 20.2.4 ``pip upgrade --pip==20.2.4`` or, in case you use Pip 20.3, you need to add option - ``--use-deprecated legacy-resolver`` to your pip install command. - - You can also build production images from specific Git version via providing ``--install-airflow-reference`` parameter to Breeze (this time constraints are taken from the ``constraints-master`` branch which is the HEAD of development for constraints): @@ -142,6 +144,15 @@ HEAD of development for constraints): pip install "https://github.com/apache/airflow/archive/<tag>.tar.gz#egg=apache-airflow" \ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-master/constraints-3.6.txt"; +You can also skip installing airflow by providing ``--install-airflow-version none`` parameter to Breeze: + +.. code-block:: bash + + ./breeze build-image --python 3.7 --additional-extras=presto \ + --production-image --install-airflow-version=none --install-from-local-files-when-building + +In this case you usually install airflow and all packages in ``docker-context-files`` folder. + Using cache during builds ========================= @@ -276,8 +287,8 @@ production image from the local sources. The image is primarily optimised for size of the final image, but also for speed of rebuilds - the 'airflow-build-image' segment uses the same technique as the CI builds for pre-installing PIP dependencies. -It first pre-installs them from the right github branch and only after that final airflow installation is -done from either local sources or remote location (PIP or github repository). +It first pre-installs them from the right GitHub branch and only after that final airflow installation is +done from either local sources or remote location (PIP or GitHub repository). Customizing the image ..................... @@ -409,7 +420,7 @@ The following build arguments (``--build-arg`` in docker build command) can be u | | | file has to be in docker context so | | | | it's best to place such file in | | | | one of the folders included in | -| | | dockerignore . for example in the | +| | | .dockerignore. for example in the | | | | 'docker-context-files'. Note that the | | | | location does not work for the first | | | | stage of installation when the | @@ -418,21 +429,31 @@ The following build arguments (``--build-arg`` in docker build command) can be u | | | set to true. Default location from | | | | GitHub is used in this case. | +------------------------------------------+------------------------------------------+------------------------------------------+ -| ``AIRFLOW_LOCAL_PIP_WHEELS`` | ``false`` | If set to true, Airflow and it's | +| ``AIRFLOW_CONSTRAINTS_REFERENCE`` | ``constraints-master`` | reference (branch or tag) from GitHub | +| | | repository from which constraints are | +| | | used. By default it is set to | +| | | ``constraints-master`` but can be | +| | | ``constraints-1-10`` for 1.10.* versions | +| | | or it could point to specific version | +| | | for example ``constraints-1.10.12`` | ++------------------------------------------+------------------------------------------+------------------------------------------+ +| ``INSTALL_FROM_DOCKER_CONTEXT_FILES`` | ``false`` | If set to true, Airflow and it's | | | | dependencies are installed from locally | | | | downloaded .whl files placed in the | | | | ``docker-context-files``. | +------------------------------------------+------------------------------------------+------------------------------------------+ | ``AIRFLOW_EXTRAS`` | ``all`` | extras to install | +------------------------------------------+------------------------------------------+------------------------------------------+ -| ``INSTALL_AIRFLOW_VIA_PIP`` | ``false`` | If set to true, Airflow is installed via | -| | | pip install. if you want to install | +| ``INSTALL_FROM_PYPI`` | ``true`` | If set to true, Airflow is installed | +| | | from pypi. If you want to install | | | | Airflow from externally provided binary | | | | package you can set it to false, place | | | | the package in ``docker-context-files`` | -| | | and set ``AIRFLOW_LOCAL_PIP_WHEELS`` to | -| | | true. You have to also set to true the | +| | | and set | +| | | ``INSTALL_FROM_DOCKER_CONTEXT_FILES`` to | +| | | true. For this you have to also set the | | | | ``AIRFLOW_PRE_CACHED_PIP_PACKAGES`` flag | +| | | to false | +------------------------------------------+------------------------------------------+------------------------------------------+ | ``AIRFLOW_PRE_CACHED_PIP_PACKAGES`` | ``true`` | Allows to pre-cache airflow PIP packages | | | | from the GitHub of Apache Airflow | @@ -530,7 +551,7 @@ Production images ----------------- You can find details about using, building, extending and customising the production images in the -`Latest documentation <https://airflow.readthedocs.io/en/latest/production-deployment.html>`_ +`Latest documentation <https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html>`_ Image manifests @@ -615,10 +636,8 @@ The entrypoint performs those operations: Using, customising, and extending the production image ====================================================== -You can read more about using, customising, and extending the production image in the documentation: - -* [Stable docs](https://airflow.apache.org/docs/stable/production-deployment.html) -* [Latest docs from master branch](https://airflow.readthedocs.io/en/latest/production-deployment.html +You can read more about using, customising, and extending the production image in the +`documentation <https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html>`_. Alpha versions of 1.10.10 production-ready images ================================================= diff --git a/README.md b/README.md index 79cceed..2d7edd6 100644 --- a/README.md +++ b/README.md @@ -82,18 +82,22 @@ Apache Airflow is tested with: | PostgreSQL | 9.6, 10, 11, 12, 13 | 9.6, 10, 11, 12, 13 | | MySQL | 5.7, 8 | 5.6, 5.7 | | SQLite | latest stable | latest stable | -| Kubernetes | 1.16.2, 1.17.0 | 1.16.2, 1.17.0 | +| Kubernetes | 1.16.9, 1.17.5, 1.18.6 | 1.16.9, 1.17.5, 1.18.6 | -**Note:** SQLite is used primarily for development purpose. +**Note:** MariaDB and MySQL 5.x are unable to or have limitations with +running multiple schedulers -- please see the "Scheduler" docs. + +**Note:** SQLite is used in Airflow tests. Do not use it in production. ### Additional notes on Python version requirements * Stable version [requires](https://github.com/apache/airflow/issues/8162) at least Python 3.5.3 when using Python 3 ## Getting started + Visit the official Airflow website documentation (latest **stable** release) for help with [installing Airflow](https://airflow.apache.org/installation.html), [getting started](https://airflow.apache.org/start.html), or walking through a more complete [tutorial](https://airflow.apache.org/tutorial.html). -> Note: If you're looking for documentation for master branch (latest development branch): you can find it on [ReadTheDocs](https://airflow.readthedocs.io/en/latest/). +> Note: If you're looking for documentation for master branch (latest development branch): you can find it on [s.apache.org/airflow-docs](https://s.apache.org/airflow-docs/). For more information on Airflow's Roadmap or Airflow Improvement Proposals (AIPs), visit the [Airflow Wiki](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Home). @@ -117,6 +121,15 @@ correct Airflow tag/version/branch and python versions in the URL. 1. Installing just Airflow: +NOTE!!! + +On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver +does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice +of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4 +`pip upgrade --pip==20.2.4` or, in case you use Pip 20.3, you need to add option +`--use-deprecated legacy-resolver` to your pip install command. + + ```bash pip install apache-airflow==1.10.14 \ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.7.txt"; @@ -130,14 +143,12 @@ depends on your choice of extras. In order to install Airflow you need to either pip to version 20.2.4 `pip upgrade --pip==20.2.4` or, in case you use Pip 20.3, you need to add option `--use-deprecated legacy-resolver` to your pip install command. - -2. Installing with extras (for example postgres,gcp) ```bash -pip install apache-airflow[postgres,gcp]==1.10.14 \ +pip install apache-airflow[postgres,google]==1.10.14 \ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.7.txt"; ``` -For information on installing backport providers check https://airflow.readthedocs.io/en/latest/backport-providers.html. +For information on installing backport providers check [/docs/backport-providers.rst][/docs/backport-providers.rst]. ## Official source code @@ -165,7 +176,7 @@ Those are - in the order of most common ways people install Airflow: - [Docker Images](https://hub.docker.com/repository/docker/apache/airflow) to install airflow via `docker` tool, use them in Kubernetes, Helm Charts, `docker-compose`, `docker swarm` etc. You can read more about using, customising, and extending the images in the - [Latest docs](https://airflow.readthedocs.io/en/latest/production-deployment.html), and + [Latest docs](https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html), and learn details on the internals in the [IMAGES.rst](IMAGES.rst) document. - [Tags in GitHub](https://github.com/apache/airflow/tags) to retrieve the git project sources that were used to generate official source packages via git diff --git a/breeze b/breeze index e9a9557..2565ad5 100755 --- a/breeze +++ b/breeze @@ -1567,7 +1567,7 @@ ${CMDNAME} exec [-- <EXTRA_ARGS>] webserver, workers, database console and interactive terminal. " export DETAILED_USAGE_BUILD_DOCS=" -${CMDNAME} build-docs [-- <EXTRA_ARGS>] +${CMDNAME} build-docs Builds Airflow documentation. The documentation is build inside docker container - to maintain the same build environment for everyone. Appropriate sources are mapped from @@ -1575,7 +1575,6 @@ ${CMDNAME} build-docs [-- <EXTRA_ARGS>] is generated ('docs/_build') are also mounted to the container - this way results of the documentation build is available in the host. - The possible extra args are: --docs-only, --spellcheck-only, --help " readonly DETAILED_USAGE_BUILD_DOCS export DETAILED_USAGE_BUILD_IMAGE=" diff --git a/breeze-complete b/breeze-complete index df81093..5502eec 100644 --- a/breeze-complete +++ b/breeze-complete @@ -34,6 +34,7 @@ _breeze_allowed_mysql_versions="5.6 5.7" _breeze_allowed_postgres_versions="9.6 10 11 12 13" _breeze_allowed_kind_operations="start stop restart status deploy test shell k9s" _breeze_allowed_test_types="All Core Integration Heisentests Postgres MySQL Helm" +_breeze_allowed_package_formats="wheel sdist both" # shellcheck disable=SC2034 { @@ -46,21 +47,16 @@ _breeze_allowed_test_types="All Core Integration Heisentests Postgres MySQL Helm _breeze_default_postgres_version=$(echo "${_breeze_allowed_postgres_versions}" | awk '{print $1}') _breeze_default_mysql_version=$(echo "${_breeze_allowed_mysql_versions}" | awk '{print $1}') _breeze_default_test_type=$(echo "${_breeze_allowed_test_types}" | awk '{print $1}') + _breeze_default_package_format=$(echo "${_breeze_allowed_package_formats}" | awk '{print $1}') } _breeze_allowed_install_airflow_versions=$(cat <<-EOF +1.10.14 1.10.13 1.10.12 1.10.11 1.10.10 1.10.9 -1.10.8 -1.10.7 -1.10.6 -1.10.5 -1.10.4 -1.10.3 -1.10.2 wheel none EOF @@ -162,6 +158,7 @@ exec generate-constraints push-image initialize-local-virtualenv +prepare-airflow-packages setup-autocomplete start-airflow stop @@ -265,6 +262,9 @@ function breeze_complete::get_known_values_breeze() { --test-type) _breeze_known_values="${_breeze_allowed_test_types}" ;; + --package-format) + _breeze_known_values="${_breeze_allowed_package_formats}" + ;; *) _breeze_known_values="" ;;
