This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v2-6-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 27ec3fb4196a935735f3bc05b9f7a7c10229e3c8 Author: Jarek Potiuk <[email protected]> AuthorDate: Mon May 22 00:23:59 2023 +0200 Add description in a few more places about adding airflow to pip install (#31448) It is important to keep airflow as a dependency with pinned version if you are adding new dependencies. This PR adds it in case of docker-compose documentation and also improves examples in our images, by using AIRFLOW_VERSION variable defined in the images. This way users can write future-proof ways of installind dependencies basing on our examples. Follow-up after #30813 (cherry picked from commit d722fba54e844b83025a7189b28371fba7e1af88) --- docker_tests/test_examples_of_prod_image_building.py | 1 - docs/apache-airflow/howto/docker-compose/index.rst | 11 ++++++++--- docs/apache-airflow/installation/installing-from-pypi.rst | 9 +++++++++ .../extending/add-build-essential-extend/Dockerfile | 2 +- .../docker-examples/extending/add-providers/Dockerfile | 2 +- .../docker-examples/extending/add-pypi-packages/Dockerfile | 2 +- .../extending/add-requirement-packages/Dockerfile | 2 +- .../docker-examples/extending/custom-providers/Dockerfile | 2 +- 8 files changed, 22 insertions(+), 9 deletions(-) diff --git a/docker_tests/test_examples_of_prod_image_building.py b/docker_tests/test_examples_of_prod_image_building.py index 178d62680a..f67efcd067 100644 --- a/docker_tests/test_examples_of_prod_image_building.py +++ b/docker_tests/test_examples_of_prod_image_building.py @@ -59,7 +59,6 @@ def test_dockerfile_example(dockerfile): new_content = re.sub( r"FROM apache/airflow:.*", rf"FROM apache/airflow:{latest_released_version}", content ) - new_content = re.sub(r"apache-airflow==\S*", rf"apache-airflow=={latest_released_version}", new_content) try: run_command( ["docker", "build", ".", "--tag", image_name, "-f", "-"], diff --git a/docs/apache-airflow/howto/docker-compose/index.rst b/docs/apache-airflow/howto/docker-compose/index.rst index b18f54097d..1ab56b2347 100644 --- a/docs/apache-airflow/howto/docker-compose/index.rst +++ b/docs/apache-airflow/howto/docker-compose/index.rst @@ -307,18 +307,23 @@ you should do those steps: to (use correct image tag): ``` -#image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.5.1} +#image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.6.1} build: . ``` 2) Create ``Dockerfile`` in the same folder your ``docker-compose.yaml`` file is with content similar to: ``` -FROM apache/airflow:2.5.1 +FROM apache/airflow:2.6.1 ADD requirements.txt . -RUN pip install -r requirements.txt +RUN pip install apache-airflow==${AIRFLOW_VERSION} -r requirements.txt ``` +It is the best practice to install apache-airflow in the same version as the one that comes from the +original image. This way you can be sure that ``pip`` will not try to downgrade or upgrade apache +airflow while installing other requirements, which might happen in case you try to add a dependency +that conflicts with the version of apache-airflow that you are using. + 3) Place ``requirements.txt`` file in the same directory. Run ``docker compose build`` to build the image, or add ``--build`` flag to ``docker compose up`` or diff --git a/docs/apache-airflow/installation/installing-from-pypi.rst b/docs/apache-airflow/installation/installing-from-pypi.rst index e6cf162bc3..97cf3ba41c 100644 --- a/docs/apache-airflow/installation/installing-from-pypi.rst +++ b/docs/apache-airflow/installation/installing-from-pypi.rst @@ -181,6 +181,15 @@ Airflow versions or other providers. Some providers have minimum-required versio versions of providers might have conflicting requirements with Airflow or other dependencies you might have installed. +It is the best practice to install apache-airflow in the same version as the one that comes from the +original image. This way you can be sure that ``pip`` will not try to downgrade or upgrade apache +airflow while installing other requirements, which might happen in case you try to add a dependency +that conflicts with the version of apache-airflow that you are using: + +.. code-block:: bash + + pip install "apache-airflow==|version|" "apache-airflow-providers-google==8.0.0" + Installation and upgrade of Airflow core ======================================== diff --git a/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile b/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile index 9aae369dc9..0a54cc4531 100644 --- a/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile +++ b/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile @@ -24,5 +24,5 @@ RUN apt-get update \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* USER airflow -RUN pip install --no-cache-dir mpi4py +RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" mpi4py # [END Dockerfile] diff --git a/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile b/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile index d625b56e04..63f8ce3e83 100644 --- a/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile +++ b/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile @@ -25,5 +25,5 @@ RUN apt-get update \ && rm -rf /var/lib/apt/lists/* USER airflow ENV JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 -RUN pip install --no-cache-dir apache-airflow-providers-apache-spark==2.1.3 apache-airflow==2.6.0dev0 +RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" apache-airflow-providers-apache-spark==2.1.3 # [END Dockerfile] diff --git a/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile b/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile index 4aded7f158..473f6d5988 100644 --- a/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile +++ b/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile @@ -16,5 +16,5 @@ # This is an example Dockerfile. It is not intended for PRODUCTION use # [START Dockerfile] FROM apache/airflow:2.6.1 -RUN pip install --no-cache-dir lxml apache-airflow==2.6.0.dev0 +RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" lxml # [END Dockerfile] diff --git a/docs/docker-stack/docker-examples/extending/add-requirement-packages/Dockerfile b/docs/docker-stack/docker-examples/extending/add-requirement-packages/Dockerfile index ec27e9808d..cb0b02739b 100644 --- a/docs/docker-stack/docker-examples/extending/add-requirement-packages/Dockerfile +++ b/docs/docker-stack/docker-examples/extending/add-requirement-packages/Dockerfile @@ -17,5 +17,5 @@ # [START Dockerfile] FROM apache/airflow:2.6.1 COPY requirements.txt / -RUN pip install --no-cache-dir -r /requirements.txt apache-airflow==2.6.0.dev0 +RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r /requirements.txt # [END Dockerfile] diff --git a/docs/docker-stack/docker-examples/extending/custom-providers/Dockerfile b/docs/docker-stack/docker-examples/extending/custom-providers/Dockerfile index 82eff95211..32a8480a30 100644 --- a/docs/docker-stack/docker-examples/extending/custom-providers/Dockerfile +++ b/docs/docker-stack/docker-examples/extending/custom-providers/Dockerfile @@ -16,5 +16,5 @@ # This is an example Dockerfile. It is not intended for PRODUCTION use # [START Dockerfile] FROM apache/airflow:2.6.1 -RUN pip install --no-cache-dir apache-airflow-providers-docker==2.5.1 apache-airflow==2.6.0.dev0 +RUN pip install "apache-airflow==${AIRFLOW_VERSION}" --no-cache-dir apache-airflow-providers-docker==2.5.1 # [END Dockerfile]
