jedcunningham commented on a change in pull request #22492: URL: https://github.com/apache/airflow/pull/22492#discussion_r833767053
########## File path: docs/docker-stack/build.rst ########## @@ -345,6 +351,44 @@ The following example adds ``test_dag.py`` to your image in the ``/opt/airflow/d Customizing the image --------------------- +.. warning:: + BREAKING CHANGE! As of Airflow 2.3.0 you need to use + `Buildkit <https://docs.docker.com/develop/develop-images/build_enhancements/>`_ to build customized + Airflow Docker image. We are using new features of Building (and ``dockerfile:1.4`` syntax) + to make our image faster to build and ``standalone`` - i.e. not needing any extra files from + Airflow in order to be build. As of Airflow 2.3.0, the ``Dockerfile`` that is released with Airflow + does not need any extra folders or files and can be copied and used from any folder. + Previously you needed to copy Airflow sources together with the Dockerfile as some scripts were + needed to make it works. You also need to use ``DOCKER_CONTEXT_FILES`` build arg if you want to + use your own custom files during the build (see + :ref:`Using docker context files <using-docker-context-files>` for details) + +.. note:: + You can usually use the latest ``Dockerfile`` released by Airflow to build previous Airflow versions. + Note however, that there are slight changes in the Dockerfile and entrypoint scripts that can make it + behaves slightly differently, depending which Dockerfile version you used. Details of what has changed + in each of the released versions of Docker image can be found in the :doc:`Changelog <changelog>`. + +Prerequisites for building customized docker image: + +* You need to enable `Buildkit <https://docs.docker.com/develop/develop-images/build_enhancements/>`_ to + build the image. This can be done by setting ``DOCKER_BUILDKIT=1`` as an environment variable + or by installing `the buildx plugin <https://docs.docker.com/buildx/working-with-buildx/>`_ + and running ``docker buildx build`` command. + +* You need to have a new Docker installed to handle ``1.4`` syntax of the Dockerfile. + Docker version ``20.10.7`` and above is known to work. + +Before attempting to customize the image, you need to download flexible and customizable ``Airflow Dockerfile`` +You can extract the officially released version of the Dockerfile from the +`released sources <https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-sources.html>`_ +You can also conveniently download the latest released version +`from GitHub <https://raw.githubusercontent.com/apache/airflow/|version|/Dockerfile>`_. You can save it +in any directory - there is no need for any other files to be present there. If you wish to use your own +files (for example custom configuration of ``pip`` or your own ``requirements`` or custom dependencies, +you need to use ``DOCKER_CONTEXT_FILES`` build arg and place the files in the directory pointed at by +the arg (see :ref:`Using docker context files <using-docker-context-files>` for details) Review comment: ```suggestion the arg (see :ref:`Using docker context files <using-docker-context-files>` for details). ``` ########## File path: docs/docker-stack/build.rst ########## @@ -345,6 +351,44 @@ The following example adds ``test_dag.py`` to your image in the ``/opt/airflow/d Customizing the image --------------------- +.. warning:: + BREAKING CHANGE! As of Airflow 2.3.0 you need to use + `Buildkit <https://docs.docker.com/develop/develop-images/build_enhancements/>`_ to build customized + Airflow Docker image. We are using new features of Building (and ``dockerfile:1.4`` syntax) + to make our image faster to build and ``standalone`` - i.e. not needing any extra files from + Airflow in order to be build. As of Airflow 2.3.0, the ``Dockerfile`` that is released with Airflow + does not need any extra folders or files and can be copied and used from any folder. + Previously you needed to copy Airflow sources together with the Dockerfile as some scripts were + needed to make it works. You also need to use ``DOCKER_CONTEXT_FILES`` build arg if you want to + use your own custom files during the build (see + :ref:`Using docker context files <using-docker-context-files>` for details) Review comment: ```suggestion :ref:`Using docker context files <using-docker-context-files>` for details). ``` ########## File path: docs/docker-stack/changelog.rst ########## @@ -0,0 +1,174 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Dockerfile Changelog +==================== + +he ``Dockerfile`` does not strictly follow the `SemVer <https://semver.org/>`_ approach of Review comment: ```suggestion The ``Dockerfile`` does not strictly follow the `SemVer <https://semver.org/>`_ approach of ``` ########## File path: docs/docker-stack/changelog.rst ########## @@ -0,0 +1,174 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Dockerfile Changelog +==================== + +he ``Dockerfile`` does not strictly follow the `SemVer <https://semver.org/>`_ approach of +Apache Airflow when it comes to features and backwards compatibility. While Airflow code strictly +follows it, the ``Dockerfile`` is really a way to conveniently package Airflow using standard container +approach, occasionally there are some changes in the building process or in the entrypoint of the image +that require slight adaptation. + +The Changelog below describes the changes introduced in each version of the docker images released by Review comment: ```suggestion The changelog below describes the changes introduced in each version of the docker images released by ``` ########## File path: docs/docker-stack/build.rst ########## @@ -428,6 +469,57 @@ right version of python base image: * ``--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster`` uses buster version of Debian (Debian 10) * ``--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-bullseye`` uses bullseye version of Debian (Debian 11) +.. _using-docker-context-files: + +Using docker-context-files +-------------------------- + +When customizing the image, you can optionally make Airflow install custom binaries or provider custom +configuration for your pip in ``docker-context-files``. In order to enable it, you need to add +``--build-arg DOCKER_CONTEXT_FILES=docker-context-files`` build arg when you build the image. +You can pass any subdirectory of your docker context, it will always be mapped to ``/docker-context-files`` +during the build. + +You can use ``docker-context-files`` for the following purposes: + +* you can place ``requirements.txt`` in the ``docker-context-file`` and additional ``pip`` packages + specified there will be installed during the build. + +* you can place ``pip.conf`` and ``.piprc`` in the ``docker-context-files`` folder and they will be used + for all ``pip`` commands (for example you can configure your own sources or authentication mechanisms) + +* you can place .whl packages that you downloaded and install them with ``INSTALL_DOCKER_CONTEXT_FILES`` set + to ``true`` ( see: :ref:`image-build-secure-environments`). It's useful if you build the image in + restricted security environments + +Example command on how you can use the ``docker-context-files`` to add your own dependencies from the +``requirements.txt``: + +.. exampleinclude:: docker-examples/customizing/own-requirements.sh + :language: bash + :start-after: [START build] + :end-before: [END build] + +.. note:: + You can also pass ``--build-arg DOCKER_CONTEXT_FILES=.`` if you want to place your ``requirements.txt`` + in main directory without creating a dedicated folder, however this is a good practice to keep any files + that you copy to the image context in a sub-folder. This makes it easier to separate things that + are used on the host from those that are passed to the Docker. Of course, by default when you run + ``docker build .`` the whole folder is available as "Docker build context" and sent to the docker + engine, but the ``DOCKER_CONTEXT_FILES`` are always copied to the ``build`` segment of the image so + copying all your local folder might unnecessarily increase time needed to build the image and your + cache will be invalidated every time any of the files in your local folder change. + +.. warning:: + BREAKING CHANGE! As of Airflow 2.3.0 you need to specify additional flag: + ``--build-arg DOCKER_CONTEXT_Files=docker-context-files`` in order to use the files placed + in ``docker-context-files``. Previously that switch was not needed. Unfortunately this change is needed + in order to enable ``Dockerfile`` as standalone Dockerfile without any extra files. As of Airflow 2.3.0 + the ``Dockerfile`` that is released with Airflow does not need any extra folders or files and can + be copied and used from any folder. Previously you needed to copy Airflow sources together with the + Dockerfile as some scripts were needed to make it works. With Airflow 2.3.0, we are using ``Buildkit`` Review comment: ```suggestion Dockerfile as some scripts were needed to make it work. With Airflow 2.3.0, we are using ``Buildkit`` ``` ########## File path: docs/docker-stack/changelog.rst ########## @@ -0,0 +1,174 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Dockerfile Changelog +==================== + +he ``Dockerfile`` does not strictly follow the `SemVer <https://semver.org/>`_ approach of +Apache Airflow when it comes to features and backwards compatibility. While Airflow code strictly +follows it, the ``Dockerfile`` is really a way to conveniently package Airflow using standard container +approach, occasionally there are some changes in the building process or in the entrypoint of the image +that require slight adaptation. + +The Changelog below describes the changes introduced in each version of the docker images released by +the Airflow team. + +:note: The changelog below concerns only the convenience production images released at + `Airflow DockerHub <https://hub.docker.com/r/apache/airflow>`_ . The images that are released + There are usually built using the ``Dockerfile`` released together with Airflow. However You are + free to take latest released ``Dockerfile`` from Airflow and use it to build an image for + any Airflow version from ``Airflow 2`` line. There is no guarantee that it works, but if it does + then you can use latest features from that image to build the previous Airflow versions. + +:warning: Some of the images below (as noted in the Changelog) have been regenerated using newer + ``Dockerfiles``. This happens when there is a breaking change that invalidates already released images + and the images need regeneration. This has happened already when MySQL changed the keys they + used to release their packages: `Issue here <https://github.com/apache/airflow/issues/20911>`_ + and 2.1 images are all regenerated using the 2.2 ``Dockerfile``. This is a rare event and + we do it only when we have no choice because of external factors. In such case, newer version of Review comment: ```suggestion we do it only when we have no choice because of external factors. In such case, the newer version of ``` ########## File path: docs/docker-stack/build.rst ########## @@ -428,6 +469,57 @@ right version of python base image: * ``--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster`` uses buster version of Debian (Debian 10) * ``--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-bullseye`` uses bullseye version of Debian (Debian 11) +.. _using-docker-context-files: + +Using docker-context-files +-------------------------- + +When customizing the image, you can optionally make Airflow install custom binaries or provider custom +configuration for your pip in ``docker-context-files``. In order to enable it, you need to add +``--build-arg DOCKER_CONTEXT_FILES=docker-context-files`` build arg when you build the image. +You can pass any subdirectory of your docker context, it will always be mapped to ``/docker-context-files`` +during the build. + +You can use ``docker-context-files`` for the following purposes: + +* you can place ``requirements.txt`` in the ``docker-context-file`` and additional ``pip`` packages Review comment: ```suggestion * you can place ``requirements.txt`` in the ``docker-context-file`` folder and any additional ``pip`` packages ``` ########## File path: docs/docker-stack/build.rst ########## @@ -345,6 +351,44 @@ The following example adds ``test_dag.py`` to your image in the ``/opt/airflow/d Customizing the image --------------------- +.. warning:: + BREAKING CHANGE! As of Airflow 2.3.0 you need to use + `Buildkit <https://docs.docker.com/develop/develop-images/build_enhancements/>`_ to build customized + Airflow Docker image. We are using new features of Building (and ``dockerfile:1.4`` syntax) + to make our image faster to build and ``standalone`` - i.e. not needing any extra files from Review comment: ```suggestion to make our image faster to build and "standalone" - i.e. not needing any extra files from ``` ########## File path: docs/docker-stack/build.rst ########## @@ -428,6 +469,57 @@ right version of python base image: * ``--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster`` uses buster version of Debian (Debian 10) * ``--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-bullseye`` uses bullseye version of Debian (Debian 11) +.. _using-docker-context-files: + +Using docker-context-files +-------------------------- + +When customizing the image, you can optionally make Airflow install custom binaries or provider custom +configuration for your pip in ``docker-context-files``. In order to enable it, you need to add +``--build-arg DOCKER_CONTEXT_FILES=docker-context-files`` build arg when you build the image. +You can pass any subdirectory of your docker context, it will always be mapped to ``/docker-context-files`` +during the build. + +You can use ``docker-context-files`` for the following purposes: + +* you can place ``requirements.txt`` in the ``docker-context-file`` and additional ``pip`` packages + specified there will be installed during the build. + +* you can place ``pip.conf`` and ``.piprc`` in the ``docker-context-files`` folder and they will be used + for all ``pip`` commands (for example you can configure your own sources or authentication mechanisms) + +* you can place .whl packages that you downloaded and install them with ``INSTALL_DOCKER_CONTEXT_FILES`` set Review comment: ```suggestion * you can place ``.whl`` packages that you downloaded and install them with ``INSTALL_DOCKER_CONTEXT_FILES`` set ``` ########## File path: docs/docker-stack/build.rst ########## @@ -345,6 +351,44 @@ The following example adds ``test_dag.py`` to your image in the ``/opt/airflow/d Customizing the image --------------------- +.. warning:: + BREAKING CHANGE! As of Airflow 2.3.0 you need to use + `Buildkit <https://docs.docker.com/develop/develop-images/build_enhancements/>`_ to build customized + Airflow Docker image. We are using new features of Building (and ``dockerfile:1.4`` syntax) + to make our image faster to build and ``standalone`` - i.e. not needing any extra files from + Airflow in order to be build. As of Airflow 2.3.0, the ``Dockerfile`` that is released with Airflow + does not need any extra folders or files and can be copied and used from any folder. + Previously you needed to copy Airflow sources together with the Dockerfile as some scripts were + needed to make it works. You also need to use ``DOCKER_CONTEXT_FILES`` build arg if you want to + use your own custom files during the build (see + :ref:`Using docker context files <using-docker-context-files>` for details) + +.. note:: + You can usually use the latest ``Dockerfile`` released by Airflow to build previous Airflow versions. + Note however, that there are slight changes in the Dockerfile and entrypoint scripts that can make it + behaves slightly differently, depending which Dockerfile version you used. Details of what has changed Review comment: ```suggestion behave slightly differently, depending which Dockerfile version you used. Details of what has changed ``` ########## File path: docs/docker-stack/build.rst ########## @@ -345,6 +351,44 @@ The following example adds ``test_dag.py`` to your image in the ``/opt/airflow/d Customizing the image --------------------- +.. warning:: + BREAKING CHANGE! As of Airflow 2.3.0 you need to use + `Buildkit <https://docs.docker.com/develop/develop-images/build_enhancements/>`_ to build customized + Airflow Docker image. We are using new features of Building (and ``dockerfile:1.4`` syntax) + to make our image faster to build and ``standalone`` - i.e. not needing any extra files from + Airflow in order to be build. As of Airflow 2.3.0, the ``Dockerfile`` that is released with Airflow + does not need any extra folders or files and can be copied and used from any folder. + Previously you needed to copy Airflow sources together with the Dockerfile as some scripts were + needed to make it works. You also need to use ``DOCKER_CONTEXT_FILES`` build arg if you want to + use your own custom files during the build (see + :ref:`Using docker context files <using-docker-context-files>` for details) + +.. note:: + You can usually use the latest ``Dockerfile`` released by Airflow to build previous Airflow versions. + Note however, that there are slight changes in the Dockerfile and entrypoint scripts that can make it + behaves slightly differently, depending which Dockerfile version you used. Details of what has changed + in each of the released versions of Docker image can be found in the :doc:`Changelog <changelog>`. + +Prerequisites for building customized docker image: + +* You need to enable `Buildkit <https://docs.docker.com/develop/develop-images/build_enhancements/>`_ to + build the image. This can be done by setting ``DOCKER_BUILDKIT=1`` as an environment variable + or by installing `the buildx plugin <https://docs.docker.com/buildx/working-with-buildx/>`_ + and running ``docker buildx build`` command. + +* You need to have a new Docker installed to handle ``1.4`` syntax of the Dockerfile. + Docker version ``20.10.7`` and above is known to work. + +Before attempting to customize the image, you need to download flexible and customizable ``Airflow Dockerfile`` Review comment: ```suggestion Before attempting to customize the image, you need to download the flexible and customizable Airflow ``Dockerfile``. ``` ########## File path: docs/docker-stack/build.rst ########## @@ -345,6 +351,44 @@ The following example adds ``test_dag.py`` to your image in the ``/opt/airflow/d Customizing the image --------------------- +.. warning:: + BREAKING CHANGE! As of Airflow 2.3.0 you need to use + `Buildkit <https://docs.docker.com/develop/develop-images/build_enhancements/>`_ to build customized + Airflow Docker image. We are using new features of Building (and ``dockerfile:1.4`` syntax) + to make our image faster to build and ``standalone`` - i.e. not needing any extra files from + Airflow in order to be build. As of Airflow 2.3.0, the ``Dockerfile`` that is released with Airflow + does not need any extra folders or files and can be copied and used from any folder. + Previously you needed to copy Airflow sources together with the Dockerfile as some scripts were + needed to make it works. You also need to use ``DOCKER_CONTEXT_FILES`` build arg if you want to + use your own custom files during the build (see + :ref:`Using docker context files <using-docker-context-files>` for details) + +.. note:: + You can usually use the latest ``Dockerfile`` released by Airflow to build previous Airflow versions. + Note however, that there are slight changes in the Dockerfile and entrypoint scripts that can make it + behaves slightly differently, depending which Dockerfile version you used. Details of what has changed + in each of the released versions of Docker image can be found in the :doc:`Changelog <changelog>`. + +Prerequisites for building customized docker image: + +* You need to enable `Buildkit <https://docs.docker.com/develop/develop-images/build_enhancements/>`_ to + build the image. This can be done by setting ``DOCKER_BUILDKIT=1`` as an environment variable + or by installing `the buildx plugin <https://docs.docker.com/buildx/working-with-buildx/>`_ + and running ``docker buildx build`` command. + +* You need to have a new Docker installed to handle ``1.4`` syntax of the Dockerfile. + Docker version ``20.10.7`` and above is known to work. + +Before attempting to customize the image, you need to download flexible and customizable ``Airflow Dockerfile`` +You can extract the officially released version of the Dockerfile from the +`released sources <https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-sources.html>`_ Review comment: ```suggestion `released sources <https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-sources.html>`_. ``` ########## File path: docs/docker-stack/changelog.rst ########## @@ -0,0 +1,174 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Dockerfile Changelog +==================== + +he ``Dockerfile`` does not strictly follow the `SemVer <https://semver.org/>`_ approach of +Apache Airflow when it comes to features and backwards compatibility. While Airflow code strictly +follows it, the ``Dockerfile`` is really a way to conveniently package Airflow using standard container +approach, occasionally there are some changes in the building process or in the entrypoint of the image Review comment: ```suggestion approach, so occasionally there are some changes in the building process or in the entrypoint of the image ``` ########## File path: docs/docker-stack/changelog.rst ########## @@ -0,0 +1,174 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Dockerfile Changelog +==================== + +he ``Dockerfile`` does not strictly follow the `SemVer <https://semver.org/>`_ approach of +Apache Airflow when it comes to features and backwards compatibility. While Airflow code strictly +follows it, the ``Dockerfile`` is really a way to conveniently package Airflow using standard container Review comment: ```suggestion follows it, the ``Dockerfile`` is really a way to conveniently packaged Airflow using standard container ``` ########## File path: docs/docker-stack/build.rst ########## @@ -428,6 +469,57 @@ right version of python base image: * ``--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster`` uses buster version of Debian (Debian 10) * ``--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-bullseye`` uses bullseye version of Debian (Debian 11) +.. _using-docker-context-files: + +Using docker-context-files +-------------------------- + +When customizing the image, you can optionally make Airflow install custom binaries or provider custom Review comment: ```suggestion When customizing the image, you can optionally make Airflow install custom binaries or provide custom ``` ########## File path: docs/docker-stack/changelog.rst ########## @@ -0,0 +1,174 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Dockerfile Changelog +==================== + +he ``Dockerfile`` does not strictly follow the `SemVer <https://semver.org/>`_ approach of +Apache Airflow when it comes to features and backwards compatibility. While Airflow code strictly +follows it, the ``Dockerfile`` is really a way to conveniently package Airflow using standard container +approach, occasionally there are some changes in the building process or in the entrypoint of the image +that require slight adaptation. + +The Changelog below describes the changes introduced in each version of the docker images released by +the Airflow team. + +:note: The changelog below concerns only the convenience production images released at + `Airflow DockerHub <https://hub.docker.com/r/apache/airflow>`_ . The images that are released + There are usually built using the ``Dockerfile`` released together with Airflow. However You are Review comment: ```suggestion there are usually built using the ``Dockerfile`` released together with Airflow. However you are ``` ########## File path: docs/docker-stack/build.rst ########## @@ -345,6 +351,44 @@ The following example adds ``test_dag.py`` to your image in the ``/opt/airflow/d Customizing the image --------------------- +.. warning:: + BREAKING CHANGE! As of Airflow 2.3.0 you need to use + `Buildkit <https://docs.docker.com/develop/develop-images/build_enhancements/>`_ to build customized + Airflow Docker image. We are using new features of Building (and ``dockerfile:1.4`` syntax) + to make our image faster to build and ``standalone`` - i.e. not needing any extra files from + Airflow in order to be build. As of Airflow 2.3.0, the ``Dockerfile`` that is released with Airflow + does not need any extra folders or files and can be copied and used from any folder. + Previously you needed to copy Airflow sources together with the Dockerfile as some scripts were + needed to make it works. You also need to use ``DOCKER_CONTEXT_FILES`` build arg if you want to Review comment: ```suggestion needed to make it work. You also need to use ``DOCKER_CONTEXT_FILES`` build arg if you want to ``` ########## File path: docs/docker-stack/changelog.rst ########## @@ -0,0 +1,174 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Dockerfile Changelog +==================== + +he ``Dockerfile`` does not strictly follow the `SemVer <https://semver.org/>`_ approach of +Apache Airflow when it comes to features and backwards compatibility. While Airflow code strictly +follows it, the ``Dockerfile`` is really a way to conveniently package Airflow using standard container +approach, occasionally there are some changes in the building process or in the entrypoint of the image +that require slight adaptation. + +The Changelog below describes the changes introduced in each version of the docker images released by +the Airflow team. + +:note: The changelog below concerns only the convenience production images released at + `Airflow DockerHub <https://hub.docker.com/r/apache/airflow>`_ . The images that are released + There are usually built using the ``Dockerfile`` released together with Airflow. However You are + free to take latest released ``Dockerfile`` from Airflow and use it to build an image for + any Airflow version from ``Airflow 2`` line. There is no guarantee that it works, but if it does + then you can use latest features from that image to build the previous Airflow versions. + +:warning: Some of the images below (as noted in the Changelog) have been regenerated using newer Review comment: ```suggestion :warning: Some of the images below (as noted in the changelog) have been regenerated using newer ``` ########## File path: docs/docker-stack/build.rst ########## @@ -428,6 +469,57 @@ right version of python base image: * ``--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster`` uses buster version of Debian (Debian 10) * ``--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-bullseye`` uses bullseye version of Debian (Debian 11) +.. _using-docker-context-files: + +Using docker-context-files +-------------------------- + +When customizing the image, you can optionally make Airflow install custom binaries or provider custom +configuration for your pip in ``docker-context-files``. In order to enable it, you need to add +``--build-arg DOCKER_CONTEXT_FILES=docker-context-files`` build arg when you build the image. +You can pass any subdirectory of your docker context, it will always be mapped to ``/docker-context-files`` +during the build. + +You can use ``docker-context-files`` for the following purposes: + +* you can place ``requirements.txt`` in the ``docker-context-file`` and additional ``pip`` packages + specified there will be installed during the build. + +* you can place ``pip.conf`` and ``.piprc`` in the ``docker-context-files`` folder and they will be used + for all ``pip`` commands (for example you can configure your own sources or authentication mechanisms) + +* you can place .whl packages that you downloaded and install them with ``INSTALL_DOCKER_CONTEXT_FILES`` set + to ``true`` ( see: :ref:`image-build-secure-environments`). It's useful if you build the image in + restricted security environments + +Example command on how you can use the ``docker-context-files`` to add your own dependencies from the +``requirements.txt``: + +.. exampleinclude:: docker-examples/customizing/own-requirements.sh + :language: bash + :start-after: [START build] + :end-before: [END build] + +.. note:: + You can also pass ``--build-arg DOCKER_CONTEXT_FILES=.`` if you want to place your ``requirements.txt`` + in main directory without creating a dedicated folder, however this is a good practice to keep any files + that you copy to the image context in a sub-folder. This makes it easier to separate things that + are used on the host from those that are passed to the Docker. Of course, by default when you run Review comment: ```suggestion are used on the host from those that are passed in Docker context. Of course, by default when you run ``` ########## File path: docs/docker-stack/changelog.rst ########## @@ -0,0 +1,174 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Dockerfile Changelog +==================== + +he ``Dockerfile`` does not strictly follow the `SemVer <https://semver.org/>`_ approach of +Apache Airflow when it comes to features and backwards compatibility. While Airflow code strictly +follows it, the ``Dockerfile`` is really a way to conveniently package Airflow using standard container +approach, occasionally there are some changes in the building process or in the entrypoint of the image +that require slight adaptation. + +The Changelog below describes the changes introduced in each version of the docker images released by +the Airflow team. + +:note: The changelog below concerns only the convenience production images released at + `Airflow DockerHub <https://hub.docker.com/r/apache/airflow>`_ . The images that are released + There are usually built using the ``Dockerfile`` released together with Airflow. However You are + free to take latest released ``Dockerfile`` from Airflow and use it to build an image for + any Airflow version from ``Airflow 2`` line. There is no guarantee that it works, but if it does Review comment: ```suggestion any Airflow version from the ``Airflow 2`` line. There is no guarantee that it works, but if it does ``` ########## File path: docs/docker-stack/changelog.rst ########## @@ -0,0 +1,174 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Dockerfile Changelog +==================== + +he ``Dockerfile`` does not strictly follow the `SemVer <https://semver.org/>`_ approach of +Apache Airflow when it comes to features and backwards compatibility. While Airflow code strictly +follows it, the ``Dockerfile`` is really a way to conveniently package Airflow using standard container +approach, occasionally there are some changes in the building process or in the entrypoint of the image +that require slight adaptation. + +The Changelog below describes the changes introduced in each version of the docker images released by +the Airflow team. + +:note: The changelog below concerns only the convenience production images released at + `Airflow DockerHub <https://hub.docker.com/r/apache/airflow>`_ . The images that are released + There are usually built using the ``Dockerfile`` released together with Airflow. However You are + free to take latest released ``Dockerfile`` from Airflow and use it to build an image for + any Airflow version from ``Airflow 2`` line. There is no guarantee that it works, but if it does + then you can use latest features from that image to build the previous Airflow versions. + +:warning: Some of the images below (as noted in the Changelog) have been regenerated using newer + ``Dockerfiles``. This happens when there is a breaking change that invalidates already released images + and the images need regeneration. This has happened already when MySQL changed the keys they + used to release their packages: `Issue here <https://github.com/apache/airflow/issues/20911>`_ + and 2.1 images are all regenerated using the 2.2 ``Dockerfile``. This is a rare event and Review comment: ```suggestion and 2.1 images were all regenerated using the 2.2 ``Dockerfile``. This is a rare event and ``` ########## File path: docs/docker-stack/changelog.rst ########## @@ -0,0 +1,174 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Dockerfile Changelog +==================== + +he ``Dockerfile`` does not strictly follow the `SemVer <https://semver.org/>`_ approach of +Apache Airflow when it comes to features and backwards compatibility. While Airflow code strictly +follows it, the ``Dockerfile`` is really a way to conveniently package Airflow using standard container +approach, occasionally there are some changes in the building process or in the entrypoint of the image +that require slight adaptation. + +The Changelog below describes the changes introduced in each version of the docker images released by +the Airflow team. + +:note: The changelog below concerns only the convenience production images released at + `Airflow DockerHub <https://hub.docker.com/r/apache/airflow>`_ . The images that are released + There are usually built using the ``Dockerfile`` released together with Airflow. However You are + free to take latest released ``Dockerfile`` from Airflow and use it to build an image for + any Airflow version from ``Airflow 2`` line. There is no guarantee that it works, but if it does + then you can use latest features from that image to build the previous Airflow versions. + +:warning: Some of the images below (as noted in the Changelog) have been regenerated using newer + ``Dockerfiles``. This happens when there is a breaking change that invalidates already released images + and the images need regeneration. This has happened already when MySQL changed the keys they + used to release their packages: `Issue here <https://github.com/apache/airflow/issues/20911>`_ Review comment: ```suggestion used to release their packages (`#20911 <https://github.com/apache/airflow/issues/20911>`_) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
