mik-laj commented on a change in pull request #14765: URL: https://github.com/apache/airflow/pull/14765#discussion_r593816550
########## File path: docs/docker-stack/build.rst ########## @@ -0,0 +1,380 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Building the image +================== + +Before you dive-deeply in the way how the Airflow Image is build, named and why we are doing it the +way we do, you might want to know very quickly how you can extend or customize the existing image +for Apache Airflow. This chapter gives you a short answer to those questions. + +Airflow Summit 2020's `Production Docker Image <https://youtu.be/wDr3Y7q2XoI>`_ talk provides more +details about the context, architecture and customization/extension methods for the Production Image. + +Extending the image +------------------- + +Extending the image is easiest if you just need to add some dependencies that do not require +compiling. The compilation framework of Linux (so called ``build-essential``) is pretty big, and +for the production images, size is really important factor to optimize for, so our Production Image +does not contain ``build-essential``. If you need compiler like gcc or g++ or make/cmake etc. - those +are not found in the image and it is recommended that you follow the "customize" route instead. + +How to extend the image - it is something you are most likely familiar with - simply +build a new image using Dockerfile's ``FROM`` directive and add whatever you need. Then you can add your +Debian dependencies with ``apt`` or PyPI dependencies with ``pip install`` or any other stuff you need. + +You should be aware, about a few things: + +* The production image of airflow uses "airflow" user, so if you want to add some of the tools + as ``root`` user, you need to switch to it with ``USER`` directive of the Dockerfile. Also you + should remember about following the + `best practises of Dockerfiles <https://docs.docker.com/develop/develop-images/dockerfile_best-practices/>`_ + to make sure your image is lean and small. + + .. code-block:: dockerfile + + FROM apache/airflow:2.0.1 + USER root + RUN apt-get update \ + && apt-get install -y --no-install-recommends \ + my-awesome-apt-dependency-to-add \ + && apt-get autoremove -yqq --purge \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* + USER airflow + + +* PyPI dependencies in Apache Airflow are installed in the user library, of the "airflow" user, so + you need to install them with the ``--user`` flag and WITHOUT switching to airflow user. Note also + that using --no-cache-dir is a good idea that can help to make your image smaller. + + .. code-block:: dockerfile + + FROM apache/airflow:2.0.1 + RUN pip install --no-cache-dir --user my-awesome-pip-dependency-to-add + +* As of 2.0.1 image the ``--user`` flag is turned on by default by setting ``PIP_USER`` environment variable + to ``true``. This can be disabled by un-setting the variable or by setting it to ``false``. + + +* If your apt, or PyPI dependencies require some of the build-essentials, then your best choice is + to follow the "Customize the image" route. However it requires to checkout sources of Apache Airflow, + so you might still want to choose to add build essentials to your image, even if your image will + be significantly bigger. + + .. code-block:: dockerfile + + FROM apache/airflow:2.0.1 + USER root + RUN apt-get update \ + && apt-get install -y --no-install-recommends \ + build-essential my-awesome-apt-dependency-to-add \ + && apt-get autoremove -yqq --purge \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* + USER airflow + RUN pip install --no-cache-dir --user my-awesome-pip-dependency-to-add + +* You can also embed your dags in the image by simply adding them with COPY directive of Airflow. + The DAGs in production image are in ``/opt/airflow/dags`` folder. + +Customizing the image +--------------------- + +Customizing the image is an alternative way of adding your own dependencies to the image - better +suited to prepare optimized production images. + +The advantage of this method is that it produces optimized image even if you need some compile-time +dependencies that are not needed in the final image. You need to use Airflow Sources to build such images +from the `official distribution folder of Apache Airflow <https://downloads.apache.org/airflow/>`_ for the +released versions, or checked out from the GitHub project if you happen to do it from git sources. + +The easiest way to build the image is to use ``breeze`` script, but you can also build such customized +image by running appropriately crafted docker build in which you specify all the ``build-args`` +that you need to add to customize it. You can read about all the args and ways you can build the image +in :doc:`build-arg-ref`. + +Here just a few examples are presented which should give you general understanding of what you can customize. + +This builds production image in version 3.6 with default extras from the local sources (master version Review comment: We had these examples repeated twice in the documentation - once in the "Customizing the image" section and the second time under the tables in the "Production image build arguments" section. Some examples were common and some were different. Now we have one place with examples. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
