potiuk commented on a change in pull request #7832: Add production image support URL: https://github.com/apache/airflow/pull/7832#discussion_r402352209
########## File path: Dockerfile ########## @@ -0,0 +1,333 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# THIS DOCKERFILE IS INTENDED FOR PRODUCTION USE AND DEPLOYMENT. +# NOTE! IT IS ALPHA-QUALITY FOR NOW - WE ARE IN A PROCESS OF TESTING IT +# +# +# This is a multi-segmented image. It actually contains two images: +# +# airflow-build-image - there all airflow dependencies can be installed (and +# built - for those dependencies that require +# build essentials). Airflow is installed there with +# --user switch so that all the dependencies are +# installed to ${HOME}/.local +# +# main - this is the actual production image that is much +# smaller because it does not contain all the build +# essentials. Instead the ${HOME}/.local folder +# is copied from the build-image - this way we have +# only result of installation and we do not need +# all the build essentials. This makes the image +# much smaller. +# +ARG AIRFLOW_VERSION="2.0.0.dev0" +ARG WWW_FOLDER="www" + +ARG AIRFLOW_EXTRAS="async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv" + +ARG AIRFLOW_HOME=/opt/airflow +ARG AIRFLOW_UID="50000" +ARG AIRFLOW_GID="50000" + +ARG PIP_VERSION="19.0.2" +ARG CASS_DRIVER_BUILD_CONCURRENCY="8" + +ARG PYTHON_BASE_IMAGE="python:3.6-slim-buster" +ARG PYTHON_MAJOR_MINOR_VERSION="3.6" + +############################################################################################## +# This is the build image where we build all dependencies +############################################################################################## +FROM ${PYTHON_BASE_IMAGE} as airflow-build-image +SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"] + +LABEL org.apache.airflow.distro="debian" +LABEL org.apache.airflow.distro.version="buster" +LABEL org.apache.airflow.module="airflow" +LABEL org.apache.airflow.component="airflow" +LABEL org.apache.airflow.image="airflow-build-image" + +ARG PYTHON_BASE_IMAGE +ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE} + +ARG PYTHON_MAJOR_MINOR_VERSION +ENV PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION} + +# Make sure noninteractive debian install is used and language variables set +ENV DEBIAN_FRONTEND=noninteractive LANGUAGE=C.UTF-8 LANG=C.UTF-8 LC_ALL=C.UTF-8 \ + LC_CTYPE=C.UTF-8 LC_MESSAGES=C.UTF-8 + +# Install curl and gnupg2 - needed to download nodejs in the next step +RUN apt-get update \ + && apt-get install -y --no-install-recommends \ + curl \ + gnupg2 \ + && apt-get autoremove -yqq --purge \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* + +# Install basic apt dependencies +RUN curl --fail --location https://deb.nodesource.com/setup_10.x | bash - \ + && curl https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - > /dev/null \ + && echo "deb https://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarn.list \ + # Note missing man directories on debian-buster + # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199 + && mkdir -pv /usr/share/man/man1 \ + && mkdir -pv /usr/share/man/man7 \ + && apt-get update \ + && apt-get install -y --no-install-recommends \ + apt-transport-https \ + apt-utils \ + build-essential \ + ca-certificates \ + curl \ + gnupg \ + dirmngr \ + freetds-bin \ + freetds-dev \ + gosu \ + krb5-user \ + ldap-utils \ + libffi-dev \ + libkrb5-dev \ + libpq-dev \ + libsasl2-2 \ + libsasl2-dev \ + libsasl2-modules \ + libssl-dev \ + locales \ + lsb-release \ + nodejs \ + openssh-client \ + postgresql-client \ + python-selinux \ + sasl2-bin \ + software-properties-common \ + sqlite3 \ + sudo \ + unixodbc \ + unixodbc-dev \ + yarn \ + && apt-get autoremove -yqq --purge \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* + +# Install MySQL client from Oracle repositories (Debian installs mariadb) +RUN KEY="A4A9406876FCBD3C456770C88C718D3B5072E1F5" \ + && GNUPGHOME="$(mktemp -d)" \ + && export GNUPGHOME \ + && for KEYSERVER in $(shuf -e \ + ha.pool.sks-keyservers.net \ + hkp://p80.pool.sks-keyservers.net:80 \ + keyserver.ubuntu.com \ + hkp://keyserver.ubuntu.com:80 \ + pgp.mit.edu) ; do \ + gpg --keyserver "${KEYSERVER}" --recv-keys "${KEY}" && break || true ; \ + done \ + && gpg --export "${KEY}" | apt-key add - \ + && gpgconf --kill all \ + rm -rf "${GNUPGHOME}"; \ + apt-key list > /dev/null \ + && echo "deb http://repo.mysql.com/apt/debian/ stretch mysql-5.7" | tee -a /etc/apt/sources.list.d/mysql.list \ + && apt-get update \ + && apt-get install --no-install-recommends -y \ + libmysqlclient-dev \ + mysql-client \ + && apt-get autoremove -yqq --purge \ + && apt-get clean && rm -rf /var/lib/apt/lists/* + +# disable bytecode generation (only in build image - it does not affect the main image) +ENV PYTHONDONTWRITEBYTECODE=1 + +ARG PIP_VERSION +ENV PIP_VERSION=${PIP_VERSION} + +RUN pip install --upgrade pip==${PIP_VERSION} + +ARG AIRFLOW_SOURCES_TO="/opt/airflow" +ENV AIRFLOW_SOURCES_TO=${AIRFLOW_SOURCES_TO} + +COPY ${AIRFLOW_SOURCES_FROM} ${AIRFLOW_SOURCES_TO} + +ARG CASS_DRIVER_BUILD_CONCURRENCY +ENV CASS_DRIVER_BUILD_CONCURRENCY=${CASS_DRIVER_BUILD_CONCURRENCY} + +ARG AIRFLOW_VERSION +ENV AIRFLOW_VERSION=${AIRFLOW_VERSION} + +ARG AIRFLOW_EXTRAS +ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS} + +ARG AIRFLOW_INSTALL_SOURCES="." +ENV AIRFLOW_INSTALL_SOURCES=${AIRFLOW_INSTALL_SOURCES} + +ARG AIRFLOW_INSTALL_VERSION="" +ENV AIRFLOW_INSTALL_VERSION=${AIRFLOW_INSTALL_VERSION} + +ARG CONSTRAINT_REQUIREMENTS="requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt" +ENV CONSTRAINT_REQUIREMENTS=${CONSTRAINT_REQUIREMENTS} + +ARG AIRFLOW_SOURCES_FROM="." +ENV AIRFLOW_SOURCES_FROM=${AIRFLOW_SOURCES_FROM} + +WORKDIR /opt/airflow + +# hadolint ignore=DL3020 +ADD "${CONSTRAINT_REQUIREMENTS}" /requirements.txt + +ENV PATH=${PATH}:/root/.local/bin + +RUN pip install --user "${AIRFLOW_INSTALL_SOURCES}[${AIRFLOW_EXTRAS}]${AIRFLOW_INSTALL_VERSION}" \ + --constraint /requirements.txt && \ + find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r && \ + find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r + + +ARG WWW_FOLDER +ENV WWW_FOLDER=${WWW_FOLDER} + +ENV AIRFLOW_WWW=/root/.local/lib/python${PYTHON_MAJOR_MINOR_VERSION}/site-packages/airflow/${WWW_FOLDER} + +RUN yarn --cwd ${AIRFLOW_WWW} install --frozen-lockfile --no-cache \ + && yarn --cwd ${AIRFLOW_WWW} run prod \ + && rm -rf ${AIRFLOW_WWW}/node_modules + +ARG ENTRYPOINT_FILE="entrypoint.sh" +ENV ENTRYPOINT_FILE="${ENTRYPOINT_FILE}" Review comment: I checked how it is in Breeze - it is only needed in case we have different entrypoint.sh for different versions of Airflow. I am using the same Dockerfile to build 2.0 and 1.10 when installing from pypi or URL. for now there is no need indeed as I removed resetdb vs. dbreset -- so entrypoints will be the same but I want to keep the possibility that entrypoints will be different in 1.10 and 2.0 and that we would like to keep them separated. I really like (and yes) all the functionality to be able to build and run production-version of the released airflow in the master branch. That's really helpful to test 1.10 images. It's the same thing as the requirements.txt - we get them from v1_10_test branch in order to build 1.10 image from master. I think it's rather useful. I found it useful for my testing at least ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
