Davide95 opened a new issue #15493: URL: https://github.com/apache/airflow/issues/15493
**Apache Airflow version**: 2.0.1 - **OS**: Linux (tested both on a GitLab pipeline and using Ubuntu 20.10) - **Kernel** : 5.8.0-50-generic **What happened**: `docker build` does not work if `--build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-2.0.2"` is specified as a build arg on `v2-0-stable` branch: ```sh docker build --pull --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-2.0.2" --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" --build-arg PYTHON_MAJOR_MINOR_VERSION="3.7" --build-arg AIRFLOW_VERSION="2.0.2" --build-arg AIRFLOW_INSTALL_VERSION="==2.0.2" -t airflow:2.0.2 . ``` The returned error is: <details> <summary> ERROR: 404 Client Error: Not Found for url: https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-2.0-3.7.txt </summary> Sending build context to Docker daemon 31.52MB Step 1/164 : ARG AIRFLOW_VERSION="2.0.1" Step 2/164 : ARG AIRFLOW_EXTRAS="async,amazon,celery,cncf.kubernetes,docker,dask,elasticsearch,ftp,grpc,hashicorp,http,ldap,google,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv" Step 3/164 : ARG ADDITIONAL_AIRFLOW_EXTRAS="" Step 4/164 : ARG ADDITIONAL_PYTHON_DEPS="" Step 5/164 : ARG AIRFLOW_HOME=/opt/airflow Step 6/164 : ARG AIRFLOW_UID="50000" Step 7/164 : ARG AIRFLOW_GID="50000" Step 8/164 : ARG CASS_DRIVER_BUILD_CONCURRENCY="8" Step 9/164 : ARG PYTHON_BASE_IMAGE="python:3.6-slim-buster" Step 10/164 : ARG AIRFLOW_PIP_VERSION=20.2.4 Step 11/164 : ARG PIP_PROGRESS_BAR="on" Step 12/164 : FROM ${PYTHON_BASE_IMAGE} as airflow-build-image 3.7-slim-buster: Pulling from library/python Digest: sha256:f6bcc6d1fe0efe4ff1fb5590c23fddbae36c8757921d5d89da4e2b24d0ae704a Status: Image is up to date for python:3.7-slim-buster ---> 30a42f143b4e Step 13/164 : SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"] ---> Using cache ---> 073cc7b1d93f Step 14/164 : ARG PYTHON_BASE_IMAGE ---> Using cache ---> 07fd5a8530bd Step 15/164 : ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE} ---> Using cache ---> 6ad7ba8c32c6 Step 16/164 : ENV DEBIAN_FRONTEND=noninteractive LANGUAGE=C.UTF-8 LANG=C.UTF-8 LC_ALL=C.UTF-8 LC_CTYPE=C.UTF-8 LC_MESSAGES=C.UTF-8 ---> Using cache ---> 90946c667620 Step 17/164 : RUN apt-get update && apt-get install -y --no-install-recommends curl gnupg2 && apt-get autoremove -yqq --purge && apt-get clean && rm -rf /var/lib/apt/lists/* ---> Using cache ---> df99afd34b26 Step 18/164 : ARG DEV_APT_DEPS=" apt-transport-https apt-utils build-essential ca-certificates gnupg dirmngr freetds-bin freetds-dev gosu krb5-user ldap-utils libffi-dev libkrb5-dev libldap2-dev libpq-dev libsasl2-2 libsasl2-dev libsasl2-modules libssl-dev locales lsb-release nodejs openssh-client postgresql-client python-selinux sasl2-bin software-properties-common sqlite3 sudo unixodbc unixodbc-dev yarn" ---> Using cache ---> 74bfa6455df1 Step 19/164 : ENV DEV_APT_DEPS=${DEV_APT_DEPS} ---> Using cache ---> 419bd22d689e Step 20/164 : ARG ADDITIONAL_DEV_APT_DEPS="" ---> Using cache ---> 1f69197ad8b8 Step 21/164 : ENV ADDITIONAL_DEV_APT_DEPS=${ADDITIONAL_DEV_APT_DEPS} ---> Using cache ---> d3401572cd99 Step 22/164 : ARG DEV_APT_COMMAND=" curl --fail --location https://deb.nodesource.com/setup_10.x | bash - && curl https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - > /dev/null && echo 'deb https://dl.yarnpkg.com/debian/ stable main' > /etc/apt/sources.list.d/yarn.list" ---> Using cache ---> 5453be9d6ea3 Step 23/164 : ENV DEV_APT_COMMAND=${DEV_APT_COMMAND} ---> Using cache ---> d2a07de63d88 Step 24/164 : ARG ADDITIONAL_DEV_APT_COMMAND="echo" ---> Using cache ---> 3ba10c8fed2d Step 25/164 : ENV ADDITIONAL_DEV_APT_COMMAND=${ADDITIONAL_DEV_APT_COMMAND} ---> Using cache ---> 203a950784fc Step 26/164 : ARG ADDITIONAL_DEV_APT_ENV="" ---> Using cache ---> c24d0ff0e145 Step 27/164 : RUN mkdir -pv /usr/share/man/man1 && mkdir -pv /usr/share/man/man7 && export ${ADDITIONAL_DEV_APT_ENV?} && bash -o pipefail -e -u -x -c "${DEV_APT_COMMAND}" && bash -o pipefail -e -u -x -c "${ADDITIONAL_DEV_APT_COMMAND}" && apt-get update && apt-get install -y --no-install-recommends ${DEV_APT_DEPS} ${ADDITIONAL_DEV_APT_DEPS} && apt-get autoremove -yqq --purge && apt-get clean && rm -rf /var/lib/apt/lists/* ---> Using cache ---> 8ddec6404e9a Step 28/164 : ARG INSTALL_MYSQL_CLIENT="true" ---> Using cache ---> ef368f06324b Step 29/164 : ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT} ---> Using cache ---> 99dd0db02c95 Step 30/164 : COPY scripts/docker/install_mysql.sh /scripts/docker/install_mysql.sh ---> Using cache ---> 6b57562736d8 Step 31/164 : COPY docker-context-files /docker-context-files ---> Using cache ---> 3fa61dcc27a1 Step 32/164 : RUN bash ./scripts/docker/install_mysql.sh dev ---> Using cache ---> 4a39ffd3eb95 Step 33/164 : ARG AIRFLOW_REPO=apache/airflow ---> Using cache ---> 1ce430cea37b Step 34/164 : ENV AIRFLOW_REPO=${AIRFLOW_REPO} ---> Using cache ---> 0f1ea52fb56b Step 35/164 : ARG AIRFLOW_BRANCH=master ---> Using cache ---> 0ab86ebd84bd Step 36/164 : ENV AIRFLOW_BRANCH=${AIRFLOW_BRANCH} ---> Using cache ---> 1653bf37a629 Step 37/164 : ARG AIRFLOW_EXTRAS ---> Using cache ---> cfc2a7d5b8d2 Step 38/164 : ARG ADDITIONAL_AIRFLOW_EXTRAS="" ---> Using cache ---> 64d10026c456 Step 39/164 : ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS} ---> Using cache ---> 94b7c449d1f4 Step 40/164 : ARG CONSTRAINTS_GITHUB_REPOSITORY="apache/airflow" ---> Using cache ---> a4c42604db01 Step 41/164 : ENV CONSTRAINTS_GITHUB_REPOSITORY=${CONSTRAINTS_GITHUB_REPOSITORY} ---> Using cache ---> b096c4c56bf6 Step 42/164 : ARG AIRFLOW_CONSTRAINTS="constraints-2.0" ---> Using cache ---> d934685366ce Step 43/164 : ENV AIRFLOW_CONSTRAINTS=${AIRFLOW_CONSTRAINTS} ---> Using cache ---> 4463555e3108 Step 44/164 : ARG AIRFLOW_CONSTRAINTS_REFERENCE="" ---> Using cache ---> 28e3663aa67a Step 45/164 : ENV AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE} ---> Using cache ---> f742d6fd13f6 Step 46/164 : ARG AIRFLOW_CONSTRAINTS_LOCATION="" ---> Using cache ---> b492816e07dd Step 47/164 : ENV AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION} ---> Using cache ---> b6357a41e012 Step 48/164 : ARG DEFAULT_CONSTRAINTS_BRANCH="constraints-master" ---> Using cache ---> 9bfc76bc345d Step 49/164 : ENV DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH} ---> Using cache ---> e3a8a75d1efc Step 50/164 : ENV PATH=${PATH}:/root/.local/bin ---> Using cache ---> 4251b29f4a77 Step 51/164 : RUN mkdir -p /root/.local/bin ---> Using cache ---> 8e8cdfdd911b Step 52/164 : RUN if [[ -f /docker-context-files/.pypirc ]]; then cp /docker-context-files/.pypirc /root/.pypirc; fi ---> Using cache ---> f719c4209ef5 Step 53/164 : ARG AIRFLOW_PIP_VERSION ---> Using cache ---> c3cd71821ce9 Step 54/164 : ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} ---> Using cache ---> 0d9fe5d7d839 Step 55/164 : ARG PIP_PROGRESS_BAR ---> Using cache ---> c8d7d0ff4112 Step 56/164 : ENV PIP_PROGRESS_BAR=${PIP_PROGRESS_BAR} ---> Using cache ---> f53a45e612be Step 57/164 : ENV AIRFLOW_INSTALL_USER_FLAG="--user" ---> Using cache ---> aef348201641 Step 58/164 : ENV AIRFLOW_INSTALL_EDITABLE_FLAG="" ---> Using cache ---> 7ad5cdc9a28b Step 59/164 : RUN pip install --no-cache-dir --upgrade "pip==${AIRFLOW_PIP_VERSION}" ---> Using cache ---> 7c1542ace7ea Step 60/164 : ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="false" ---> Using cache ---> ec98972f93b3 Step 61/164 : ENV AIRFLOW_PRE_CACHED_PIP_PACKAGES=${AIRFLOW_PRE_CACHED_PIP_PACKAGES} ---> Using cache ---> 87dc43479c44 Step 62/164 : ARG INSTALL_PROVIDERS_FROM_SOURCES="false" ---> Using cache ---> a54b8079b5b2 Step 63/164 : ENV INSTALL_PROVIDERS_FROM_SOURCES=${INSTALL_PROVIDERS_FROM_SOURCES} ---> Using cache ---> 7748f5159b2b Step 64/164 : ARG AIRFLOW_VERSION ---> Using cache ---> fc451eb5c5ec Step 65/164 : ENV AIRFLOW_VERSION=${AIRFLOW_VERSION} ---> Using cache ---> 962e8db3b320 Step 66/164 : ARG AIRFLOW_INSTALLATION_METHOD="apache-airflow" ---> Using cache ---> 9c327e191a12 Step 67/164 : ENV AIRFLOW_INSTALLATION_METHOD=${AIRFLOW_INSTALLATION_METHOD} ---> Using cache ---> 7fcc8a7e35d1 Step 68/164 : ARG AIRFLOW_VERSION_SPECIFICATION="" ---> Using cache ---> 65c0cef85e83 Step 69/164 : ENV AIRFLOW_VERSION_SPECIFICATION=${AIRFLOW_VERSION_SPECIFICATION} ---> Using cache ---> d71a809b3bab Step 70/164 : COPY scripts/docker/common.sh /scripts/docker/common.sh ---> cce345b407b5 Step 71/164 : COPY scripts/docker/install_airflow_from_branch_tip.sh /scripts/docker/install_airflow_from_branch_tip.sh ---> 90a9d7a2dbf6 Step 72/164 : ARG UPGRADE_TO_NEWER_DEPENDENCIES="false" ---> Running in 351a03c5eadb Removing intermediate container 351a03c5eadb ---> 2a1e3d73796e Step 73/164 : ENV UPGRADE_TO_NEWER_DEPENDENCIES=${UPGRADE_TO_NEWER_DEPENDENCIES} ---> Running in 7f276dceff28 Removing intermediate container 7f276dceff28 ---> aa223dd2ed3c Step 74/164 : RUN if [[ ${AIRFLOW_PRE_CACHED_PIP_PACKAGES} == "true" && ${UPGRADE_TO_NEWER_DEPENDENCIES} == "false" ]]; then bash /scripts/docker/install_airflow_from_branch_tip.sh; fi ---> Running in 072dfa8d3e3c + [[ false == \t\r\u\e ]] Removing intermediate container 072dfa8d3e3c ---> 9ffda1fd10a1 Step 75/164 : ARG AIRFLOW_SOURCES_FROM="empty" ---> Running in 3ad0497283dc Removing intermediate container 3ad0497283dc ---> 6240b8996415 Step 76/164 : ENV AIRFLOW_SOURCES_FROM=${AIRFLOW_SOURCES_FROM} ---> Running in 5135b01b14aa Removing intermediate container 5135b01b14aa ---> 9d08170562ed Step 77/164 : ARG AIRFLOW_SOURCES_TO="/empty" ---> Running in ee5f63bab546 Removing intermediate container ee5f63bab546 ---> 1b96d150d9a4 Step 78/164 : ENV AIRFLOW_SOURCES_TO=${AIRFLOW_SOURCES_TO} ---> Running in 8a78f1a01f39 Removing intermediate container 8a78f1a01f39 ---> b12c53e9e969 Step 79/164 : COPY ${AIRFLOW_SOURCES_FROM} ${AIRFLOW_SOURCES_TO} ---> 25f3f9c5a9c0 Step 80/164 : ARG CASS_DRIVER_BUILD_CONCURRENCY ---> Running in df1afcd9d6eb Removing intermediate container df1afcd9d6eb ---> f5d602b5cd39 Step 81/164 : ENV CASS_DRIVER_BUILD_CONCURRENCY=${CASS_DRIVER_BUILD_CONCURRENCY} ---> Running in ae8b9d677f86 Removing intermediate container ae8b9d677f86 ---> 175e9317ccbf Step 82/164 : ARG ADDITIONAL_PYTHON_DEPS="" ---> Running in 67835e4a2067 Removing intermediate container 67835e4a2067 ---> caa7d1fd5d36 Step 83/164 : ENV ADDITIONAL_PYTHON_DEPS=${ADDITIONAL_PYTHON_DEPS} ---> Running in 4cf6636e86fc Removing intermediate container 4cf6636e86fc ---> 02b8b71b491d Step 84/164 : ARG INSTALL_FROM_DOCKER_CONTEXT_FILES="" ---> Running in a48e6e74a34a Removing intermediate container a48e6e74a34a ---> 07781f6dbe16 Step 85/164 : ENV INSTALL_FROM_DOCKER_CONTEXT_FILES=${INSTALL_FROM_DOCKER_CONTEXT_FILES} ---> Running in 5ee6968fcb42 Removing intermediate container 5ee6968fcb42 ---> 34704d3dc4df Step 86/164 : ARG INSTALL_FROM_PYPI="true" ---> Running in 3ed6e67b7218 Removing intermediate container 3ed6e67b7218 ---> b818018f22b7 Step 87/164 : ENV INSTALL_FROM_PYPI=${INSTALL_FROM_PYPI} ---> Running in 1c34feaf15b5 Removing intermediate container 1c34feaf15b5 ---> 85aba433d76c Step 88/164 : ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 urllib3<1.26 pyjwt<2.0.0 dill<0.3.3" ---> Running in 4164e06f93cc Removing intermediate container 4164e06f93cc ---> 70ab1d79aeb8 Step 89/164 : WORKDIR /opt/airflow ---> Running in 0d1865dda571 Removing intermediate container 0d1865dda571 ---> 3f9d122e9a5e Step 90/164 : ARG CONTINUE_ON_PIP_CHECK_FAILURE="false" ---> Running in 8f84390bb706 Removing intermediate container 8f84390bb706 ---> 95dc9d5fe166 Step 91/164 : COPY scripts/docker/install*.sh /scripts/docker/ ---> acd7e23c242a Step 92/164 : RUN if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "true" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; elif [[ ${INSTALL_FROM_PYPI} == "true" ]]; then bash /scripts/docker/install_airflow.sh; fi; if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r || true ; find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r || true ---> Running in 0583608402ee + [[ '' == \t\r\u\e ]] + [[ true == \t\r\u\e ]] + bash /scripts/docker/install_airflow.sh ++ set -euo pipefail ++ test -v INSTALL_MYSQL_CLIENT ++ test -v AIRFLOW_INSTALL_USER_FLAG ++ test -v AIRFLOW_REPO ++ test -v AIRFLOW_BRANCH ++ test -v AIRFLOW_PIP_VERSION ++ set -x ++ set +x Installing all packages with constraints and upgrade if needed ERROR: 404 Client Error: Not Found for url: https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-2.0-3.7.txt WARNING: You are using pip version 20.2.4; however, version 21.0.1 is available. You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command. The command '/bin/bash -o pipefail -e -u -x -c if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "true" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; elif [[ ${INSTALL_FROM_PYPI} == "true" ]]; then bash /scripts/docker/install_airflow.sh; fi; if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r || true ; find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r || true' returned a non-zero code: 1 </details> **What you expected to happen**: `docker build` already works in these scenarios: * launching the previous above docker build using the `master` branch; * using `AIRFLOW_CONSTRAINTS_LOCATION="https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.7.txt"` instead of `AIRFLOW_CONSTRAINTS_REFERENCE` on `v2-0-stable` branch. **How to reproduce it**: ```sh git clone https://github.com/apache/airflow.git git checkout v2-0-stable docker build --pull --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-2.0.2" --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" --build-arg PYTHON_MAJOR_MINOR_VERSION="3.7" --build-arg AIRFLOW_VERSION="2.0.2" --build-arg AIRFLOW_INSTALL_VERSION="==2.0.2" -t airflow:2.0.2 . ``` **Anything else we need to know**: I've also noticed that https://airflow.apache.org/docs/apache-airflow/2.0.1/production-deployment.html is different from https://airflow.apache.org/docs/apache-airflow/2.0.2/production-deployment.html: there is no mention to these build args anymore. As suggested by @ashb I've put `set -x` / `set +x` at the start / end of `common::get_constraints_location` function to have more debug info (thanks for the hint). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
