potiuk commented on a change in pull request #4937: [AIRFLOW-4116]
Multi-staging includes CI image [Step 2/3]
URL: https://github.com/apache/airflow/pull/4937#discussion_r275214819
##########
File path: Dockerfile
##########
@@ -85,14 +85,134 @@ RUN adduser airflow \
&& echo "airflow ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/airflow \
&& chmod 0440 /etc/sudoers.d/airflow
+############################################################################################################
+# This is an image with all APT dependencies needed by CI. It is built on top
of the airlfow APT image
+# Parameters:
+# airflow-apt-deps - this is the base image for CI deps image.
+############################################################################################################
+FROM airflow-apt-deps as airflow-ci-apt-deps
+
+SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
+
+ENV JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
+
+ARG APT_DEPS_IMAGE
+ENV APT_DEPS_IMAGE=${APT_DEPS_IMAGE}
+
+RUN echo "${APT_DEPS_IMAGE}"
+
+# Note the ifs below might be removed if Buildkit will become usable. It
should skip building this
+# image automatically if it is not used. For now we still go through all
layers below but they are empty
+# Note missing directories on debian-stretch
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199
+RUN if [[ "${APT_DEPS_IMAGE}" == "airflow-ci-apt-deps" ]]; then \
+ mkdir -pv /usr/share/man/man1 \
+ && mkdir -pv /usr/share/man/man7 \
+ && apt-get update \
+ && apt-get install --no-install-recommends -y \
+ lsb-release gnupg dirmngr openjdk-8-jdk \
+ vim tmux less unzip net-tools netcat \
+ ldap-utils postgresql-client sqlite3 \
+ krb5-user openssh-client openssh-server \
+ python-selinux \
+ && apt-get autoremove -yqq --purge \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/* \
+ ;\
+ fi
+
+RUN if [[ "${APT_DEPS_IMAGE}" == "airflow-ci-apt-deps" ]]; then \
+ KEY="A4A9406876FCBD3C456770C88C718D3B5072E1F5" \
+ && GNUPGHOME="$(mktemp -d)" \
+ && export GNUPGHOME \
+ && for KEYSERVER in $(shuf -e \
+ ha.pool.sks-keyservers.net \
+ hkp://p80.pool.sks-keyservers.net:80 \
+ keyserver.ubuntu.com \
+ hkp://keyserver.ubuntu.com:80 \
+ pgp.mit.edu) ; do \
+ gpg --keyserver "${KEYSERVER}" --recv-keys "${KEY}" && break ||
true ; \
+ done \
+ && gpg --export "${KEY}" > /etc/apt/trusted.gpg.d/mysql.gpg \
+ && gpgconf --kill all \
+ rm -rf "${GNUPGHOME}"; \
+ apt-key list > /dev/null \
+ && echo "deb http://repo.mysql.com/apt/ubuntu/ trusty mysql-5.7" | \
Review comment:
I think option 2 is the best indeed.
Out of curiosity I also looking into option 1) of baking-in python
installation following
https://github.com/docker-library/python/blob/b9cb77020447a1ac30f5f1e17f31e534826db7bb/3.6/stretch/slim/Dockerfile
and I am close to having a working version, but I think that's an overkill -
we would indeed need to make sure to maintain our own versions.
Right now when we build the image on Dockerhub in case new python images are
released with some security fixes - the image will be rebuilt using those newly
released images. So having them maintained as official images has great benefit.
I think we could simply advise our users to upgrade to latest from 3.5
series or even 3.6 if they are still on ubuntu xenial. It's a bit of pain (just
a little) but there are multiple ways
(https://askubuntu.com/questions/865554/how-do-i-install-python-3-6-using-apt-get).
This might not always be possible of course, but at least for Airflow they
could use virtualenv for example. And xenial is not the latest LTS out there
for quite some time - with Bionic we are ok as it installs 3.6 by default.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
