potiuk commented on a change in pull request #4937: [AIRFLOW-4116]
Multi-staging includes CI image [Step 2/3]
URL: https://github.com/apache/airflow/pull/4937#discussion_r268681317
##########
File path: Dockerfile
##########
@@ -85,14 +85,134 @@ RUN adduser airflow \
&& echo "airflow ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/airflow \
&& chmod 0440 /etc/sudoers.d/airflow
+############################################################################################################
+# This is an image with all APT dependencies needed by CI. It is built on top
of the airlfow APT image
+# Parameters:
+# airflow-apt-deps - this is the base image for CI deps image.
+############################################################################################################
+FROM airflow-apt-deps as airflow-ci-apt-deps
+
+SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
+
+ENV JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
+
+ARG APT_DEPS_IMAGE
+ENV APT_DEPS_IMAGE=${APT_DEPS_IMAGE}
+
+RUN echo "${APT_DEPS_IMAGE}"
+
+# Note the ifs below might be removed if Buildkit will become usable. It
should skip building this
+# image automatically if it is not used. For now we still go through all
layers below but they are empty
+# Note missing directories on debian-stretch
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199
+RUN if [[ "${APT_DEPS_IMAGE}" == "airflow-ci-apt-deps" ]]; then \
Review comment:
I tried it initially and it was indeed very complex :).
I think there is a huge benefit on being able to run "docker build ." and
get the Airflow images build via single command (this is how it works now). We
can build the main image by just 'docker build ." but also CI image in a single
build by just changing the "APT_DEPS_IMAGE" build-arg to be
"airflow-ci-apt-deps". If we split it to separate files, we can still have one
'docker build' for the for "Airflow" image but it will no longer be possible to
build CI image with single "docker build . --build-arg=..." command.
The main problem if you keep separate files is that you cannot use the nice
multi-staging "local" image names. You have to use tags or hashes and make sure
that you use the right image build in a separate Docker build. So to build an
image you need to coordinate several 'docker builds'. It's messy. Been there,
done that. Multi-staging simplifies it a lot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
