potiuk commented on a change in pull request #4936: [AIRFLOW-4115]
Multi-staging Aiflow Docker image [Step 1/3]
URL: https://github.com/apache/airflow/pull/4936#discussion_r268671279
##########
File path: Dockerfile
##########
@@ -13,46 +13,179 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
+#
+# WARNING: THIS DOCKERFILE IS NOT INTENDED FOR PRODUCTION USE OR DEPLOYMENT.
+#
+# Arguments of the build
+ARG PYTHON_BASE_IMAGE="python:3.6-slim"
+ARG AIRFLOW_VERSION="2.0.0.dev0"
+# Which image is used as dependency for the main image
+ARG APT_DEPS_IMAGE="airflow-apt-deps"
+# Speeds up building the image - cassandra driver without CYTHON saves around
10 minutes
+ARG CASS_DRIVER_NO_CYTHON="1"
+# Build cassandra driver on multiple CPUs
+ARG CASS_DRIVER_BUILD_CONCURRENCY="8"
+# By default PIP install is run without cache to make image smaller
+ARG PIP_NO_CACHE_DIR="true"
+# Additional python deps to install
+ARG ADDITIONAL_PYTHON_DEPS=""
+# PIP version used to install dependencies
+ARG PIP_VERSION="19.0.1"
+# By increasing this number we can do force build of all dependencies
+ARG DEPENDENCIES_EPOCH_NUMBER="1"
+############################################################################################################
+# This is base image with APT dependencies needed by Airflow. It is based on a
python slim image
+# Parameters:
+# PYTHON_BASE_IMAGE - base python image (python:x.y-slim)
+############################################################################################################
+FROM ${PYTHON_BASE_IMAGE} as airflow-apt-deps
+
+SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
+
+ARG PYTHON_BASE_IMAGE
+ARG AIRFLOW_VERSION
+ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE}
+ENV AIRFLOW_VERSION=$AIRFLOW_VERSION
+
+# Print versions
+RUN echo "Base image: ${PYTHON_BASE_IMAGE}"
+RUN echo "Airflow version: ${AIRFLOW_VERSION}"
+
+# Make sure noninteractie debian install is used and language variab1les set
+ENV DEBIAN_FRONTEND=noninteractive LANGUAGE=C.UTF-8 LANG=C.UTF-8
LC_ALL=C.UTF-8 \
+ LC_CTYPE=C.UTF-8 LC_MESSAGES=C.UTF-8
+
+# Increase the value below to force renstalling of all dependencies
+ENV DEPENDENCIES_EPOCH_NUMBER=${DEPENDENCIES_EPOCH_NUMBER}
+
+# Install curl and gnupg2 - needed to download nodejs in next step
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends \
+ curl gnupg2 \
+ && apt-get autoremove -yqq --purge \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+
+# Install basic apt dependencies
+RUN curl -sL https://deb.nodesource.com/setup_10.x | bash - \
Review comment:
But that would prevent you from re-running npm ci/npm run prod when you work
with mounted sources. It's not really acceptable for local development with the
CI image.
I think it's good to keep it - at the very least - for the CI image. I can
think about some optional step to build it as another stage for "Airflow" image
- but I think it's not really good idea. I will think a bit more about it, but
as I see it, it would complicate the whole Dockerfile and image a lot with
little gain. I'd rather leave that in for now.
Especially that we have bright future. I think it's a good candidate for
upcoming changes in Docker - especially Buildkit. There is a new, experimental
feature with mount type in RUN command. It could be used for that purpose.
Secret type seems especially interesting as it prevents the layer generated by
a RUN command to ever appear in the final image (it will be present during the
build):
https://github.com/moby/buildkit/blob/b521aae3ea1e1bb73298481ac538f9880ccd3854/frontend/dockerfile/docs/experimental.md#run---mounttypesecret
I already saw people commenting that they would use secret for that very
purpose you described. It was foreseen for other purpose (to keep secretes from
leaking) but maybe in the final buildkit release there will be a new type
"ignored" or smth. :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
