potiuk commented on a change in pull request #4543: [AIRFLOW-3718] [WIP]
Multi-layered version of the docker image
URL: https://github.com/apache/airflow/pull/4543#discussion_r262504995
##########
File path: Dockerfile
##########
@@ -13,45 +13,72 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
+# WARNING: THIS DOCKERFILE IS NOT INTENDED FOR PRODUCTION USE OR DEPLOYMENT.
+ARG BASE_IMAGE
+FROM ${BASE_IMAGE}
-FROM python:3.6-slim
+WORKDIR /opt/airflow
+
+RUN echo "Airflow version: ${AIRFLOW_VERSION}"
+
+ARG AIRFLOW_HOME=/usr/local/airflow
+RUN mkdir -p $AIRFLOW_HOME
-ENV AIRFLOW_HOME=/usr/local/airflow
-ARG AIRFLOW_DEPS="all"
-ARG PYTHON_DEPS=""
-ARG BUILD_DEPS="freetds-dev libkrb5-dev libssl-dev libffi-dev libpq-dev git"
-ARG APT_DEPS="libsasl2-dev freetds-bin build-essential
default-libmysqlclient-dev apt-utils curl rsync netcat locales"
+# Airflow extras to be installed
+ARG AIRFLOW_EXTRAS="all"
-ENV PATH="$HOME/.npm-packages/bin:$PATH"
+# Increase the value here to force reinstalling Apache Airflow pip dependencies
+ENV FORCE_REINSTALL_ALL_PIP_DEPENDENCIES=1
-RUN set -euxo pipefail \
- && apt update \
- && if [ -n "${APT_DEPS}" ]; then apt install -y $APT_DEPS; fi \
- && curl -sL https://deb.nodesource.com/setup_10.x | bash - \
- && apt update \
- && apt install -y nodejs \
- && apt autoremove -yqq --purge \
- && apt clean
+# Airflow sources change frequently but dependency onfiguration won't change
that often
+# We copy setup.py and other files needed to perform setup of dependencies
+# This way cache here will only be invalidated if any of the
+# version/setup configuration change but not when airflow sources change
+COPY setup.py /opt/airflow/setup.py
+COPY setup.cfg /opt/airflow/setup.cfg
+COPY airflow/version.py /opt/airflow/airflow/version.py
+COPY airflow/__init__.py /opt/airflow/airflow/__init__.py
+COPY airflow/bin/airflow /opt/airflow/airflow/bin/airflow
+
+# First install only dependencies but no Apache Airflow itself
+# This way regular changes in sources of Airflow will not trigger
reinstallation of all dependencies
+# And this Docker layer will be reused between builds.
+RUN pip install --no-cache-dir --upgrade pip==19.0.1 && \
Review comment:
Yeah. I will optimise it a bit more - I will describe in more general terms
in a moment where i am heading to at the moment and where I am now :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
