potiuk commented on a change in pull request #19189:
URL: https://github.com/apache/airflow/pull/19189#discussion_r737936033
##########
File path: Dockerfile
##########
@@ -183,16 +183,19 @@ ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT} \
AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE} \
AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION} \
DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH} \
- PATH=${PATH}:/root/.local/bin \
+ PATH=/.venv/bin:${PATH} \
AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \
PIP_PROGRESS_BAR=${PIP_PROGRESS_BAR} \
- # Install Airflow with "--user" flag, so that we can copy the whole .local
folder to the final image
+ # Install Airflow in venv, so that we can copy the whole .venv folder to
the final image
# from the build image and always in non-editable mode
- AIRFLOW_INSTALL_USER_FLAG="--user" \
AIRFLOW_INSTALL_EDITABLE_FLAG="" \
UPGRADE_TO_NEWER_DEPENDENCIES=${UPGRADE_TO_NEWER_DEPENDENCIES}
-COPY scripts/docker/*.sh /scripts/docker/
+# Only copy mysql/mssql installation scripts for now - so that changing the
other
+# scripts which are needed much later will not invalidate the docker layer here
+COPY scripts/docker/common.sh scripts/docker/install_mysql.sh
scripts/docker/install_mssql.sh \
+ /scripts/docker/
Review comment:
This optimisation is not minor. It's crucial for fast iterations on the
image. Previously, if you changed just one script (which is used many steps
later for installing airflow), the docker image will be invalidated at COPY
scrpts/docker/*.sh and every time your run make a typo in the script you have
to wait 30-40 seconds or more until mysql and mssql is installed. With the
aproach above I was able to signficantly speed up iteration speed whenever any
of the ramaining scripts changed.
It has no implications overall- it increases number of layers and slightly
increases the size of the "build segment" but since build segment is used only
temporarily to be source of airflow venv, it has no impact whatsoever on the
final image.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
