potiuk commented on a change in pull request #4938: [AIRFLOW-4117]
Multi-staging Image - Travis CI tests [Step 3/3]
URL: https://github.com/apache/airflow/pull/4938#discussion_r297407943
##########
File path: Dockerfile
##########
@@ -278,6 +278,34 @@ RUN echo "Pip version: ${PIP_VERSION}"
RUN pip install --upgrade pip==${PIP_VERSION}
+ARG AIRFLOW_REPO=apache/airflow
+ENV AIRFLOW_REPO=${AIRFLOW_REPO}
+
+ARG AIRFLOW_BRANCH=master
+ENV AIRFLOW_BRANCH=${AIRFLOW_BRANCH}
+
+ENV
AIRFLOW_GITHUB_DOWNLOAD=https://raw.githubusercontent.com/${AIRFLOW_REPO}/${AIRFLOW_BRANCH}
+# Get basic dependencies from the master version of Airflow in order to avoid
cache invalidation
Review comment:
The main reason is to have a "seed" for pip install . Using master from
airflow is a good way to populate "most" of dependencies.
This way we have a "base" layer that will not be rebuilt every time (only
when EPOCH will be updated and when you run a "no-cache" build - daily build).
This really won't be a problem in regular builds - this layer will not be
invalidated after it's built. But I tested that if you do afterwards `pip
instal --upgrade [ALL installed packages]` followed by `pip install -e
.[EXTRAS]` pip will install the latest deps and then downgrade those which are
limited in setup.py . I found this really good - this way we will always test
against the latest deps that are following setup.py limitations. And it is
super-optimised for build time.
I can add more description if you think it makes sense @Fokko
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
