potiuk commented on a change in pull request #4543: [AIRFLOW-3718] [WIP/SPLIT]
Multi-layered version of the docker image
URL: https://github.com/apache/airflow/pull/4543#discussion_r267810916
##########
File path: hooks/build
##########
@@ -144,6 +144,14 @@ echo
# So for example 'latest-3.6' will produce PYTHON_VERSION=3.6
export PYTHON_VERSION=$(echo "${LOCAL_IMAGE}" | rev | cut -d '-' -f1 | rev )
+
+# In case of CRON jobs on Travis we run builds without cache
Review comment:
Here is an example of such Cron job @Fokko:
https://travis-ci.org/potiuk/airflow/jobs/509211696. It can run daily on travis
and it will run all the tests on a "from the scratch" build. It takes 7:20
minutes to build the image from scratch on Travis worker (per job) but we have
clear indication whether clean build of Docker works from the scratch with
latest apt-get and pip dependencies and whether all tests are successful.
This way all the PR will use much faster built incremental image, but we
also have full, longer build that is performed automatically and regularly.
At the same time such daily build - since it will be build daily (for
example) might be an early indication that one of the transitive dependencies
is now failing the installation (but it will not disrupt tests for opened PRs
because they will use latest "correctly building" dependencies from the last
successful master build.
I think this is a very nice way of addressing both problems - stability for
people working on their PRs and assurance for maintainers that the build is
"safe".
I think if we do it this way, we might even not need to solve the transient
dependencies case for work in progress - we could focus only on making sure
that there is a stable set of dependencies for official releases somehow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
