potiuk edited a comment on issue #4543: [AIRFLOW-3718] [WIP] Multi-layered version of the docker image URL: https://github.com/apache/airflow/pull/4543#issuecomment-470036483 @ashb @Fokko # Getting greener I am on the path to get green Travis CI builds :) : https://travis-ci.org/potiuk/airflow/jobs/502325942 There are still some tests failing due to differences in environment (SSH authentication / KADM/ HADOOP tests) but in general the whole toolchain works without TOX and uses Docker images as cache layer (efficiently). # Timing Some preliminary timing (this is the most common case with only changed sources and no setup.py changes / no new dependencies): - Preparing the Travis CI worker **28 seconds** - (basically installing latest docker compose): - similar to what we have in current build. - Pulling airflow images **142 s** (all images used by docker_compose including mysql/postgres and others) - that's similar to what we have on current build. - Rebuilding local Airflow Travis CI image **42 seconds** (checking via docker cache that the base images do not need to be rebuilt (no changes in Dockerfiles), pulling cache image with .wheels, running docker build locally - checks if new packages need to be installed, rebuild latest layers with changed sources in the image, running `pip install -e . ['devel_ci']` and checking that no requirements changed): (!). (in those 42 seconds pulling .wheel images takes 12 s). This will still be faster when I decrease number of Dockerfiles/images. Then the tests start immediately using prepared image, no further installation/setup etc. happens inside the running airflow-testing container. You can see the timings here: <img width="1121" alt="screenshot 2019-03-06 at 10 03 38" src="https://user-images.githubusercontent.com/595491/53870967-8c052780-3ffb-11e9-9e8b-79010d56a319.png";> # Docker Hub builds In case when the build will run from master branch. DockerHub will run it's own build when it will rebuild the airflow images used for future Travis builds - it will continue to catch-up with latest master sources and requirements. You can see such builds here: https://cloud.docker.com/repository/docker/potiuk/airflow/timeline They take around 15 minutes on DockerHub slow machines. And sometimes they are delayed by many hours (last two builds were queued for 8 hours). However this is not a big issue as the whole system will work incrementally using latest available builds as base and it will get eventually consistent: 
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
![https://user-images.githubusercontent.com/595491/53870967-8c052780-3ffb-11e9-9e8b-79010d56a319.png"](https://user-images.githubusercontent.com/595491/53870967-8c052780-3ffb-11e9-9e8b-79010d56a319.png"){kind=link}
