potiuk commented on issue #4543: [AIRFLOW-3718] Multi-layered version of the docker image URL: https://github.com/apache/airflow/pull/4543#issuecomment-456029101 Here are some examples from the POC I set-up in my own repo. Those are three different builds in Docker Hub and they are build automatically on push, I hope you do not need any permissions to see it - if you won't see it, I can copy the logs elsewhere: 1) "latest" build using previously cached layers: https://cloud.docker.com/repository/registry-1.docker.io/potiuk/airflow/builds/c68a9564-f40a-492f-8cb3-d7126af73e12 It was built using previous "latest" as cache. You can see that only sources of airflow changed, no setup.py related files changed. Since last "clean" build there were 3 packages released in ubuntu: They are automatically upgraded with `apt-get upgrade`. Those are security fixes + timezone change: > The following packages will be upgraded: > libsystemd0 libudev1 tzdata > 3 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. > Need to get 679 kB of archives. > After this operation, 14.3 kB of additional disk space will be used. After that `pip install -e` was run again and all the requirements were "already satisified". It took 5 minutes to run the build, it is incremental difference vs. previous one - less than < 1MB to download by anyone who already downloaded previous "latest" image. 2) Then (result of the same push) we have "latest-clean" build - rebuilt from scratch: https://cloud.docker.com/repository/registry-1.docker.io/potiuk/airflow/builds/b8e1ac70-a8fa-49a3-93ad-d27a43670e1c. --no-cache directive was used to build that one (based on the latest-clean image tag). It took 30 minutes to build. This is equivalent to the current build from master. It's almost 1GB to download for anyone who wants to get this version and already downloaded previous "latest-clean" - and that's how it should be. 3) Then we have "release build" - I pushed an artificial tag 1.10.99999 which resulted in `release-1.10.9999` image. It is built from the scratch as well based on "release-.*" regexp matching the tag: https://cloud.docker.com/repository/registry-1.docker.io/potiuk/airflow/builds/12ede2d4-235d-4e1e-a127-7349bb94abce . This one is only built if you push a tag following the regexp "^[0-9].+". Here is the screenshot of the DockerHub config I use (in apache repo the source should eventually be "master" rather than "multi-layered-dockerfile" as is in my POC): <img width="1077" alt="screenshot 2019-01-20 at 23 34 20" src="https://user-images.githubusercontent.com/595491/51445999-0871ce80-1d0c-11e9-9493-86c0ac53c68f.png";>
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
![https://user-images.githubusercontent.com/595491/51445999-0871ce80-1d0c-11e9-9493-86c0ac53c68f.png"](https://user-images.githubusercontent.com/595491/51445999-0871ce80-1d0c-11e9-9493-86c0ac53c68f.png"){kind=link}