potiuk commented on issue #4543: [AIRFLOW-3718] Multi-layered version of the docker image URL: https://github.com/apache/airflow/pull/4543#issuecomment-455998180 Ok. I will show you some working examples/POC shortly of where the full control over caching works with real DockerHub setup. I takes a bit of time to iterate with DockerHub builds as they take some time, but I think it will get there shortly. Just to summarise what I will have now: I will show you running examples where we can see that we can see with every commit that the build from scratch still works (:latest-clean), where the (:latest) build is usually incremental for the benefit of the downloading users. The latest will be periodically refreshed from scratch whenever any of the events happens: - base image is updated (with --pull flag of the build) - whole build is rerun - setup.py changes (pip install is run from the scratch) - we decide to force rebuild it from the scratch by incrementing the variable in Dockerfile - I will show an example where `apt-get upgrade` makes perfect sense. In one of the latest builds I made there were two security fixes and a timezone fix applied thanks to running `apt-get upgrade` - it took like 4 minutes to build and likely few MB more to download the latest layer in order to get the latest security fixes in the latest build. I hope that might be a good solution that addresses all your concerns and my concern about full downloads with every commit by the users. And again - I can add it as experimental for the time being, but I think it's worth to merge it to see how it behaves with real usage.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
