potiuk commented on pull request #17635: URL: https://github.com/apache/airflow/pull/17635#issuecomment-900175666
The nice side-effect of this approach is that now we can do "eager-upgrade" in both - "schedule" and "push" runs. This means that constraints might get upgraded even if there is not a single PR merged - by scheduled build. The images for "schedule" builds are always build from scratch (because then we pull latest Python base image and test if the Dockerflles are still working "from scratch". Previously we could not really push those images as cache (because they were built from scratch). Now - even in scheduled build - we are rebuilding the images from cache in this "push cache images" step (using the generated constraints) so we will not loose cache. But what we also do at those rebuilds (and this is also smarter than before) we check if there were newer python base images released in the meantime and we pull them when we rebuild the 'cache image' if they did - which is the "smoothest" way I could figure our we can automatically updated to the latest base images automatically. Previously when we upgraded to latest python base image, there was potentially long-ish period of time where base python image and latest cache image were not synchronized (Resulting in way longer build-image step > 10 minutes rather than ~1 minute). That made Breeze sometime use different python base image and not rebuilding as quickly as possible in development environment as it could. With the current approach we will only ever push the latest base python image and cache image together when the latest python base image changed. There are still a few edge cases where this might cause longer rebuilds than needed but I have some ideas how to get rid of those as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
