potiuk commented on PR #35026: URL: https://github.com/apache/airflow/pull/35026#issuecomment-1770749079
> Also, I don't have the feeling that your [CI is using cache layers](https://github.com/apache/airflow/actions/runs/6572143194/job/17852736374#step:6:697) even though [it should?](https://github.com/apache/airflow/blob/5c2dc53bcb17ae515f9565c41d15cc6d1693382c/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py#L445). As @hussein-awala said, once buildkit will have implemented a way to use `binded cache` in CI, it would also help to improve build speed It's because you were extremely unlucky when checking it. Python **just** released a new base image 4 hours ago, and our cache will refresh with the new base image after we merge something to main and the build in `main` succeeds:  The way how we are using remote caching is specifically designed to automatically handle such cache refreshes. The whole CI of ourse is designed to automatically upgrade to latest versions of the base images, apt dependencies and Python dependencies as soon as: * they are released * all our tests pass in main (also called `canary`) run This allows us to automtically upgrade and test our software and test suite to "latest", "most secure" and "consistent" set of dependencies (including base dependencies for the image), while our PRs keeps on using the "stable and confirmed to work in the past" set of python dependencies. This way we can detect breaking changes in incoming dependencies. We detected many breaking changes this way "early" - hence `canary` name (including even couple of breaking changes in Python patchlevel releases) Indeed the build you see does not use the cache that corresponds to the latest pyhon version that's why it takes 14 minutes. Because cache is not yet refreshed. But it still works (which is great) while the cache is being rebuilt and refreshed. You can see the same job from 5 hours ago https://github.com/apache/airflow/actions/runs/6565914251/job/17835531939#step:6:1070 which lasted 1 minute and actually used the cache. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
