potiuk commented on PR #35026: URL: https://github.com/apache/airflow/pull/35026#issuecomment-1775028567
> Thank you @potiuk for this detailed comment on how the Dockerfile is built. I came up with the following ([d952c9c](https://github.com/apache/airflow/commit/d952c9c5b1464087ad8e0bf510b84a71bcb49afc)): > > To invalidate cache between two different python versions, I am using the `id` param from `--mount=type=cache` [argument](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#run---mounttypecache). This allow to specify the version of python and the architecture on which it is built. I hope this will solve the issue of cached invalidation you mentioned above. > > Result : > > ```js > => [airflow-build-image 15/16] RUN --mount=type=cache,[id=dockerhub.containers.mpi-internal.com/python:3.11-slim-bullseye-arm64,target=/home/airflow/.cache/pip,uid=50000](http://id%3Ddockerhub.containers.mpi-internal.com/python:3.11-slim-bullseye-arm64,target=/home/airflow/.cache/pip,uid=50000) if [[ false == "true" ]]; then bash /scr 7.7s > ``` Yep. I like this approach a lot. If we make it a little more elaborated, I am perfectly ok with it and solves all my caching problems. There are two things I would like to add: * We should also add AIRFLOW_PIP_VERSION ARG part of the ID * Also similarly to https://github.com/apache/airflow/blob/e19ce433be66015aaa87b566d94a66dfb360260f/Dockerfile.ci#L1292 we can add (for example) "PIP_CACHE_EPOCH" arg to be part of the Dockerfile and also make it part of the ID - this way we will be able to invalidate all caches with simply bumping the PIP_CACHE_EPOCH default value or passing a new value with --build-arg And of course we need to add documentation to https://github.com/apache/airflow/blob/main/docs/docker-stack/build.rst and https://github.com/apache/airflow/blob/main/docs/docker-stack/build-arg-ref.rst and release notes here: https://github.com/apache/airflow/blob/main/docs/docker-stack/changelog.rst (targetting it for 2.7.3) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
