potiuk commented on a change in pull request #20664:
URL: https://github.com/apache/airflow/pull/20664#discussion_r782049978
##########
File path: IMAGES.rst
##########
@@ -502,116 +487,58 @@ This builds the CI image in version 3.7 with default
extras ("all").
.. code-block:: bash
- docker build . -f Dockerfile.ci --build-arg
PYTHON_BASE_IMAGE="python:3.7-slim-buster" --tag my-image:0.0.1
+ DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \
+ --pull \
+ --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" --tag
my-image:0.0.1
This builds the CI image in version 3.6 with "gcp" extra only.
.. code-block:: bash
- docker build . -f Dockerfile.ci --build-arg
PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
+ DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \
+ --pull \
+ --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg AIRFLOW_EXTRAS=gcp --tag my-image:0.0.1
This builds the CI image in version 3.6 with "apache-beam" extra added.
.. code-block:: bash
- docker build . -f Dockerfile.ci --build-arg
PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
+ DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \
+ --pull \
+ --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="apache-beam" --tag my-image:0.0.1
This builds the CI image in version 3.6 with "mssql" additional package added.
.. code-block:: bash
- docker build . -f Dockerfile.ci --build-arg
PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
+ DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \
+ --pull \
+ --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg ADDITIONAL_PYTHON_DEPS="mssql" --tag my-image:0.0.1
This builds the CI image in version 3.6 with "gcc" and "g++" additional apt
dev dependencies added.
.. code-block::
- docker build . -f Dockerfile.ci --build-arg
PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
+ DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \
+ --pull
+ --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg ADDITIONAL_DEV_APT_DEPS="gcc g++" --tag my-image:0.0.1
This builds the CI image in version 3.6 with "jdbc" extra and
"default-jre-headless" additional apt runtime dependencies added.
.. code-block::
- docker build . -f Dockerfile.ci --build-arg
PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
+ DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \
+ --pull \
+ --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg AIRFLOW_EXTRAS=jdbc --build-arg
ADDITIONAL_RUNTIME_DEPS="default-jre-headless" \
--tag my-image:0.0.1
-CI Image manifests
Review comment:
No need for image manifest either :). We needed it to be able to
determine how "big" the difference is of the current Dockerfile /image we had
locally vs the one available as cache. This was mainly because of catch-22 -
you had to pull the image first to use it as cache, and you could not really
inspect the content of the remote image before pulling it without
authenticating (no public API for that). The manifest was really implemented
to overcome that limitation.
With Buildkit cache, each layer is checked separataly and independently
from each other without actually pulling the image. This is nicely visualized
during the build process where CACHE/REBUILD status is shown very quickly
without actually pulling any image. This way we completely do not have to worry
how "big" the difference is vs locally rebuilt image. If the difference is
"big" the necessary layers will be pulled, if we have modified the image/copied
files in the way that they are different than cache - they will be rebuilt. All
super-optimised.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
