o-nikolas commented on code in PR #23391:
URL: https://github.com/apache/airflow/pull/23391#discussion_r865174249
##########
docs/docker-stack/build.rst:
##########
@@ -23,8 +23,103 @@ Building the image
Before you dive-deeply in the way how the Airflow Image is built, let us first
explain why you might need
to build the custom container image and we show a few typical ways you can do
it.
-Why custom image ?
-------------------
+Quick start scenarios of image extending
+----------------------------------------
+
+The most common scenarios where you want to build your own image are adding a
new ``apt`` package,
+adding a new ``PyPI`` dependency and embedding DAGs into the image.
+Example Dockerfiles for those scenarios are below, and you can read further
+for more complex cases which might involve either extending or customizing the
image. You will find
+more information about more complex scenarios below, but if your goal is to
quickly extend airflow
+image with new provider, package, etc. then here is a quick start for you.
+
+Adding new ``apt`` package
+..........................
+
+The following example adds ``vim`` to the airflow image. When adding packages
via ``apt`` you should
+switch to ``root`` user for the time of installation, but do not forget to
switch back to the
+``airflow`` user after installation is complete.
+
+.. exampleinclude:: docker-examples/extending/add-apt-packages/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+
+Adding a new ``PyPI`` package
+.............................
+
+The following example adds ``lxml`` python package from PyPI to the image.
When adding packages via
+``pip`` you need to use ``airflow`` user rather than ``root``. Attempts to
install ``pip`` packages
+with root, when you using typical ``pip install`` command will fail with
appropriate error message.
Review Comment:
```suggestion
with root, when using typical ``pip install`` command, will fail with an
appropriate error message.
```
##########
dev/breeze/src/airflow_breeze/commands/release_management.py:
##########
@@ -106,6 +99,32 @@
],
},
],
+ "breeze release-prod-images": [
+ {
+ "name": "Release PROD IMAGE params",
Review Comment:
Nit: Looks like most of the others go with "flags" instead of "params"
##########
docs/docker-stack/build.rst:
##########
@@ -115,109 +210,63 @@ In the simplest case building your image consists of
those steps:
of storing and exposing the images, and it is most portable way of
publishing the image. Both
Docker-Compose and Kubernetes can make use of images exposed via registries.
-The most common scenarios where you want to build your own image are adding a
new ``apt`` package,
-adding a new ``PyPI`` dependency and embedding DAGs into the image.
-Example Dockerfiles for those scenarios are below, and you can read further
-for more complex cases which might involve either extending or customizing the
image.
-Adding new ``apt`` package
-..........................
-
-The following example adds ``vim`` to the airflow image. When adding packages
via ``apt`` you should
-switch to ``root`` user for the time of installation, but do not forget to
switch back to the
-``airflow`` user after installation is complete.
-
-.. exampleinclude:: docker-examples/extending/add-apt-packages/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
-
-
-Adding a new ``PyPI`` package
-.............................
-
-The following example adds ``lxml`` python package from PyPI to the image.
When adding packages via
-``pip`` you need to use ``airflow`` user rather than ``root``. Attempts to
install ``pip`` packages
-with root, when you using typical ``pip install`` command will fail with
appropriate error message.
-
-.. exampleinclude:: docker-examples/extending/add-pypi-packages/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
-
-Embedding DAGs
-..............
-
-The following example adds ``test_dag.py`` to your image in the
``/opt/airflow/dags`` folder.
-
-.. exampleinclude:: docker-examples/extending/embedding-dags/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
-
-
-.. exampleinclude:: docker-examples/extending/embedding-dags/test_dag.py
- :language: Python
- :start-after: [START dag]
- :end-before: [END dag]
+Extending the image
+-------------------
-Extending vs. customizing the image
------------------------------------
+Extending the image is easiest if you just need to add some dependencies that
do not require
+compiling. The compilation framework of Linux (so called ``build-essential``)
is pretty big, and
+for the production images, size is really important factor to optimize for, so
our Production Image
+does not contain ``build-essential``. If you need compiler like gcc or g++ or
make/cmake etc. - those
+are not found in the image and it is recommended that you follow the
"customize" route instead.
-You might want to know very quickly how you can extend or customize the
existing image
-for Apache Airflow. This chapter gives you a short answer to those questions.
+How to extend the image - it is something you are most likely familiar with -
simply
+build a new image using Dockerfile's ``FROM`` directive and add whatever you
need. Then you can add your
+Debian dependencies with ``apt`` or PyPI dependencies with ``pip install`` or
any other stuff you need.
+Base images
+...........
-Here is the comparison of the two types of building images. Here is your guide
if you want to choose
-how you want to build your image.
+There are two types if images you can extend your image from:
-+----------------------------------------------------+-----------+-------------+
-| | Extending | Customizing
|
-+====================================================+===========+=============+
-| Uses familiar 'FROM ' pattern of image building | Yes | No
|
-+----------------------------------------------------+-----------+-------------+
-| Requires only basic knowledge about images | Yes | No
|
-+----------------------------------------------------+-----------+-------------+
-| Builds quickly | Yes | No
|
-+----------------------------------------------------+-----------+-------------+
-| Produces image heavily optimized for size | No | Yes
|
-+----------------------------------------------------+-----------+-------------+
-| Can build from custom airflow sources (forks) | No | Yes
|
-+----------------------------------------------------+-----------+-------------+
-| Can build on air-gaped system | No | Yes
|
-+----------------------------------------------------+-----------+-------------+
+1) "Standard" airflow image that contains most useful extras and providers,
supports mysql, postgres, mssql
-TL;DR; If you have a need to build custom image, it is easier to start with
"Extending" however if your
-dependencies require compilation step or when your require to build the image
from security vetted
-packages, switching to "Customizing" the image provides much more optimized
images. In the example further
-where we compare equivalent "Extending" and "Customizing" the image, similar
images build by
-Extending vs. Customization had shown 1.1GB vs 874MB image sizes respectively
- with 20% improvement in
-size of the Customized image.
+2) "Slim" airflow image which is a minimum image you can start from - it
contains just one of the supported
+ database clients installed (Postgres/MySQL/MsSQL) and no extras, no
providers installed. This image is
+ small comparing to standard image (~NN MB) and you might need to add a lot
more packages and providers
+ in order to make it useful for your case (but if you use only a small
subset of providers and one
+ database only - which most of the users do, it might be a good starting
point for you). The slim images
+ might have - dependencies in different versions than those used when
providers are preinstalled, simply
+ because core airflow might have less limits on the versions on its own.
When you install some providers
+ they might require downgrading some dependencies if the providers require
different limits for the
+ same dependencies.
-.. note::
+Naming conventions for those images:
- You can also combine both - customizing & extending the image in one. You
can build your
- optimized base image first using ``customization`` method (for example by
your admin team) with all
- the heavy compilation required dependencies and you can publish it in your
registry and let others
- ``extend`` your image using ``FROM`` and add their own lightweight
dependencies. This reflects well
- the split where typically "Casual" users will Extend the image and
"Power-users" will customize it.
++-----------------------+-------------------+----------------------------------+------------------------------------------------+
+| Image type | Python | Standard image
| Slim image |
++=======================+===================+==================================+================================================+
+| Latest default image | 3.7 | apache/airflow:latest
| apache/airflow:slim-DATABASE-latest |
+| Default image | 3.7 | apache/airflow:X.Y.Z
| apache/airflow:slim-DATABASE-X.Y.Z |
+| Latest regular image | 3.7,3.8,3.8,3.10 | apache/airflow:latest-pythonN.M
| apache/airflow:slim-DATABASE-latest-pythonN.M |
+| Regular image | 3.7,3.8,3.8,3.10 | apache/airflow:X.Y.Z-pythonN.M
| apache/airflow:slim-DATABASE-X.Y.Z-pythonN.M |
++-----------------------+-------------------+----------------------------------+------------------------------------------------+
-Airflow Summit 2020's `Production Docker Image
<https://youtu.be/wDr3Y7q2XoI>`_ talk provides more
-details about the context, architecture and customization/extension methods
for the Production Image.
+* The "latest" image is always latest released stable version available.
+* the DATABASE: one of mysql, postgres, mssql
+The platform support for slim images depends on the database used. While for
"regular" images, there
+is a support for both AMD64 and ARM64 platform (where ARM64 images do not
support MySQL and MsSQL), for
Review Comment:
```suggestion
is support for both AMD64 and ARM64 platform (where ARM64 images do not
support MySQL and MsSQL), for
```
##########
dev/breeze/src/airflow_breeze/commands/release_management.py:
##########
@@ -378,6 +398,228 @@ def generate_constraints(
sys.exit(return_code)
+def update_build_args_for_backend(installed_backend: str, build_args:
Dict[str, str]):
+ for backend in ALLOWED_PROD_BACKENDS:
+ build_args[f'INSTALL_{backend.upper()}'] = "true" if installed_backend
== backend else "false"
+
+
+def get_platform_per_backend(installed_backend: str, platform: str) -> str:
+
+ if installed_backend in ['mysql', 'mssql']:
+ if platform != "linux/amd64":
+ platform = "linux/amd64"
+ get_console().print(f"[warning]For {installed_backend} the
platform is forced to {platform}")
+ return platform
+
+
+def convert_build_args_dict_to_array_of_args(build_args: Dict[str, str]) ->
List[str]:
+ array_of_args = []
+ for key, value in build_args.items():
+ array_of_args.append("--build-arg")
+ array_of_args.append(f'{key}={value}')
+ return array_of_args
+
+
+def alias_image(image_from: str, image_to: str, dry_run: bool, verbose: bool):
+ get_console().print(f"[info]Creating {image_to} alias for {image_from}[/]")
+ run_command(["regctl", "image", "copy", image_from, image_to],
dry_run=dry_run, verbose=verbose)
+
+
[email protected](
Review Comment:
I've not used click before, it looks pretty cool :)
##########
BREEZE.rst:
##########
@@ -917,6 +918,34 @@ Those are all available flags of ``verify-prod-image``
command:
:width: 100%
:alt: Breeze verify-prod-image
+Releasing Production images to DockerHub
+----------------------------------------
+
+The **Production image** can be released by release managers who have
permissions to push the image. This
+happens only when there is an RC candidate or final released version of
Airflow released.
+
+Typically you can release "regular" and "slim" images separately.
+
+Releasing "regular" images:
+
+.. code-block:: bash
+
+ breeze release-prod-images --airflow-version 2.4.0
+
+Or "slim" images
+
+.. code-block:: bash
+
+ breeze release-prod-images --airflow-version 2.4.0 --slim-images
+
+By default when you are releasing the "final" image, we also tag image with
"latest" tags but this
+step can be skipped if you pas ``--skip-latest`` flag.
Review Comment:
s/pas/pass/g
##########
dev/README_RELEASE_AIRFLOW.md:
##########
@@ -998,19 +1008,28 @@ At this point we release an official package:
## Manually prepare production Docker Image
-
Note that this scripts prepares multi-platform image, so you need to fulfill
prerequisites as
described above in the preparation of RC images.
+Note that the `--latest` flag below should be used when you are releasing the
latest images only (which
+should be the default). For example when we are releasing 2.3.N image and 2.3
is our latest branch we
+should add "latest" to update "latest" aliases for our images.
+
+In case we are releasing (which almost never happens) a critical bugfix
release in one of the older branches,
+we should drop the `--latest` flag.
+
+Preparing regular images:
+
+```shell script
+breeze release-prod-images --airflow-version "${VERSION}" --latest
+```
+
+Preparing slim images
Review Comment:
```suggestion
Preparing slim images:
```
##########
docs/docker-stack/build.rst:
##########
@@ -115,109 +210,63 @@ In the simplest case building your image consists of
those steps:
of storing and exposing the images, and it is most portable way of
publishing the image. Both
Docker-Compose and Kubernetes can make use of images exposed via registries.
-The most common scenarios where you want to build your own image are adding a
new ``apt`` package,
-adding a new ``PyPI`` dependency and embedding DAGs into the image.
-Example Dockerfiles for those scenarios are below, and you can read further
-for more complex cases which might involve either extending or customizing the
image.
-Adding new ``apt`` package
-..........................
-
-The following example adds ``vim`` to the airflow image. When adding packages
via ``apt`` you should
-switch to ``root`` user for the time of installation, but do not forget to
switch back to the
-``airflow`` user after installation is complete.
-
-.. exampleinclude:: docker-examples/extending/add-apt-packages/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
-
-
-Adding a new ``PyPI`` package
-.............................
-
-The following example adds ``lxml`` python package from PyPI to the image.
When adding packages via
-``pip`` you need to use ``airflow`` user rather than ``root``. Attempts to
install ``pip`` packages
-with root, when you using typical ``pip install`` command will fail with
appropriate error message.
-
-.. exampleinclude:: docker-examples/extending/add-pypi-packages/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
-
-Embedding DAGs
-..............
-
-The following example adds ``test_dag.py`` to your image in the
``/opt/airflow/dags`` folder.
-
-.. exampleinclude:: docker-examples/extending/embedding-dags/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
-
-
-.. exampleinclude:: docker-examples/extending/embedding-dags/test_dag.py
- :language: Python
- :start-after: [START dag]
- :end-before: [END dag]
+Extending the image
+-------------------
-Extending vs. customizing the image
------------------------------------
+Extending the image is easiest if you just need to add some dependencies that
do not require
+compiling. The compilation framework of Linux (so called ``build-essential``)
is pretty big, and
+for the production images, size is really important factor to optimize for, so
our Production Image
+does not contain ``build-essential``. If you need compiler like gcc or g++ or
make/cmake etc. - those
+are not found in the image and it is recommended that you follow the
"customize" route instead.
-You might want to know very quickly how you can extend or customize the
existing image
-for Apache Airflow. This chapter gives you a short answer to those questions.
+How to extend the image - it is something you are most likely familiar with -
simply
+build a new image using Dockerfile's ``FROM`` directive and add whatever you
need. Then you can add your
+Debian dependencies with ``apt`` or PyPI dependencies with ``pip install`` or
any other stuff you need.
+Base images
+...........
-Here is the comparison of the two types of building images. Here is your guide
if you want to choose
-how you want to build your image.
+There are two types if images you can extend your image from:
Review Comment:
```suggestion
There are two types of images you can extend your image from:
```
##########
dev/README_RELEASE_AIRFLOW.md:
##########
@@ -998,19 +1008,28 @@ At this point we release an official package:
## Manually prepare production Docker Image
-
Note that this scripts prepares multi-platform image, so you need to fulfill
prerequisites as
described above in the preparation of RC images.
+Note that the `--latest` flag below should be used when you are releasing the
latest images only (which
+should be the default). For example when we are releasing 2.3.N image and 2.3
is our latest branch we
+should add "latest" to update "latest" aliases for our images.
+
+In case we are releasing (which almost never happens) a critical bugfix
release in one of the older branches,
+we should drop the `--latest` flag.
Review Comment:
Is it worth considering to invert the flag? If latest is by far the most
common, then it doesn't make sense to pass a flag for it every time, it could
just be assumed. And then in the rare cases when it's not latest, then pass a
flag to indicate that? Though, the name of such a flag is harder to come up
with :laughing:
##########
dev/README_RELEASE_AIRFLOW.md:
##########
@@ -540,9 +543,16 @@ docker buildx ls
airflow_cache1 tcp://127.0.0.1:2375
```
+Preparing regular images:
```shell script
-./dev/prepare_prod_docker_images.sh ${VERSION}
+breeze release-prod-images --airflow-version "${VERSION}"
+```
+
+Preparing slim images
Review Comment:
```suggestion
Preparing slim images:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
