jedcunningham commented on code in PR #23391:
URL: https://github.com/apache/airflow/pull/23391#discussion_r866061961
##########
BREEZE.rst:
##########
@@ -917,6 +918,34 @@ Those are all available flags of ``verify-prod-image``
command:
:width: 100%
:alt: Breeze verify-prod-image
+Releasing Production images to DockerHub
+----------------------------------------
+
+The **Production image** can be released by release managers who have
permissions to push the image. This
+happens only when there is an RC candidate or final released version of
Airflow released.
+
+Typically you can release "regular" and "slim" images separately.
+
+Releasing "regular" images:
+
+.. code-block:: bash
+
+ breeze release-prod-images --airflow-version 2.4.0
+
+Or "slim" images
+
+.. code-block:: bash
+
+ breeze release-prod-images --airflow-version 2.4.0 --slim-images
+
+By default when you are releasing the "final" image, we also tag image with
"latest" tags but this
+step can be skipped if you pass ``--skip-latest`` flag.
+
+Those are all available flags of ``release-prod-images`` command:
Review Comment:
```suggestion
These are of the all available flags for the ``release-prod-images`` command:
```
##########
dev/README_RELEASE_AIRFLOW.md:
##########
@@ -998,19 +1008,34 @@ At this point we release an official package:
## Manually prepare production Docker Image
-
Note that this scripts prepares multi-platform image, so you need to fulfill
prerequisites as
described above in the preparation of RC images.
+Note that by default the `latest` images tagged are aliased to the just
released image which is the usual
+way we release. For example when you are releasing 2.3.N image and 2.3 is our
latest branch the new image is
+marked as "latest".
+
+In case we are releasing (which almost never happens so far) a critical bugfix
release in one of
+the older branches, you should add `--skip-latest` flag.
Review Comment:
```suggestion
the older branches, you should add the `--skip-latest` flag.
```
##########
BREEZE.rst:
##########
@@ -917,6 +918,34 @@ Those are all available flags of ``verify-prod-image``
command:
:width: 100%
:alt: Breeze verify-prod-image
+Releasing Production images to DockerHub
+----------------------------------------
+
+The **Production image** can be released by release managers who have
permissions to push the image. This
+happens only when there is an RC candidate or final released version of
Airflow released.
+
+Typically you can release "regular" and "slim" images separately.
+
+Releasing "regular" images:
+
+.. code-block:: bash
+
+ breeze release-prod-images --airflow-version 2.4.0
+
+Or "slim" images
+
+.. code-block:: bash
+
+ breeze release-prod-images --airflow-version 2.4.0 --slim-images
+
+By default when you are releasing the "final" image, we also tag image with
"latest" tags but this
+step can be skipped if you pass ``--skip-latest`` flag.
Review Comment:
```suggestion
step can be skipped if you pass the ``--skip-latest`` flag.
```
##########
dev/breeze/src/airflow_breeze/commands/release_management.py:
##########
@@ -374,6 +393,205 @@ def generate_constraints(
sys.exit(return_code)
+def convert_build_args_dict_to_array_of_args(build_args: Dict[str, str]) ->
List[str]:
+ array_of_args = []
+ for key, value in build_args.items():
+ array_of_args.append("--build-arg")
+ array_of_args.append(f'{key}={value}')
+ return array_of_args
+
+
+def alias_image(image_from: str, image_to: str, dry_run: bool, verbose: bool):
+ get_console().print(f"[info]Creating {image_to} alias for {image_from}[/]")
+ run_command(
+ ["regctl", "image", "copy", "--force-recursive", "--digest-tags",
image_from, image_to],
+ dry_run=dry_run,
+ verbose=verbose,
+ )
+
+
[email protected](
+ name="release-prod-images", help="Release production images to DockerHub
(needs DockerHub permissions)."
+)
[email protected]('--airflow-version', required=True, help="Airflow version to
release (2.3.0, 2.3.0rc1 etc.)")
[email protected](
+ '--dockerhub-repo',
+ default="apache/airflow",
+ show_default=True,
+ help="DockerHub repository for the images",
+)
[email protected](
+ '--slim-images',
+ is_flag=True,
+ help='Whether to prepare slim images rather than regular ones.',
+)
[email protected](
+ '--limit-python',
+ type=BetterChoice(CURRENT_PYTHON_MAJOR_MINOR_VERSIONS),
+ help="Specific python to build slim images for (if not specified - the
images are build for all"
Review Comment:
```suggestion
help="Specific python to build slim images for (if not specified - the
images are built for all"
```
##########
BREEZE.rst:
##########
@@ -917,6 +918,34 @@ Those are all available flags of ``verify-prod-image``
command:
:width: 100%
:alt: Breeze verify-prod-image
+Releasing Production images to DockerHub
+----------------------------------------
+
+The **Production image** can be released by release managers who have
permissions to push the image. This
+happens only when there is an RC candidate or final released version of
Airflow released.
+
+Typically you can release "regular" and "slim" images separately.
+
+Releasing "regular" images:
+
+.. code-block:: bash
+
+ breeze release-prod-images --airflow-version 2.4.0
+
+Or "slim" images
Review Comment:
```suggestion
Or "slim" images:
```
##########
docs/docker-stack/build.rst:
##########
@@ -23,8 +23,103 @@ Building the image
Before you dive-deeply in the way how the Airflow Image is built, let us first
explain why you might need
to build the custom container image and we show a few typical ways you can do
it.
-Why custom image ?
-------------------
+Quick start scenarios of image extending
+----------------------------------------
+
+The most common scenarios where you want to build your own image are adding a
new ``apt`` package,
+adding a new ``PyPI`` dependency and embedding DAGs into the image.
+Example Dockerfiles for those scenarios are below, and you can read further
+for more complex cases which might involve either extending or customizing the
image. You will find
+more information about more complex scenarios below, but if your goal is to
quickly extend airflow
+image with new provider, package, etc. then here is a quick start for you.
+
+Adding new ``apt`` package
+..........................
+
+The following example adds ``vim`` to the airflow image. When adding packages
via ``apt`` you should
+switch to ``root`` user for the time of installation, but do not forget to
switch back to the
+``airflow`` user after installation is complete.
+
+.. exampleinclude:: docker-examples/extending/add-apt-packages/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+
+Adding a new ``PyPI`` package
+.............................
+
+The following example adds ``lxml`` python package from PyPI to the image.
When adding packages via
+``pip`` you need to use ``airflow`` user rather than ``root``. Attempts to
install ``pip`` packages
+with root, when you use typical ``pip install`` command will fail with
appropriate error message.
+
+.. exampleinclude:: docker-examples/extending/add-pypi-packages/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+Embedding DAGs
+..............
+
+The following example adds ``test_dag.py`` to your image in the
``/opt/airflow/dags`` folder.
+
+.. exampleinclude:: docker-examples/extending/embedding-dags/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+
+.. exampleinclude:: docker-examples/extending/embedding-dags/test_dag.py
+ :language: Python
+ :start-after: [START dag]
+ :end-before: [END dag]
+
+
+Extending vs. customizing the image
+-----------------------------------
+
+You might want to know very quickly whether you need to extend or customize
the existing image
+for Apache Airflow. This chapter gives you a short answer to those questions.
+
+Here is the comparison of the two types of building images. Here is your guide
if you want to choose
+how you want to build your image.
+
++----------------------------------------------------+-----------+-------------+
+| | Extending | Customizing
|
++====================================================+===========+=============+
+| Uses familiar 'FROM ' pattern of image building | Yes | No
|
++----------------------------------------------------+-----------+-------------+
+| Requires only basic knowledge about images | Yes | No
|
++----------------------------------------------------+-----------+-------------+
+| Builds quickly | Yes | No
|
++----------------------------------------------------+-----------+-------------+
+| Produces image heavily optimized for size | No | Yes
|
++----------------------------------------------------+-----------+-------------+
+| Can build from custom airflow sources (forks) | No | Yes
|
++----------------------------------------------------+-----------+-------------+
+| Can build on air-gaped system | No | Yes
|
++----------------------------------------------------+-----------+-------------+
+
+TL;DR; If you have a need to build custom image, it is easier to start with
"Extending" however if your
Review Comment:
```suggestion
TL;DR; If you have a need to build a custom image, it is easier to start
with "Extending". However, if your
```
##########
docs/docker-stack/build.rst:
##########
@@ -115,109 +210,62 @@ In the simplest case building your image consists of
those steps:
of storing and exposing the images, and it is most portable way of
publishing the image. Both
Docker-Compose and Kubernetes can make use of images exposed via registries.
-The most common scenarios where you want to build your own image are adding a
new ``apt`` package,
-adding a new ``PyPI`` dependency and embedding DAGs into the image.
-Example Dockerfiles for those scenarios are below, and you can read further
-for more complex cases which might involve either extending or customizing the
image.
-Adding new ``apt`` package
-..........................
-
-The following example adds ``vim`` to the airflow image. When adding packages
via ``apt`` you should
-switch to ``root`` user for the time of installation, but do not forget to
switch back to the
-``airflow`` user after installation is complete.
-
-.. exampleinclude:: docker-examples/extending/add-apt-packages/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
-
-
-Adding a new ``PyPI`` package
-.............................
-
-The following example adds ``lxml`` python package from PyPI to the image.
When adding packages via
-``pip`` you need to use ``airflow`` user rather than ``root``. Attempts to
install ``pip`` packages
-with root, when you using typical ``pip install`` command will fail with
appropriate error message.
-
-.. exampleinclude:: docker-examples/extending/add-pypi-packages/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
-
-Embedding DAGs
-..............
-
-The following example adds ``test_dag.py`` to your image in the
``/opt/airflow/dags`` folder.
-
-.. exampleinclude:: docker-examples/extending/embedding-dags/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
+Extending the image
+-------------------
+Extending the image is easiest if you just need to add some dependencies that
do not require
+compiling. The compilation framework of Linux (so called ``build-essential``)
is pretty big, and
+for the production images, size is really important factor to optimize for, so
our Production Image
+does not contain ``build-essential``. If you need compiler like gcc or g++ or
make/cmake etc. - those
+are not found in the image and it is recommended that you follow the
"customize" route instead.
-.. exampleinclude:: docker-examples/extending/embedding-dags/test_dag.py
- :language: Python
- :start-after: [START dag]
- :end-before: [END dag]
+How to extend the image - it is something you are most likely familiar with -
simply
+build a new image using Dockerfile's ``FROM`` directive and add whatever you
need. Then you can add your
+Debian dependencies with ``apt`` or PyPI dependencies with ``pip install`` or
any other stuff you need.
-Extending vs. customizing the image
------------------------------------
+Base images
+...........
-You might want to know very quickly how you can extend or customize the
existing image
-for Apache Airflow. This chapter gives you a short answer to those questions.
+There are two types Of images you can extend your image from:
Review Comment:
```suggestion
There are two types of images you can extend your image from:
```
##########
docs/docker-stack/build.rst:
##########
@@ -23,8 +23,103 @@ Building the image
Before you dive-deeply in the way how the Airflow Image is built, let us first
explain why you might need
to build the custom container image and we show a few typical ways you can do
it.
-Why custom image ?
-------------------
+Quick start scenarios of image extending
+----------------------------------------
+
+The most common scenarios where you want to build your own image are adding a
new ``apt`` package,
+adding a new ``PyPI`` dependency and embedding DAGs into the image.
+Example Dockerfiles for those scenarios are below, and you can read further
+for more complex cases which might involve either extending or customizing the
image. You will find
+more information about more complex scenarios below, but if your goal is to
quickly extend airflow
+image with new provider, package, etc. then here is a quick start for you.
+
+Adding new ``apt`` package
+..........................
+
+The following example adds ``vim`` to the airflow image. When adding packages
via ``apt`` you should
+switch to ``root`` user for the time of installation, but do not forget to
switch back to the
+``airflow`` user after installation is complete.
+
+.. exampleinclude:: docker-examples/extending/add-apt-packages/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+
+Adding a new ``PyPI`` package
+.............................
+
+The following example adds ``lxml`` python package from PyPI to the image.
When adding packages via
+``pip`` you need to use ``airflow`` user rather than ``root``. Attempts to
install ``pip`` packages
+with root, when you use typical ``pip install`` command will fail with
appropriate error message.
+
+.. exampleinclude:: docker-examples/extending/add-pypi-packages/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+Embedding DAGs
+..............
+
+The following example adds ``test_dag.py`` to your image in the
``/opt/airflow/dags`` folder.
+
+.. exampleinclude:: docker-examples/extending/embedding-dags/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+
+.. exampleinclude:: docker-examples/extending/embedding-dags/test_dag.py
+ :language: Python
+ :start-after: [START dag]
+ :end-before: [END dag]
+
+
+Extending vs. customizing the image
+-----------------------------------
+
+You might want to know very quickly whether you need to extend or customize
the existing image
+for Apache Airflow. This chapter gives you a short answer to those questions.
+
+Here is the comparison of the two types of building images. Here is your guide
if you want to choose
+how you want to build your image.
+
++----------------------------------------------------+-----------+-------------+
+| | Extending | Customizing
|
++====================================================+===========+=============+
+| Uses familiar 'FROM ' pattern of image building | Yes | No
|
++----------------------------------------------------+-----------+-------------+
+| Requires only basic knowledge about images | Yes | No
|
++----------------------------------------------------+-----------+-------------+
+| Builds quickly | Yes | No
|
++----------------------------------------------------+-----------+-------------+
+| Produces image heavily optimized for size | No | Yes
|
++----------------------------------------------------+-----------+-------------+
+| Can build from custom airflow sources (forks) | No | Yes
|
++----------------------------------------------------+-----------+-------------+
+| Can build on air-gaped system | No | Yes
|
++----------------------------------------------------+-----------+-------------+
+
+TL;DR; If you have a need to build custom image, it is easier to start with
"Extending" however if your
+dependencies require compilation step or when your require to build the image
from security vetted
+packages, switching to "Customizing" the image provides much more optimized
images. In the example further
+where we compare equivalent "Extending" and "Customizing" the image, similar
images build by
+Extending vs. Customization had shown 1.1GB vs 874MB image sizes respectively
- with 20% improvement in
+size of the Customized image.
Review Comment:
I'm having trouble interpreting this section. I think this is what it should
be?
```suggestion
packages, switching to "Customizing" the image provides much more optimized
images. For example,
if we compare equivalent images built by "Extending" and "Customization",
they end up being
1.1GB and 874MB respectively - a 20% improvement in size for the Customized
image.
```
##########
BREEZE.rst:
##########
@@ -917,6 +918,34 @@ Those are all available flags of ``verify-prod-image``
command:
:width: 100%
:alt: Breeze verify-prod-image
+Releasing Production images to DockerHub
+----------------------------------------
+
+The **Production image** can be released by release managers who have
permissions to push the image. This
+happens only when there is an RC candidate or final released version of
Airflow released.
+
+Typically you can release "regular" and "slim" images separately.
Review Comment:
```suggestion
You release "regular" and "slim" images separately.
```
This implies you can do them at the same time, but you tend to do it one at
a time. If I'm not mistaken, you have to do them one at a time, right?
##########
docs/docker-stack/build.rst:
##########
@@ -23,8 +23,103 @@ Building the image
Before you dive-deeply in the way how the Airflow Image is built, let us first
explain why you might need
to build the custom container image and we show a few typical ways you can do
it.
-Why custom image ?
-------------------
+Quick start scenarios of image extending
+----------------------------------------
+
+The most common scenarios where you want to build your own image are adding a
new ``apt`` package,
+adding a new ``PyPI`` dependency and embedding DAGs into the image.
+Example Dockerfiles for those scenarios are below, and you can read further
+for more complex cases which might involve either extending or customizing the
image. You will find
+more information about more complex scenarios below, but if your goal is to
quickly extend airflow
+image with new provider, package, etc. then here is a quick start for you.
+
+Adding new ``apt`` package
+..........................
+
+The following example adds ``vim`` to the airflow image. When adding packages
via ``apt`` you should
+switch to ``root`` user for the time of installation, but do not forget to
switch back to the
+``airflow`` user after installation is complete.
+
+.. exampleinclude:: docker-examples/extending/add-apt-packages/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+
+Adding a new ``PyPI`` package
+.............................
+
+The following example adds ``lxml`` python package from PyPI to the image.
When adding packages via
+``pip`` you need to use ``airflow`` user rather than ``root``. Attempts to
install ``pip`` packages
+with root, when you use typical ``pip install`` command will fail with
appropriate error message.
+
+.. exampleinclude:: docker-examples/extending/add-pypi-packages/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+Embedding DAGs
+..............
+
+The following example adds ``test_dag.py`` to your image in the
``/opt/airflow/dags`` folder.
+
+.. exampleinclude:: docker-examples/extending/embedding-dags/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+
+.. exampleinclude:: docker-examples/extending/embedding-dags/test_dag.py
+ :language: Python
+ :start-after: [START dag]
+ :end-before: [END dag]
+
+
+Extending vs. customizing the image
+-----------------------------------
+
+You might want to know very quickly whether you need to extend or customize
the existing image
+for Apache Airflow. This chapter gives you a short answer to those questions.
+
+Here is the comparison of the two types of building images. Here is your guide
if you want to choose
+how you want to build your image.
+
++----------------------------------------------------+-----------+-------------+
+| | Extending | Customizing
|
++====================================================+===========+=============+
+| Uses familiar 'FROM ' pattern of image building | Yes | No
|
++----------------------------------------------------+-----------+-------------+
+| Requires only basic knowledge about images | Yes | No
|
++----------------------------------------------------+-----------+-------------+
+| Builds quickly | Yes | No
|
++----------------------------------------------------+-----------+-------------+
+| Produces image heavily optimized for size | No | Yes
|
++----------------------------------------------------+-----------+-------------+
+| Can build from custom airflow sources (forks) | No | Yes
|
++----------------------------------------------------+-----------+-------------+
+| Can build on air-gaped system | No | Yes
|
++----------------------------------------------------+-----------+-------------+
+
+TL;DR; If you have a need to build custom image, it is easier to start with
"Extending" however if your
+dependencies require compilation step or when your require to build the image
from security vetted
Review Comment:
```suggestion
dependencies require compilation steps or when your require to build the
image from security vetted
```
##########
BREEZE.rst:
##########
@@ -917,6 +918,34 @@ Those are all available flags of ``verify-prod-image``
command:
:width: 100%
:alt: Breeze verify-prod-image
+Releasing Production images to DockerHub
+----------------------------------------
+
+The **Production image** can be released by release managers who have
permissions to push the image. This
+happens only when there is an RC candidate or final released version of
Airflow released.
Review Comment:
```suggestion
happens only when there is an RC candidate or final version of Airflow
released.
```
The double released sounds weird to me.
##########
docs/docker-stack/build.rst:
##########
@@ -23,8 +23,103 @@ Building the image
Before you dive-deeply in the way how the Airflow Image is built, let us first
explain why you might need
to build the custom container image and we show a few typical ways you can do
it.
-Why custom image ?
-------------------
+Quick start scenarios of image extending
+----------------------------------------
+
+The most common scenarios where you want to build your own image are adding a
new ``apt`` package,
+adding a new ``PyPI`` dependency and embedding DAGs into the image.
+Example Dockerfiles for those scenarios are below, and you can read further
+for more complex cases which might involve either extending or customizing the
image. You will find
+more information about more complex scenarios below, but if your goal is to
quickly extend airflow
Review Comment:
```suggestion
more information about more complex scenarios below, but if your goal is to
quickly extend the Airflow
```
##########
docs/docker-stack/build.rst:
##########
@@ -23,8 +23,103 @@ Building the image
Before you dive-deeply in the way how the Airflow Image is built, let us first
explain why you might need
to build the custom container image and we show a few typical ways you can do
it.
-Why custom image ?
-------------------
+Quick start scenarios of image extending
+----------------------------------------
+
+The most common scenarios where you want to build your own image are adding a
new ``apt`` package,
+adding a new ``PyPI`` dependency and embedding DAGs into the image.
+Example Dockerfiles for those scenarios are below, and you can read further
+for more complex cases which might involve either extending or customizing the
image. You will find
+more information about more complex scenarios below, but if your goal is to
quickly extend airflow
+image with new provider, package, etc. then here is a quick start for you.
+
+Adding new ``apt`` package
+..........................
+
+The following example adds ``vim`` to the airflow image. When adding packages
via ``apt`` you should
Review Comment:
```suggestion
The following example adds ``vim`` to the Airflow image. When adding
packages via ``apt`` you should
```
##########
dev/breeze/src/airflow_breeze/commands/release_management.py:
##########
@@ -374,6 +393,205 @@ def generate_constraints(
sys.exit(return_code)
+def convert_build_args_dict_to_array_of_args(build_args: Dict[str, str]) ->
List[str]:
+ array_of_args = []
+ for key, value in build_args.items():
+ array_of_args.append("--build-arg")
+ array_of_args.append(f'{key}={value}')
+ return array_of_args
+
+
+def alias_image(image_from: str, image_to: str, dry_run: bool, verbose: bool):
+ get_console().print(f"[info]Creating {image_to} alias for {image_from}[/]")
+ run_command(
+ ["regctl", "image", "copy", "--force-recursive", "--digest-tags",
image_from, image_to],
+ dry_run=dry_run,
+ verbose=verbose,
+ )
+
+
[email protected](
+ name="release-prod-images", help="Release production images to DockerHub
(needs DockerHub permissions)."
+)
[email protected]('--airflow-version', required=True, help="Airflow version to
release (2.3.0, 2.3.0rc1 etc.)")
[email protected](
+ '--dockerhub-repo',
+ default="apache/airflow",
+ show_default=True,
+ help="DockerHub repository for the images",
+)
[email protected](
+ '--slim-images',
+ is_flag=True,
+ help='Whether to prepare slim images rather than regular ones.',
Review Comment:
```suggestion
help='Whether to prepare slim images instead of the regular ones.',
```
##########
docs/docker-stack/build.rst:
##########
@@ -23,8 +23,103 @@ Building the image
Before you dive-deeply in the way how the Airflow Image is built, let us first
explain why you might need
to build the custom container image and we show a few typical ways you can do
it.
-Why custom image ?
-------------------
+Quick start scenarios of image extending
+----------------------------------------
+
+The most common scenarios where you want to build your own image are adding a
new ``apt`` package,
+adding a new ``PyPI`` dependency and embedding DAGs into the image.
+Example Dockerfiles for those scenarios are below, and you can read further
+for more complex cases which might involve either extending or customizing the
image. You will find
+more information about more complex scenarios below, but if your goal is to
quickly extend airflow
+image with new provider, package, etc. then here is a quick start for you.
+
+Adding new ``apt`` package
+..........................
+
+The following example adds ``vim`` to the airflow image. When adding packages
via ``apt`` you should
+switch to ``root`` user for the time of installation, but do not forget to
switch back to the
+``airflow`` user after installation is complete.
+
+.. exampleinclude:: docker-examples/extending/add-apt-packages/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+
+Adding a new ``PyPI`` package
+.............................
+
+The following example adds ``lxml`` python package from PyPI to the image.
When adding packages via
+``pip`` you need to use ``airflow`` user rather than ``root``. Attempts to
install ``pip`` packages
+with root, when you use typical ``pip install`` command will fail with
appropriate error message.
+
+.. exampleinclude:: docker-examples/extending/add-pypi-packages/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+Embedding DAGs
+..............
+
+The following example adds ``test_dag.py`` to your image in the
``/opt/airflow/dags`` folder.
+
+.. exampleinclude:: docker-examples/extending/embedding-dags/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+
+.. exampleinclude:: docker-examples/extending/embedding-dags/test_dag.py
+ :language: Python
+ :start-after: [START dag]
+ :end-before: [END dag]
+
+
+Extending vs. customizing the image
+-----------------------------------
+
+You might want to know very quickly whether you need to extend or customize
the existing image
+for Apache Airflow. This chapter gives you a short answer to those questions.
+
+Here is the comparison of the two types of building images. Here is your guide
if you want to choose
+how you want to build your image.
+
++----------------------------------------------------+-----------+-------------+
+| | Extending | Customizing
|
++====================================================+===========+=============+
+| Uses familiar 'FROM ' pattern of image building | Yes | No
|
Review Comment:
```suggestion
| Uses familiar 'FROM' pattern of image building | Yes | No
|
```
##########
dev/breeze/src/airflow_breeze/commands/release_management.py:
##########
@@ -374,6 +393,205 @@ def generate_constraints(
sys.exit(return_code)
+def convert_build_args_dict_to_array_of_args(build_args: Dict[str, str]) ->
List[str]:
+ array_of_args = []
+ for key, value in build_args.items():
+ array_of_args.append("--build-arg")
+ array_of_args.append(f'{key}={value}')
+ return array_of_args
+
+
+def alias_image(image_from: str, image_to: str, dry_run: bool, verbose: bool):
+ get_console().print(f"[info]Creating {image_to} alias for {image_from}[/]")
+ run_command(
+ ["regctl", "image", "copy", "--force-recursive", "--digest-tags",
image_from, image_to],
+ dry_run=dry_run,
+ verbose=verbose,
+ )
+
+
[email protected](
+ name="release-prod-images", help="Release production images to DockerHub
(needs DockerHub permissions)."
+)
[email protected]('--airflow-version', required=True, help="Airflow version to
release (2.3.0, 2.3.0rc1 etc.)")
[email protected](
+ '--dockerhub-repo',
+ default="apache/airflow",
+ show_default=True,
+ help="DockerHub repository for the images",
+)
[email protected](
+ '--slim-images',
+ is_flag=True,
+ help='Whether to prepare slim images rather than regular ones.',
+)
[email protected](
+ '--limit-python',
+ type=BetterChoice(CURRENT_PYTHON_MAJOR_MINOR_VERSIONS),
+ help="Specific python to build slim images for (if not specified - the
images are build for all"
+ " available python versions)",
+)
[email protected](
+ '--limit-platform',
+ type=BetterChoice(ALLOWED_PLATFORMS),
+ default=MULTI_PLATFORM,
+ show_default=True,
+ help="Specific platform to build images for (if not specified,
multiplatform images" " will be built.",
Review Comment:
```suggestion
help="Specific platform to build images for (if not specified,
multiplatform images will be built.",
```
##########
docs/docker-stack/build.rst:
##########
@@ -23,8 +23,103 @@ Building the image
Before you dive-deeply in the way how the Airflow Image is built, let us first
explain why you might need
to build the custom container image and we show a few typical ways you can do
it.
-Why custom image ?
-------------------
+Quick start scenarios of image extending
+----------------------------------------
+
+The most common scenarios where you want to build your own image are adding a
new ``apt`` package,
+adding a new ``PyPI`` dependency and embedding DAGs into the image.
+Example Dockerfiles for those scenarios are below, and you can read further
+for more complex cases which might involve either extending or customizing the
image. You will find
+more information about more complex scenarios below, but if your goal is to
quickly extend airflow
+image with new provider, package, etc. then here is a quick start for you.
+
+Adding new ``apt`` package
+..........................
+
+The following example adds ``vim`` to the airflow image. When adding packages
via ``apt`` you should
+switch to ``root`` user for the time of installation, but do not forget to
switch back to the
Review Comment:
```suggestion
switch to the ``root`` user when running the ``apt`` commands, but do not
forget to switch back to the
```
##########
docs/docker-stack/build.rst:
##########
@@ -23,8 +23,103 @@ Building the image
Before you dive-deeply in the way how the Airflow Image is built, let us first
explain why you might need
to build the custom container image and we show a few typical ways you can do
it.
-Why custom image ?
-------------------
+Quick start scenarios of image extending
+----------------------------------------
+
+The most common scenarios where you want to build your own image are adding a
new ``apt`` package,
+adding a new ``PyPI`` dependency and embedding DAGs into the image.
+Example Dockerfiles for those scenarios are below, and you can read further
+for more complex cases which might involve either extending or customizing the
image. You will find
+more information about more complex scenarios below, but if your goal is to
quickly extend airflow
+image with new provider, package, etc. then here is a quick start for you.
+
+Adding new ``apt`` package
+..........................
+
+The following example adds ``vim`` to the airflow image. When adding packages
via ``apt`` you should
+switch to ``root`` user for the time of installation, but do not forget to
switch back to the
+``airflow`` user after installation is complete.
+
+.. exampleinclude:: docker-examples/extending/add-apt-packages/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+
+Adding a new ``PyPI`` package
+.............................
+
+The following example adds ``lxml`` python package from PyPI to the image.
When adding packages via
+``pip`` you need to use ``airflow`` user rather than ``root``. Attempts to
install ``pip`` packages
Review Comment:
```suggestion
``pip`` you need to use the ``airflow`` user rather than ``root``. Attempts
to install ``pip`` packages
```
##########
docs/docker-stack/build.rst:
##########
@@ -23,8 +23,103 @@ Building the image
Before you dive-deeply in the way how the Airflow Image is built, let us first
explain why you might need
to build the custom container image and we show a few typical ways you can do
it.
-Why custom image ?
-------------------
+Quick start scenarios of image extending
+----------------------------------------
+
+The most common scenarios where you want to build your own image are adding a
new ``apt`` package,
+adding a new ``PyPI`` dependency and embedding DAGs into the image.
+Example Dockerfiles for those scenarios are below, and you can read further
+for more complex cases which might involve either extending or customizing the
image. You will find
+more information about more complex scenarios below, but if your goal is to
quickly extend airflow
+image with new provider, package, etc. then here is a quick start for you.
+
+Adding new ``apt`` package
+..........................
+
+The following example adds ``vim`` to the airflow image. When adding packages
via ``apt`` you should
+switch to ``root`` user for the time of installation, but do not forget to
switch back to the
+``airflow`` user after installation is complete.
+
+.. exampleinclude:: docker-examples/extending/add-apt-packages/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+
+Adding a new ``PyPI`` package
+.............................
+
+The following example adds ``lxml`` python package from PyPI to the image.
When adding packages via
+``pip`` you need to use ``airflow`` user rather than ``root``. Attempts to
install ``pip`` packages
+with root, when you use typical ``pip install`` command will fail with
appropriate error message.
Review Comment:
```suggestion
as ``root`` will fail with an appropriate error message.
```
##########
docs/docker-stack/build.rst:
##########
@@ -115,109 +210,62 @@ In the simplest case building your image consists of
those steps:
of storing and exposing the images, and it is most portable way of
publishing the image. Both
Docker-Compose and Kubernetes can make use of images exposed via registries.
-The most common scenarios where you want to build your own image are adding a
new ``apt`` package,
-adding a new ``PyPI`` dependency and embedding DAGs into the image.
-Example Dockerfiles for those scenarios are below, and you can read further
-for more complex cases which might involve either extending or customizing the
image.
-Adding new ``apt`` package
-..........................
-
-The following example adds ``vim`` to the airflow image. When adding packages
via ``apt`` you should
-switch to ``root`` user for the time of installation, but do not forget to
switch back to the
-``airflow`` user after installation is complete.
-
-.. exampleinclude:: docker-examples/extending/add-apt-packages/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
-
-
-Adding a new ``PyPI`` package
-.............................
-
-The following example adds ``lxml`` python package from PyPI to the image.
When adding packages via
-``pip`` you need to use ``airflow`` user rather than ``root``. Attempts to
install ``pip`` packages
-with root, when you using typical ``pip install`` command will fail with
appropriate error message.
-
-.. exampleinclude:: docker-examples/extending/add-pypi-packages/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
-
-Embedding DAGs
-..............
-
-The following example adds ``test_dag.py`` to your image in the
``/opt/airflow/dags`` folder.
-
-.. exampleinclude:: docker-examples/extending/embedding-dags/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
+Extending the image
+-------------------
+Extending the image is easiest if you just need to add some dependencies that
do not require
+compiling. The compilation framework of Linux (so called ``build-essential``)
is pretty big, and
+for the production images, size is really important factor to optimize for, so
our Production Image
+does not contain ``build-essential``. If you need compiler like gcc or g++ or
make/cmake etc. - those
+are not found in the image and it is recommended that you follow the
"customize" route instead.
-.. exampleinclude:: docker-examples/extending/embedding-dags/test_dag.py
- :language: Python
- :start-after: [START dag]
- :end-before: [END dag]
+How to extend the image - it is something you are most likely familiar with -
simply
+build a new image using Dockerfile's ``FROM`` directive and add whatever you
need. Then you can add your
+Debian dependencies with ``apt`` or PyPI dependencies with ``pip install`` or
any other stuff you need.
-Extending vs. customizing the image
------------------------------------
+Base images
+...........
-You might want to know very quickly how you can extend or customize the
existing image
-for Apache Airflow. This chapter gives you a short answer to those questions.
+There are two types Of images you can extend your image from:
+1) Regular airflow image that contains most useful extras and providers,
contains all supported backend
+ database clients for AMD64 platform and Postgres for ARM64 platform.
-Here is the comparison of the two types of building images. Here is your guide
if you want to choose
-how you want to build your image.
+2) Slim airflow image which is a minimum image you can start from it contains
all supported backends
+ database clients installed for AMD64 platform and Postgres for ARM64
platform. It contains no extras,
+ no providers installed.
Review Comment:
```suggestion
database clients installed for AMD64 platform and Postgres for ARM64
platform, but contains no extras or
providers outside of the 4 default providers.
```
##########
docs/docker-stack/build.rst:
##########
@@ -23,8 +23,103 @@ Building the image
Before you dive-deeply in the way how the Airflow Image is built, let us first
explain why you might need
to build the custom container image and we show a few typical ways you can do
it.
-Why custom image ?
-------------------
+Quick start scenarios of image extending
+----------------------------------------
+
+The most common scenarios where you want to build your own image are adding a
new ``apt`` package,
+adding a new ``PyPI`` dependency and embedding DAGs into the image.
+Example Dockerfiles for those scenarios are below, and you can read further
+for more complex cases which might involve either extending or customizing the
image. You will find
+more information about more complex scenarios below, but if your goal is to
quickly extend airflow
+image with new provider, package, etc. then here is a quick start for you.
+
+Adding new ``apt`` package
+..........................
+
+The following example adds ``vim`` to the airflow image. When adding packages
via ``apt`` you should
+switch to ``root`` user for the time of installation, but do not forget to
switch back to the
+``airflow`` user after installation is complete.
+
+.. exampleinclude:: docker-examples/extending/add-apt-packages/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+
+Adding a new ``PyPI`` package
+.............................
+
+The following example adds ``lxml`` python package from PyPI to the image.
When adding packages via
+``pip`` you need to use ``airflow`` user rather than ``root``. Attempts to
install ``pip`` packages
+with root, when you use typical ``pip install`` command will fail with
appropriate error message.
+
+.. exampleinclude:: docker-examples/extending/add-pypi-packages/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+Embedding DAGs
+..............
+
+The following example adds ``test_dag.py`` to your image in the
``/opt/airflow/dags`` folder.
+
+.. exampleinclude:: docker-examples/extending/embedding-dags/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+
+.. exampleinclude:: docker-examples/extending/embedding-dags/test_dag.py
+ :language: Python
+ :start-after: [START dag]
+ :end-before: [END dag]
+
+
+Extending vs. customizing the image
+-----------------------------------
+
+You might want to know very quickly whether you need to extend or customize
the existing image
+for Apache Airflow. This chapter gives you a short answer to those questions.
+
+Here is the comparison of the two types of building images. Here is your guide
if you want to choose
+how you want to build your image.
Review Comment:
```suggestion
Here is the comparison of the two approaches:
```
##########
docs/docker-stack/build.rst:
##########
@@ -115,109 +210,62 @@ In the simplest case building your image consists of
those steps:
of storing and exposing the images, and it is most portable way of
publishing the image. Both
Docker-Compose and Kubernetes can make use of images exposed via registries.
-The most common scenarios where you want to build your own image are adding a
new ``apt`` package,
-adding a new ``PyPI`` dependency and embedding DAGs into the image.
-Example Dockerfiles for those scenarios are below, and you can read further
-for more complex cases which might involve either extending or customizing the
image.
-Adding new ``apt`` package
-..........................
-
-The following example adds ``vim`` to the airflow image. When adding packages
via ``apt`` you should
-switch to ``root`` user for the time of installation, but do not forget to
switch back to the
-``airflow`` user after installation is complete.
-
-.. exampleinclude:: docker-examples/extending/add-apt-packages/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
-
-
-Adding a new ``PyPI`` package
-.............................
-
-The following example adds ``lxml`` python package from PyPI to the image.
When adding packages via
-``pip`` you need to use ``airflow`` user rather than ``root``. Attempts to
install ``pip`` packages
-with root, when you using typical ``pip install`` command will fail with
appropriate error message.
-
-.. exampleinclude:: docker-examples/extending/add-pypi-packages/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
-
-Embedding DAGs
-..............
-
-The following example adds ``test_dag.py`` to your image in the
``/opt/airflow/dags`` folder.
-
-.. exampleinclude:: docker-examples/extending/embedding-dags/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
+Extending the image
+-------------------
+Extending the image is easiest if you just need to add some dependencies that
do not require
+compiling. The compilation framework of Linux (so called ``build-essential``)
is pretty big, and
+for the production images, size is really important factor to optimize for, so
our Production Image
+does not contain ``build-essential``. If you need compiler like gcc or g++ or
make/cmake etc. - those
Review Comment:
```suggestion
does not contain ``build-essential``. If you need a compiler like gcc or g++
or make/cmake etc. - those
```
##########
docs/docker-stack/build.rst:
##########
@@ -115,109 +210,62 @@ In the simplest case building your image consists of
those steps:
of storing and exposing the images, and it is most portable way of
publishing the image. Both
Docker-Compose and Kubernetes can make use of images exposed via registries.
-The most common scenarios where you want to build your own image are adding a
new ``apt`` package,
-adding a new ``PyPI`` dependency and embedding DAGs into the image.
-Example Dockerfiles for those scenarios are below, and you can read further
-for more complex cases which might involve either extending or customizing the
image.
-Adding new ``apt`` package
-..........................
-
-The following example adds ``vim`` to the airflow image. When adding packages
via ``apt`` you should
-switch to ``root`` user for the time of installation, but do not forget to
switch back to the
-``airflow`` user after installation is complete.
-
-.. exampleinclude:: docker-examples/extending/add-apt-packages/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
-
-
-Adding a new ``PyPI`` package
-.............................
-
-The following example adds ``lxml`` python package from PyPI to the image.
When adding packages via
-``pip`` you need to use ``airflow`` user rather than ``root``. Attempts to
install ``pip`` packages
-with root, when you using typical ``pip install`` command will fail with
appropriate error message.
-
-.. exampleinclude:: docker-examples/extending/add-pypi-packages/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
-
-Embedding DAGs
-..............
-
-The following example adds ``test_dag.py`` to your image in the
``/opt/airflow/dags`` folder.
-
-.. exampleinclude:: docker-examples/extending/embedding-dags/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
+Extending the image
+-------------------
+Extending the image is easiest if you just need to add some dependencies that
do not require
+compiling. The compilation framework of Linux (so called ``build-essential``)
is pretty big, and
+for the production images, size is really important factor to optimize for, so
our Production Image
+does not contain ``build-essential``. If you need compiler like gcc or g++ or
make/cmake etc. - those
+are not found in the image and it is recommended that you follow the
"customize" route instead.
-.. exampleinclude:: docker-examples/extending/embedding-dags/test_dag.py
- :language: Python
- :start-after: [START dag]
- :end-before: [END dag]
+How to extend the image - it is something you are most likely familiar with -
simply
+build a new image using Dockerfile's ``FROM`` directive and add whatever you
need. Then you can add your
+Debian dependencies with ``apt`` or PyPI dependencies with ``pip install`` or
any other stuff you need.
-Extending vs. customizing the image
------------------------------------
+Base images
+...........
-You might want to know very quickly how you can extend or customize the
existing image
-for Apache Airflow. This chapter gives you a short answer to those questions.
+There are two types Of images you can extend your image from:
+1) Regular airflow image that contains most useful extras and providers,
contains all supported backend
Review Comment:
```suggestion
1) Regular Airflow image that contains the most common extras and providers,
and all supported backend
```
##########
docs/docker-stack/build.rst:
##########
@@ -115,109 +210,62 @@ In the simplest case building your image consists of
those steps:
of storing and exposing the images, and it is most portable way of
publishing the image. Both
Docker-Compose and Kubernetes can make use of images exposed via registries.
-The most common scenarios where you want to build your own image are adding a
new ``apt`` package,
-adding a new ``PyPI`` dependency and embedding DAGs into the image.
-Example Dockerfiles for those scenarios are below, and you can read further
-for more complex cases which might involve either extending or customizing the
image.
-Adding new ``apt`` package
-..........................
-
-The following example adds ``vim`` to the airflow image. When adding packages
via ``apt`` you should
-switch to ``root`` user for the time of installation, but do not forget to
switch back to the
-``airflow`` user after installation is complete.
-
-.. exampleinclude:: docker-examples/extending/add-apt-packages/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
-
-
-Adding a new ``PyPI`` package
-.............................
-
-The following example adds ``lxml`` python package from PyPI to the image.
When adding packages via
-``pip`` you need to use ``airflow`` user rather than ``root``. Attempts to
install ``pip`` packages
-with root, when you using typical ``pip install`` command will fail with
appropriate error message.
-
-.. exampleinclude:: docker-examples/extending/add-pypi-packages/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
-
-Embedding DAGs
-..............
-
-The following example adds ``test_dag.py`` to your image in the
``/opt/airflow/dags`` folder.
-
-.. exampleinclude:: docker-examples/extending/embedding-dags/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
+Extending the image
+-------------------
+Extending the image is easiest if you just need to add some dependencies that
do not require
+compiling. The compilation framework of Linux (so called ``build-essential``)
is pretty big, and
+for the production images, size is really important factor to optimize for, so
our Production Image
+does not contain ``build-essential``. If you need compiler like gcc or g++ or
make/cmake etc. - those
+are not found in the image and it is recommended that you follow the
"customize" route instead.
-.. exampleinclude:: docker-examples/extending/embedding-dags/test_dag.py
- :language: Python
- :start-after: [START dag]
- :end-before: [END dag]
+How to extend the image - it is something you are most likely familiar with -
simply
+build a new image using Dockerfile's ``FROM`` directive and add whatever you
need. Then you can add your
+Debian dependencies with ``apt`` or PyPI dependencies with ``pip install`` or
any other stuff you need.
-Extending vs. customizing the image
------------------------------------
+Base images
+...........
-You might want to know very quickly how you can extend or customize the
existing image
-for Apache Airflow. This chapter gives you a short answer to those questions.
+There are two types Of images you can extend your image from:
+1) Regular airflow image that contains most useful extras and providers,
contains all supported backend
+ database clients for AMD64 platform and Postgres for ARM64 platform.
-Here is the comparison of the two types of building images. Here is your guide
if you want to choose
-how you want to build your image.
+2) Slim airflow image which is a minimum image you can start from it contains
all supported backends
+ database clients installed for AMD64 platform and Postgres for ARM64
platform. It contains no extras,
+ no providers installed.
-+----------------------------------------------------+-----------+-------------+
-| | Extending | Customizing
|
-+====================================================+===========+=============+
-| Uses familiar 'FROM ' pattern of image building | Yes | No
|
-+----------------------------------------------------+-----------+-------------+
-| Requires only basic knowledge about images | Yes | No
|
-+----------------------------------------------------+-----------+-------------+
-| Builds quickly | Yes | No
|
-+----------------------------------------------------+-----------+-------------+
-| Produces image heavily optimized for size | No | Yes
|
-+----------------------------------------------------+-----------+-------------+
-| Can build from custom airflow sources (forks) | No | Yes
|
-+----------------------------------------------------+-----------+-------------+
-| Can build on air-gaped system | No | Yes
|
-+----------------------------------------------------+-----------+-------------+
+.. note:: Differences of slim image vs. regular image.
-TL;DR; If you have a need to build custom image, it is easier to start with
"Extending" however if your
-dependencies require compilation step or when your require to build the image
from security vetted
-packages, switching to "Customizing" the image provides much more optimized
images. In the example further
-where we compare equivalent "Extending" and "Customizing" the image, similar
images build by
-Extending vs. Customization had shown 1.1GB vs 874MB image sizes respectively
- with 20% improvement in
-size of the Customized image.
+ The slim image is small comparing to regular image (~500 MB vs ~1.1GB) and
you might need to add a
+ lot more packages and providers in order to make it useful for your case
(but if you use only a
+ small subset of providers, it might be a good starting point for you).
-.. note::
+ The slim images might have dependencies in different versions than those
used when providers are
+ preinstalled, simply because core airflow might have less limits on the
versions on its own.
Review Comment:
```suggestion
preinstalled, simply because core Airflow might have less limits on the
versions on its own.
```
##########
docs/docker-stack/build.rst:
##########
@@ -115,109 +210,62 @@ In the simplest case building your image consists of
those steps:
of storing and exposing the images, and it is most portable way of
publishing the image. Both
Docker-Compose and Kubernetes can make use of images exposed via registries.
-The most common scenarios where you want to build your own image are adding a
new ``apt`` package,
-adding a new ``PyPI`` dependency and embedding DAGs into the image.
-Example Dockerfiles for those scenarios are below, and you can read further
-for more complex cases which might involve either extending or customizing the
image.
-Adding new ``apt`` package
-..........................
-
-The following example adds ``vim`` to the airflow image. When adding packages
via ``apt`` you should
-switch to ``root`` user for the time of installation, but do not forget to
switch back to the
-``airflow`` user after installation is complete.
-
-.. exampleinclude:: docker-examples/extending/add-apt-packages/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
-
-
-Adding a new ``PyPI`` package
-.............................
-
-The following example adds ``lxml`` python package from PyPI to the image.
When adding packages via
-``pip`` you need to use ``airflow`` user rather than ``root``. Attempts to
install ``pip`` packages
-with root, when you using typical ``pip install`` command will fail with
appropriate error message.
-
-.. exampleinclude:: docker-examples/extending/add-pypi-packages/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
-
-Embedding DAGs
-..............
-
-The following example adds ``test_dag.py`` to your image in the
``/opt/airflow/dags`` folder.
-
-.. exampleinclude:: docker-examples/extending/embedding-dags/Dockerfile
- :language: Dockerfile
- :start-after: [START Dockerfile]
- :end-before: [END Dockerfile]
+Extending the image
+-------------------
+Extending the image is easiest if you just need to add some dependencies that
do not require
+compiling. The compilation framework of Linux (so called ``build-essential``)
is pretty big, and
+for the production images, size is really important factor to optimize for, so
our Production Image
+does not contain ``build-essential``. If you need compiler like gcc or g++ or
make/cmake etc. - those
+are not found in the image and it is recommended that you follow the
"customize" route instead.
-.. exampleinclude:: docker-examples/extending/embedding-dags/test_dag.py
- :language: Python
- :start-after: [START dag]
- :end-before: [END dag]
+How to extend the image - it is something you are most likely familiar with -
simply
+build a new image using Dockerfile's ``FROM`` directive and add whatever you
need. Then you can add your
+Debian dependencies with ``apt`` or PyPI dependencies with ``pip install`` or
any other stuff you need.
-Extending vs. customizing the image
------------------------------------
+Base images
+...........
-You might want to know very quickly how you can extend or customize the
existing image
-for Apache Airflow. This chapter gives you a short answer to those questions.
+There are two types Of images you can extend your image from:
+1) Regular airflow image that contains most useful extras and providers,
contains all supported backend
+ database clients for AMD64 platform and Postgres for ARM64 platform.
-Here is the comparison of the two types of building images. Here is your guide
if you want to choose
-how you want to build your image.
+2) Slim airflow image which is a minimum image you can start from it contains
all supported backends
Review Comment:
```suggestion
2) Slim Airflow image, which is a minimal image, contains all supported
backends
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
