This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/master by this push:
new 74036a8 Links Breeze documentation to new Breeze video (#9768)
74036a8 is described below
commit 74036a8401737165f9741afb8dbfb228be6d413f
Author: Jarek Potiuk <[email protected]>
AuthorDate: Sat Jul 11 21:21:19 2020 +0200
Links Breeze documentation to new Breeze video (#9768)
---
BREEZE.rst | 1057 ++++++++++++--------
README.md | 23 +-
breeze-complete | 1 +
images/breeze/add_overlay.sh | 49 +
images/breeze/breeze.png | Bin 0 -> 492729 bytes
images/breeze/breeze_build_docs.png | Bin 0 -> 7730 bytes
images/breeze/breeze_build_images.png | Bin 0 -> 9975 bytes
images/breeze/breeze_build_images_prod.png | Bin 0 -> 12999 bytes
.../breeze_build_images_released_versions.png | Bin 0 -> 11361 bytes
images/breeze/breeze_cloud_tools.png | Bin 0 -> 7926 bytes
images/breeze/breeze_generate_requirements.png | Bin 0 -> 10410 bytes
images/breeze/breeze_initialize_virtualenv.png | Bin 0 -> 8356 bytes
images/breeze/breeze_installation.png | Bin 0 -> 7017 bytes
images/breeze/breeze_integrations.png | Bin 0 -> 8809 bytes
images/breeze/breeze_kubernetes_tests.png | Bin 0 -> 8592 bytes
images/breeze/breeze_running_tests.png | Bin 0 -> 8343 bytes
images/breeze/breeze_select_backend_python.png | Bin 0 -> 13114 bytes
images/breeze/breeze_static_checks.png | Bin 0 -> 8901 bytes
images/breeze/breeze_stop.png | Bin 0 -> 8489 bytes
images/breeze/breeze_using_exec.png | Bin 0 -> 8415 bytes
images/breeze/breeze_using_tmux.png | Bin 0 -> 8291 bytes
images/breeze/overlayed_breeze.png | Bin 0 -> 502317 bytes
images/breeze/overlayed_breeze_build_docs.png | Bin 0 -> 9198 bytes
images/breeze/overlayed_breeze_build_images.png | Bin 0 -> 11522 bytes
.../breeze/overlayed_breeze_build_images_prod.png | Bin 0 -> 14858 bytes
...layed_breeze_build_images_released_versions.png | Bin 0 -> 13213 bytes
images/breeze/overlayed_breeze_cloud_tools.png | Bin 0 -> 9075 bytes
.../overlayed_breeze_generate_requirements.png | Bin 0 -> 11928 bytes
.../overlayed_breeze_initialize_virtualenv.png | Bin 0 -> 10465 bytes
images/breeze/overlayed_breeze_installation.png | Bin 0 -> 8985 bytes
images/breeze/overlayed_breeze_integrations.png | Bin 0 -> 10993 bytes
.../breeze/overlayed_breeze_kubernetes_tests.png | Bin 0 -> 11060 bytes
images/breeze/overlayed_breeze_running_tests.png | Bin 0 -> 10590 bytes
.../overlayed_breeze_select_backend_python.png | Bin 0 -> 14527 bytes
images/breeze/overlayed_breeze_static_checks.png | Bin 0 -> 10246 bytes
images/breeze/overlayed_breeze_stop.png | Bin 0 -> 10013 bytes
images/breeze/overlayed_breeze_using_exec.png | Bin 0 -> 10087 bytes
images/breeze/overlayed_breeze_using_tmux.png | Bin 0 -> 10137 bytes
38 files changed, 720 insertions(+), 410 deletions(-)
diff --git a/BREEZE.rst b/BREEZE.rst
index 5f38b6c..8301ba6 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -15,35 +15,38 @@
specific language governing permissions and limitations
under the License.
-.. image:: images/AirflowBreeze_logo.png
- :align: center
- :alt: Airflow Breeze Logo
+.. raw:: html
+
+ <div align="center">
+ <img src="images/AirflowBreeze_logo.png"
+ alt="Airflow Breeze - Development and Test Environment for Apache
Airflow">
+ </div>
.. contents:: :local:
-Airflow Breeze CI Environment
+Airflow Breeze CI environment
=============================
-Airflow Breeze is an easy-to-use development environment using
+Airflow Breeze is an easy-to-use development and test environment using
`Docker Compose <https://docs.docker.com/compose/>`_.
The environment is available for local use and is also used in Airflow's CI
tests.
-We called it *Airflow Breeze* as **It's a Breeze to develop Airflow**.
+We called it *Airflow Breeze* as **It's a Breeze to contribute to Airflow**.
The advantages and disadvantages of using the Breeze environment vs. other
ways of testing Airflow
are described in `CONTRIBUTING.rst
<CONTRIBUTING.rst#integration-test-development-environment>`_.
-Here is a short 10-minute video about Airflow Breeze (note that it shows an
old version of Breeze. Some
-of the points in the video are not valid anymore. The video will be updated
shortly with more up-to-date
-version):
+Watch the video below about Airflow Breeze. It explains the motivation for
Breeze
+and screencasts all its uses.
+
+.. raw:: html
-.. image:: http://img.youtube.com/vi/ffKFHV6f3PQ/0.jpg
- :width: 480px
- :height: 360px
- :scale: 100 %
- :alt: Airflow Breeze Simplified Development Workflow
- :align: center
- :target: http://www.youtube.com/watch?v=ffKFHV6f3PQ
+ <div align="center">
+ <a href="https://youtu.be/4MCTXq-oF68";>
+ <img src="images/breeze/overlayed_breeze.png" width="640"
+ alt="Airflow Breeze - Development and Test Environment for Apache
Airflow">
+ </a>
+ </div>
Prerequisites
=============
@@ -66,9 +69,12 @@ Docker Community Edition
Here is an example configuration with more than 200GB disk space for Docker:
-.. image:: images/disk_space_osx.png
- :align: left
- :alt: Disk space OSX
+.. raw:: html
+
+ <div align="center">
+ <img src="images/disk_space_osx.png" width="640"
+ alt="Disk space MacOS">
+ </div>
Docker Compose
--------------
@@ -76,7 +82,7 @@ Docker Compose
- **Version**: Install the latest stable Docker Compose and add it to the PATH.
See `Docker Compose Installation Guide
<https://docs.docker.com/compose/install/>`_ for details.
-- **Permissions**: Configure to run the ``docker-compose`` command.
+- **Permissions**: Configure permission to run the ``docker-compose`` command.
Docker in WSL 2
---------------
@@ -94,9 +100,12 @@ Docker in WSL 2
- **Docker setting** :
WSL integration needs to be enabled
-.. image:: images/docker_wsl_integration.png
- :align: left
- :alt: Docker WSL2 integration
+.. raw:: html
+
+ <div align="center">
+ <img src="images/docker_wsl_integration.png" width="640"
+ alt="Airflow Breeze - Docker WSL2 integration">
+ </div>
- **WSL 2 Filesystem Performance** :
Accessing the host Windows filesystem incurs a performance penalty,
@@ -120,43 +129,6 @@ Docker in WSL 2
If VS Code is installed on the Windows host system then in the WSL Linux
Distro
you can run ``code .`` in the root directory of you Airflow repo to launch
VS Code.
-Docker Images Used by Breeze
-----------------------------
-
-For all development tasks, unit tests, integration tests, and static code
checks, we use the
-**CI image** maintained on the DockerHub in the ``apache/airflow`` repository.
-This Docker image contains a lot of test-related packages (size of ~1GB).
-Its tag follows the pattern of
``<BRANCH>-python<PYTHON_MAJOR_MINOR_VERSION>-ci``
-(for example, ``apache/airflow:master-python3.6-ci`` or
``apache/airflow:v1-10-test-python3.6-ci``).
-The image is built using the `<Dockerfile.ci>`_ Dockerfile.
-
-For testing production image, the **Production image** is used and maintained
on the DockerHub in the
-```apache/airflow`` repository. This Docker image contains only size-optimised
Airflow with selected
-extras and dependencies. Its tag follows the pattern of
``<BRANCH>-python<PYTHON_MAJOR_MINOR_VERSION>``
-(for example, ``apache/airflow:master-python3.6`` or
``apache/airflow:v1-10-test-python3.6``).
-
-More information about the images can be found in `<IMAGES.rst>`_.
-
-By default CI images are used unless ``--production-image`` flag is used.
-
-Before you run tests, enter the environment or run local static checks, the
necessary local images should be
-pulled and built from Docker Hub. This happens automatically for the test
environment but you need to
-manually trigger it for static checks as described in `Building the images
<#building-the-images>`_
-and `Pulling the latest images <#pulling-the-latest-images>`_.
-The static checks will fail and inform what to do if the image is not yet
built.
-
-Building the image first time pulls a pre-built version of images from the
Docker Hub, which may take some
-time. But for subsequent source code changes, no wait time is expected.
-However, changes to sensitive files like ``setup.py`` or ``Dockerfile.ci``
will trigger a rebuild
-that may take more time though it is highly optimized to only rebuild what is
needed.
-
-In most cases, rebuilding an image requires network connectivity (for example,
to download new
-dependencies). If you work offline and do not want to rebuild the images when
needed, you can set the
-``FORCE_ANSWER_TO_QUESTIONS`` variable to ``no`` as described in the
-`Default behaviour for user interaction
<#default-behaviour-for-user-interaction>`_ section.
-
-See the `Troubleshooting section <#troubleshooting>`_ for steps you can make
to clean the environment.
-
Getopt and gstat
----------------
@@ -183,7 +155,6 @@ If you use zsh, run this command and re-login:
echo 'export PATH="/usr/local/opt/gnu-getopt/bin:$PATH"' >> ~/.zprofile
. ~/.zprofile
-
Memory
------
@@ -195,45 +166,153 @@ On macOS, 2GB of RAM are available for your Docker
containers by default, but mo
On Windows WSL 2 expect the Linux Disto and Docker containers to use 7 - 8 GB
of RAM.
-Airflow Directory Structure inside Docker
-=========================================
+Cleaning the environment
+------------------------
-When you are in the CI container, the following directories are used:
+You may need to clean up your Docker environment occasionally. The images are
quite big
+(1.5GB for both images needed for static code analysis and CI tests) and, if
you often rebuild/update
+them, you may end up with some unused image data.
-.. code-block:: text
+To clean up the Docker environment:
- /opt/airflow - Contains sources of Airflow mounted from the host
(AIRFLOW_SOURCES).
- /root/airflow - Contains all the "dynamic" Airflow files (AIRFLOW_HOME),
such as:
- airflow.db - sqlite database in case sqlite is used;
- dags - folder with non-test dags (test dags are in
/opt/airflow/tests/dags);
- logs - logs from Airflow executions;
- unittest.cfg - unit test configuration generated when entering the
environment;
- webserver_config.py - webserver configuration generated when running
Airflow in the container.
+1. Stop Breeze with ``./breeze stop``.
-Note that when running in your local environment, the ``/root/airflow/logs``
folder is actually mounted
-from your ``logs`` directory in the Airflow sources, so all logs created in
the container are automatically
-visible in the host as well. Every time you enter the container, the ``logs``
directory is
-cleaned so that logs do not accumulate.
+2. Run the ``docker system prune`` command.
-When you are in the production container, the following directories are used:
+3. Run ``docker images --all`` and ``docker ps --all`` to verify that your
Docker is clean.
-.. code-block:: text
+ Both commands should return an empty list of images and containers
respectively.
- /opt/airflow - Contains sources of Airflow mounted from the host
(AIRFLOW_SOURCES).
- /root/airflow - Contains all the "dynamic" Airflow files (AIRFLOW_HOME),
such as:
- airflow.db - sqlite database in case sqlite is used;
- dags - folder with non-test dags (test dags are in
/opt/airflow/tests/dags);
- logs - logs from Airflow executions;
- unittest.cfg - unit test configuration generated when entering the
environment;
- webserver_config.py - webserver configuration generated when running
Airflow in the container.
+If you run into disk space errors, consider pruning your Docker images with
the ``docker system prune --all``
+command. You may need to restart the Docker Engine before running this command.
-Note that when running in your local environment, the ``/root/airflow/logs``
folder is actually mounted
-from your ``logs`` directory in the Airflow sources, so all logs created in
the container are automatically
-visible in the host as well. Every time you enter the container, the ``logs``
directory is
-cleaned so that logs do not accumulate.
+In case of disk space errors on macOS, increase the disk space available for
Docker. See
+`Prerequisites <#prerequisites>`_ for details.
+
+
+Installation
+============
+
+Installation is as easy as checking out Airflow repository and running Breeze
command.
+You enter the Breeze test environment by running the ``./breeze`` script. You
can run it with
+the ``help`` command to see the list of available options. See `Breeze
Command-Line Interface Reference`_
+for details.
+
+.. code-block:: bash
+
+ ./breeze
+
+The First time you run Breeze, it pulls and builds a local version of Docker
images.
+It pulls the latest Airflow CI images from `Airflow DockerHub
<https://hub.docker.com/r/apache/airflow>`_
+and uses them to build your local Docker images. Note that the first run (per
python) might take up to 10
+minutes on a fast connection to start. Subsequent runs should be much faster.
+
+Once you enter the environment, you are dropped into bash shell of the Airflow
container and you can
+run tests immediately.
+
+To use the full potential of breeze you should set up autocomplete and you can
+add the checked-out Airflow repository to your PATH to run Breeze without the
``./`` and from any directory.
+
+The ``breeze`` command comes with a built-in bash/zsh autocomplete setup
command. After installing, when you
+start typing the command, you can use <TAB> to show all the available switches
and get
+auto-completion on typical values of parameters that you can use.
+
+You should set up the autocomplete option automatically by running:
+
+.. code-block:: bash
+
+ ./breeze setup-autocomplete
+
+You get the auto-completion working when you re-enter the shell.
+
+When you enter the Breeze environment, automatically an environment file is
sourced from
+``files/airflow-breeze-config/variables.env``. The ``files`` folder from your
local sources is
+automatically mounted to the container under ``/files`` path and you can put
there any files you want
+to make available for the Breeze container.
+
+.. raw:: html
+
+ <div align="center">
+ <a href="https://youtu.be/4MCTXq-oF68?t=78";>
+ <img src="images/breeze/overlayed_breeze_installation.png" width="640"
+ alt="Airflow Breeze - Installation">
+ </a>
+ </div>
+
+Running tests in the CI interactive environment
+===============================================
+
+Breeze helps with running tests in the same environment/way as CI tests are
run. You can run various
+types of tests while you enter Breeze CI interactive environment - this is
described in detail
+in `<TESTING.rst>`_
+
+.. raw:: html
+
+ <div align="center">
+ <a href="https://youtu.be/4MCTXq-oF68?t=262";>
+ <img src="images/breeze/overlayed_breeze_running_tests.png" width="640"
+ alt="Airflow Breeze - Running tests">
+ </a>
+ </div>
+
+Choosing different Breeze environment configuration
+===================================================
+
+You can use additional ``breeze`` flags to choose your environment. You can
specify a Python
+version to use, and backend (the meta-data database). Thanks to that, with
Breeze, you can recreate the same
+environments as we have in matrix builds in the CI.
+
+For example, you can choose to run Python 3.6 tests with MySQL as backend and
in the Docker environment as
+follows:
+
+.. code-block:: bash
-Using the Airflow Breeze Environment
-=====================================
+ ./breeze --python 3.6 --backend mysql
+
+The choices you make are persisted in the ``./.build/`` cache directory so
that next time when you use the
+``breeze`` script, it could use the values that were used previously. This way
you do not have to specify
+them when you run the script. You can delete the ``.build/`` directory in case
you want to restore the
+default settings.
+
+The defaults when you run the Breeze environment are Python 3.6 version and
SQLite database.
+
+.. raw:: html
+
+ <div align="center">
+ <a href="https://youtu.be/4MCTXq-oF68?t=389";>
+ <img src="images/breeze/overlayed_breeze_select_backend_python.png"
width="640"
+ alt="Airflow Breeze - Selecting Python and Backend version">
+ </a>
+ </div>
+
+
+Troubleshooting
+===============
+
+If you are having problems with the Breeze environment, try the steps below.
After each step you
+can check whether your problem is fixed.
+
+1. If you are on macOS, check if you have enough disk space for Docker.
+2. Restart Breeze with ``./breeze restart``.
+3. Delete the ``.build`` directory and run ``./breeze build-image
--force-pull-images``.
+4. Clean up Docker images via ``breeze cleanup-image`` command.
+5. Restart your Docker Engine and try again.
+6. Restart your machine and try again.
+7. Re-install Docker CE and try again.
+
+In case the problems are not solved, you can set the VERBOSE_COMMANDS variable
to "true":
+
+.. code-block::
+
+ export VERBOSE_COMMANDS="true"
+
+
+Then run the failed command, copy-and-paste the output from your terminal to
the
+`Airflow Slack <https://apache-airflow-slack.herokuapp.com/>`_
#airflow-breeze channel and
+describe your problem.
+
+Other uses the Airflow Breeze environment
+=========================================
Airflow Breeze is a bash script serving as a "swiss-army-knife" of Airflow
testing. Under the
hood it uses other scripts that you can also run manually if you have problem
with running the Breeze
@@ -293,46 +372,8 @@ Set up local development environment:
* Setup local virtualenv with ``breeze setup-virtualenv`` command
* Setup autocomplete for itself with ``breeze setup-autocomplete`` command
-Interactive Breeze Environment
-------------------------------
-
-Entering Breeze environment
-...........................
-
-You enter the Breeze test environment by running the ``./breeze`` script. You
can run it with
-the ``help`` command to see the list of available options. See `Breeze
Command-Line Interface Reference`_
-for details.
-
-.. code-block:: bash
-
- ./breeze
-
-The First time you run Breeze, it pulls and builds a local version of Docker
images.
-It pulls the latest Airflow CI images from `Airflow DockerHub
<https://hub.docker.com/r/apache/airflow>`_
-and uses them to build your local Docker images. Note that the first run (per
python) might take up to 10
-minutes on a fast connection to start. Subsequent runs should be much faster.
-
-Once you enter the environment, you are dropped into bash shell of the Airflow
container and you can
-run tests immediately.
-
-You can `set up autocomplete <#setting-up-autocomplete>`_ for commands and add
the
-checked-out Airflow repository to your PATH to run Breeze without the ``./``
and from any directory.
-
-
-When you enter the Breeze environment, automatically an environment file is
sourced from
-``files/airflow-breeze-config/variables.env``. The ``files`` folder from your
local sources is
-automatically mounted to the container under ``/files`` path and you can put
there any files you want
-to make available for the Breeze container.
-
-Running tests in the CI interactive environment
------------------------------------------------
-
-Breeze helps with running tests in the same environment/way as CI tests are
run. You can run various
-types of tests while you enter Breeze CI interactive environment - this is
described in detail
-in `<TESTING.rst>`_
-
Launching multiple terminals
-............................
+----------------------------
Often if you want to run full airflow in the Breeze environment you need to
launch multiple terminals and
run ``airflow webserver``, ``airflow scheduler``, ``airflow worker`` in
separate terminals.
@@ -343,15 +384,34 @@ capability of creating multiple virtual terminals and
multiplex between them. Mo
found at `tmux github wiki page <https://github.com/tmux/tmux/wiki>`_ . Tmux
has several useful shortcuts
that allow you to split the terminals, open new tabs etc - it's pretty useful
to learn it.
-Another - slightly easier - way is to exec into Breeze terminal from the
host's terminal. Often you can
+.. raw:: html
+
+ <div align="center">
+ <a href="https://youtu.be/4MCTXq-oF68?t=824";>
+ <img src="images/breeze/overlayed_breeze_using_tmux.png" width="640"
+ alt="Airflow Breeze - Using tmux">
+ </a>
+ </div>
+
+
+Another way is to exec into Breeze terminal from the host's terminal. Often
you can
have multiple terminals in the host (Linux/MacOS/WSL2 on Windows) and you can
simply use those terminals
to enter the running container. It's as easy as launching ``breeze exec``
while you already started the
Breeze environment. You will be dropped into bash and environment variables
will be read in the same
way as when you enter the environment. You can do it multiple times and open
as many terminals as you need.
+.. raw:: html
+
+ <div align="center">
+ <a href="https://youtu.be/4MCTXq-oF68?t=978";>
+ <img src="images/breeze/overlayed_breeze_using_exec.png" width="640"
+ alt="Airflow Breeze - Using tmux">
+ </a>
+ </div>
+
CLIs for cloud providers
-........................
+------------------------
For development convenience we installed simple wrappers for the most common
cloud providers CLIs. Those
CLIs are not installed when you build or pull the image - they will be
downloaded as docker images
@@ -388,8 +448,341 @@ Also - in case you run several different Breeze
containers in parallel (from dif
with different versions) - they docker images for CLI Cloud Providers tools
are shared so if you update it
for one Breeze container, they will also get updated for all the other
containers.
-Stopping Interactive environment
-................................
+.. raw:: html
+
+ <div align="center">
+ <a href="https://youtu.be/4MCTXq-oF68?t=1072";>
+ <img src="images/breeze/overlayed_breeze_cloud_tools.png" width="640"
+ alt="Airflow Breeze - Cloud tools">
+ </a>
+ </div>
+
+
+Launching Breeze integrations
+-----------------------------
+
+When Breeze starts, it can start additional integrations. Those are additional
docker containers
+that are started in the same docker-compose command. Those are required by
some of the tests
+as described in `TESTING.rst <TESTING.rst#airflow-integration-tests>`_.
+
+By default Breeze starts only airflow container without any integration
enabled. If you selected
+``postgres`` or ``mysql`` backend, the container for the selected backend is
also started (but only the one
+that is selected). You can start the additional integrations by passing
``--integration`` flag
+with appropriate integration name when starting Breeze. You can specify
several ``--integration`` flags
+to start more than one integration at a time.
+Finally you can specify ``--integration all`` to start all integrations.
+
+Once integration is started, it will continue to run until the environment is
stopped with
+``breeze stop`` command. or restarted via ``breeze restart`` command
+
+Note that running integrations uses significant resources - CPU and memory.
+
+.. raw:: html
+
+ <div align="center">
+ <a href="https://youtu.be/4MCTXq-oF68?t=1187";>
+ <img src="images/breeze/overlayed_breeze_integrations.png" width="640"
+ alt="Airflow Breeze - Integrations">
+ </a>
+ </div>
+
+Building CI images
+------------------
+
+With Breeze you can build images that are used by Airflow CI and production
ones.
+
+For all development tasks, unit tests, integration tests, and static code
checks, we use the
+**CI image** maintained on the DockerHub in the ``apache/airflow`` repository.
+This Docker image contains a lot of test-related packages (size of ~1GB).
+Its tag follows the pattern of
``<BRANCH>-python<PYTHON_MAJOR_MINOR_VERSION>-ci``
+(for example, ``apache/airflow:master-python3.6-ci`` or
``apache/airflow:v1-10-test-python3.6-ci``).
+The image is built using the `<Dockerfile.ci>`_ Dockerfile.
+
+The CI image is built automatically as needed, however it can be rebuilt
manually with
+``build-image`` command. The production
+image should be built manually - but also a variant of this image is built
automatically when
+kubernetes tests are executed see `Running Kubernetes tests
<#running-kubernetes-tests>`_
+
+.. raw:: html
+
+ <div align="center">
+ <a href="https://youtu.be/4MCTXq-oF68?t=1387";>
+ <img src="images/breeze/overlayed_breeze_build_images.png" width="640"
+ alt="Airflow Breeze - Building images">
+ </a>
+ </div>
+
+Building the image first time pulls a pre-built version of images from the
Docker Hub, which may take some
+time. But for subsequent source code changes, no wait time is expected.
+However, changes to sensitive files like ``setup.py`` or ``Dockerfile.ci``
will trigger a rebuild
+that may take more time though it is highly optimized to only rebuild what is
needed.
+
+Breeze has built in mechanism to check if your local image has not diverged
too much from the
+latest image build on CI. This might happen when for example latest patches
have been released as new
+Python images or when significant changes are made in the Dockerfile. In such
cases, Breeze will
+download the latest images before rebuilding because this is usually faster
than rebuilding the image.
+
+In most cases, rebuilding an image requires network connectivity (for example,
to download new
+dependencies). If you work offline and do not want to rebuild the images when
needed, you can set the
+``FORCE_ANSWER_TO_QUESTIONS`` variable to ``no`` as described in the
+`Setting default behaviour for user interaction
<#setting-default-behaviour-for-user-interaction>`_ section.
+
+Building Production images
+--------------------------
+
+The **Production image** is also maintained on the DockerHub in the
+```apache/airflow`` repository. This Docker image (and Dockerfile) contains
size-optimised Airflow
+installation with selected extras and dependencies. Its tag follows the
pattern of
+``<BRANCH>-python<PYTHON_MAJOR_MINOR_VERSION>`` (for example,
``apache/airflow:master-python3.6``
+or ``apache/airflow:v1-10-test-python3.6``).
+
+However in many cases you want to add your own custom version of the image -
with added apt dependencies,
+python dependencies, additional Airflow extras. Breeze's ``build-image``
command helps to build your own,
+customised variant of the image that contains everything you need.
+
+You can switch to building the production image by adding
``--production-image`` flag to the ``build_image``
+command. Note, that the images can also be build using ``docker build``
command by passing appropriate
+build-args as described in `IMAGES.rst <IMAGES.rst>`_ , but Breeze provides
several flags that
+makes it easier to do it. You can see all the flags by running ``./breeze
build-image --help``,
+but here typical examples are presented:
+
+.. code-block:: bash
+
+ ./breeze build-image --production-image --additional-extras "jira"
+
+This installs additional ``jira`` extra while installing airflow in the image.
+
+
+.. code-block:: bash
+
+ ./breeze build-image --production-image --additional-python-deps
"torchio==0.17.10"
+
+This install additional pypi dependency - torchio in specified version.
+
+
+.. code-block:: bash
+
+ ./breeze build-image --production-image --additional-dev-deps
"libasound2" \
+ --additional-runtime-deps "libasound2"
+
+This install additional apt dependencies - ``libasound2-dev`` in build image
and ``libasound`` in the
+final image. Those are development dependencies that might be needed to build
and use python packages added
+via the ``--additional-python-deps`` flag. The ``dev`` dependencies are not
installed in the final
+production image, they are only installed in the build "segment" of the
production image that is used
+as an intermediate step to build the final image. Usually names of the ``dev``
dependencies end with ``-dev``
+suffix and they need to also be paired with corresponding runtime dependency
added for the runtime image
+(without -dev).
+
+.. raw:: html
+
+ <div align="center">
+ <a href="https://youtu.be/4MCTXq-oF68?t=1496";>
+ <img src="images/breeze/overlayed_breeze_build_images_prod.png"
width="640"
+ alt="Airflow Breeze - Building Production images">
+ </a>
+ </div>
+
+Building Production images for 1.10 Airflow versions
+----------------------------------------------------
+
+With Breeze you can also use the master Dockerfile to build custom images for
released Airflow versions.
+This works in the same way as building production image from master, but you
need to add additional switch
+``--install-airflow-version``. You should pass version of airflow (as released
in PyPI). It can be used
+to install both released versions and release candidates. Similarly as in case
of master images,
+we can pass additional extras/dependencies to install via the additional flags.
+
+.. code-block:: bash
+
+ ./breeze build-image --production-image --additional-extras "jira"
--install-airflow-version="1.10.11"
+
+Builds airflow image with released Airflow version 1.10.11 and additional
extra "jira" added.
+
+.. code-block:: bash
+
+ ./breeze build-image --production-image
--install-airflow-version="1.10.11rc2"
+
+Builds airflow image with released Airflow version 1.10.11rc2.
+
+
+You can also build airflow directly from GitHub source code - by providing Git
Reference via
+``--install-airflow-reference``. The reference can be a branch name, tag name,
or commit hash. This
+is useful mostly for testing.
+
+.. code-block:: bash
+
+ ./breeze build-image --production-image
--install-airflow-reference="v1-10-test"
+
+This Builds airflow image from the current ``v1-10-test`` branch of Airflow.
+
+.. code-block:: bash
+
+ ./breeze build-image --production-image \
+
--install-airflow-reference="0d91fcf725f69e10f0969ca36f9e38e1d74110d0"
+
+This Builds airflow image from the
``0d91fcf725f69e10f0969ca36f9e38e1d74110d0`` commit hash on
+GitHub.
+
+.. raw:: html
+
+ <div align="center">
+ <a href="https://youtu.be/4MCTXq-oF68?t=1586";>
+ <img
src="images/breeze/overlayed_breeze_build_images_released_versions.png"
width="640"
+ alt="Airflow Breeze - Building Production images for 1.10 Airflow
versions">
+ </a>
+ </div>
+
+
+Running static checks
+---------------------
+
+You can run static checks via Breeze. You can also run them via pre-commit
command but with auto-completion
+Breeze makes it easier to run selective static checks. If you press <TAB>
after the static-check and if
+you have auto-complete setup you should see auto-completable list of all
checks available.
+
+.. code-block:: bash
+
+ ./breeze static-check mypy
+
+The above will run mypy check for currently staged files.
+
+You can also add arbitrary pre-commit flag after ``--``
+
+.. code-block:: bash
+
+ ./breeze static-check mypy -- --all-files
+
+The above will run mypy check for all files.
+
+.. raw:: html
+
+ <div align="center">
+ <a href="https://youtu.be/4MCTXq-oF68?t=1675";>
+ <img src="images/breeze/overlayed_breeze_static_checks.png" width="640"
+ alt="Airflow Breeze - Static checks">
+ </a>
+ </div>
+
+Building the Documentation
+--------------------------
+
+To build documentation in Breeze, use the ``build-docs`` command:
+
+.. code-block:: bash
+
+ ./breeze build-docs
+
+Results of the build can be found in the ``docs/_build`` folder.
+
+Often errors during documentation generation come from the docstrings of
auto-api generated classes.
+During the docs building auto-api generated files are stored in the
``docs/_api`` folder. This helps you
+easily identify the location the problems with documentation originated from.
+
+.. raw:: html
+
+ <div align="center">
+ <a href="https://youtu.be/4MCTXq-oF68?t=1760";>
+ <img src="images/breeze/overlayed_breeze_build_docs.png" width="640"
+ alt="Airflow Breeze - Build docs">
+ </a>
+ </div>
+
+Generating requirements
+-----------------------
+
+Whenever you modify and commit setup.py, you need to re-generate requirement
files. Those requirement
+files ara stored separately for each python version in the ``requirements``
folder. Those are
+constraints rather than requirements as described in detail in the
+`CONTRIBUTING.rst <CONTRIBUTING.rst#pinned-requirement-files>`_ contributing
documentation.
+
+In case you modify setup.py you need to update the requirements - for every
python version supported.
+
+.. code-block:: bash
+
+ ./breeze generate-requirements --python 3.6
+
+.. code-block:: bash
+
+ ./breeze generate-requirements --python 3.7
+
+.. code-block:: bash
+
+ ./breeze generate-requirements --python 3.8
+
+
+This bumps requirements to latest versions and stores hash of setup.py so that
we are automatically
+upgrading the requirements as we add new ones.
+
+.. raw:: html
+
+ <div align="center">
+ <a href="https://youtu.be/4MCTXq-oF68?t=1823";>
+ <img src="images/breeze/overlayed_breeze_generate_requirements.png"
width="640"
+ alt="Airflow Breeze - Generate requirements">
+ </a>
+ </div>
+
+Using local virtualenv environment in Your Host IDE
+---------------------------------------------------
+
+You can set up your host IDE (for example, IntelliJ's PyCharm/Idea) to work
with Breeze
+and benefit from all the features provided by your IDE, such as local and
remote debugging,
+language auto-completion, documentation support, etc.
+
+To use your host IDE with Breeze:
+
+1. Create a local virtual environment:
+
+ You can use any of the following wrappers to create and manage your virtual
environments:
+ `pyenv <https://github.com/pyenv/pyenv>`_, `pyenv-virtualenv
<https://github.com/pyenv/pyenv-virtualenv>`_,
+ or `virtualenvwrapper
<https://virtualenvwrapper.readthedocs.io/en/latest/>`_.
+
+ Ideally, you should have virtualenvs for all Python versions supported by
Airflow (3.5, 3.6, 3.7)
+
+2. Use the right command to activate the virtualenv (``workon`` if you use
virtualenvwrapper or
+ ``pyenv activate`` if you use pyenv.
+
+3. Initialize the created local virtualenv:
+
+.. code-block:: bash
+
+ ./breeze generate-requirements --python 3.8
+
+4. Select the virtualenv you created as the project's default virtualenv in
your IDE.
+
+Note that you can also use the local virtualenv for Airflow development
without Breeze.
+This is a lightweight solution that has its own limitations.
+
+More details on using the local virtualenv are available in the
`LOCAL_VIRTUALENV.rst <LOCAL_VIRTUALENV.rst>`_.
+
+.. raw:: html
+
+ <div align="center">
+ <a href="https://youtu.be/4MCTXq-oF68?t=1920";>
+ <img src="images/breeze/overlayed_breeze_initialize_virtualenv.png"
width="640"
+ alt="Airflow Breeze - Initialize virtualenv">
+ </a>
+ </div>
+
+Running Kubernetes tests
+------------------------
+
+Breeze helps with running Kubernetes tests in the same environment/way as CI
tests are run.
+Breeze helps to setup KinD cluster for testing, setting up virtualenv and
downloads the right tools
+automatically to run the tests.
+
+This is described in detail in `Testing Kubernetes
<TESTING.rst#running-tests-with-kubernetes>`_.
+
+.. raw:: html
+
+ <div align="center">
+ <a href="https://youtu.be/4MCTXq-oF68?t=2093";>
+ <img src="images/breeze/overlayed_breeze_kubernetes_tests.png"
width="640"
+ alt="Airflow Breeze - Kubernetes tests">
+ </a>
+ </div>
+
+Stopping the interactive environment
+------------------------------------
After starting up, the environment runs in the background and takes precious
memory.
You can always stop it via:
@@ -398,57 +791,134 @@ You can always stop it via:
./breeze stop
+
+.. raw:: html
+
+ <div align="center">
+ <a href="https://youtu.be/4MCTXq-oF68?t=2639";>
+ <img src="images/breeze/overlayed_breeze_stop.png" width="640"
+ alt="Airflow Breeze - Stop environment">
+ </a>
+ </div>
+
+
+Internal details of Breeze
+==========================
+
+Airflow directory structure inside container
+--------------------------------------------
+
+When you are in the CI container, the following directories are used:
+
+.. code-block:: text
+
+ /opt/airflow - Contains sources of Airflow mounted from the host
(AIRFLOW_SOURCES).
+ /root/airflow - Contains all the "dynamic" Airflow files (AIRFLOW_HOME),
such as:
+ airflow.db - sqlite database in case sqlite is used;
+ dags - folder with non-test dags (test dags are in
/opt/airflow/tests/dags);
+ logs - logs from Airflow executions;
+ unittest.cfg - unit test configuration generated when entering the
environment;
+ webserver_config.py - webserver configuration generated when running
Airflow in the container.
+
+Note that when running in your local environment, the ``/root/airflow/logs``
folder is actually mounted
+from your ``logs`` directory in the Airflow sources, so all logs created in
the container are automatically
+visible in the host as well. Every time you enter the container, the ``logs``
directory is
+cleaned so that logs do not accumulate.
+
+When you are in the production container, the following directories are used:
+
+.. code-block:: text
+
+ /opt/airflow - Contains sources of Airflow mounted from the host
(AIRFLOW_SOURCES).
+ /root/airflow - Contains all the "dynamic" Airflow files (AIRFLOW_HOME),
such as:
+ airflow.db - sqlite database in case sqlite is used;
+ dags - folder with non-test dags (test dags are in
/opt/airflow/tests/dags);
+ logs - logs from Airflow executions;
+ unittest.cfg - unit test configuration generated when entering the
environment;
+ webserver_config.py - webserver configuration generated when running
Airflow in the container.
+
+Note that when running in your local environment, the ``/root/airflow/logs``
folder is actually mounted
+from your ``logs`` directory in the Airflow sources, so all logs created in
the container are automatically
+visible in the host as well. Every time you enter the container, the ``logs``
directory is
+cleaned so that logs do not accumulate.
+
+Running Arbitrary commands in the Breeze environment
+----------------------------------------------------
+
+To run other commands/executables inside the Breeze Docker-based environment,
use the
+``./breeze shell`` command. You should add your command as -c "command" after
``--`` as extra arguments.
+
+.. code-block:: bash
+
+ ./breeze shell -- -c "ls -la"
+
+Running "Docker Compose" commands
+---------------------------------
+
+To run Docker Compose commands (such as ``help``, ``pull``, etc), use the
+``docker-compose`` command. To add extra arguments, specify them
+after ``--`` as extra arguments.
+
+.. code-block:: bash
+
+ ./breeze docker-compose pull -- --ignore-pull-failures
+
Restarting Breeze environment
-.............................
+-----------------------------
You can also restart the environment and enter it via:
-.. code-block:: bash
+.. code-block:: bash
+
+ ./breeze restart
+
+
+Setting default answers for user interaction
+--------------------------------------------
+
+Sometimes during the build, you are asked whether to perform an action, skip
it, or quit. This happens
+when rebuilding or removing an image - actions that take a lot of time and
could be potentially destructive.
+
+For automation scripts, you can export one of the three variables to control
the default
+interaction behaviour:
+
+.. code-block::
- ./breeze restart
+ export FORCE_ANSWER_TO_QUESTIONS="yes"
-Choosing different Breeze Environment configuration
-...................................................
+If ``FORCE_ANSWER_TO_QUESTIONS`` is set to ``yes``, the images are
automatically rebuilt when needed.
+Images are deleted without asking.
-You can use additional ``breeze`` flags to customize your environment. For
example, you can specify a Python
-version to use, backend and a container environment for testing. With Breeze,
you can recreate the same
-environments as we have in matrix builds in the CI.
+.. code-block::
-For example, you can choose to run Python 3.6 tests with MySQL as backend and
in the Docker environment as
-follows:
+ export FORCE_ANSWER_TO_QUESTIONS="no"
-.. code-block:: bash
+If ``FORCE_ANSWER_TO_QUESTIONS`` is set to ``no``, the old images are used
even if rebuilding is needed.
+This is useful when you work offline. Deleting images is aborted.
- ./breeze --python 3.6 --backend mysql
+.. code-block::
-The choices you make are persisted in the ``./.build/`` cache directory so
that next time when you use the
-``breeze`` script, it could use the values that were used previously. This way
you do not have to specify
-them when you run the script. You can delete the ``.build/`` directory in case
you want to restore the
-default settings.
+ export FORCE_ANSWER_TO_QUESTIONS="quit"
-The defaults when you run the Breeze environment are Python 3.6, Sqlite, and
Docker.
+If ``FORCE_ANSWER_TO_QUESTIONS`` is set to ``quit``, the whole script is
aborted. Deleting images is aborted.
-Launching Breeze integrations
-.............................
+If more than one variable is set, ``yes`` takes precedence over ``no``, which
takes precedence over ``quit``.
-When Breeze starts, it can start additional integrations. Those are additional
docker containers
-that are started in the same docker-compose command. Those are required by
some of the tests
-as described in `TESTING.rst <TESTING.rst#airflow-integration-tests>`_.
+Fixing File/Directory Ownership
+-------------------------------
-By default Breeze starts only airflow container without any integration
enabled. If you selected
-``postgres`` or ``mysql`` backend, the container for the selected backend is
also started (but only the one
-that is selected). You can start the additional integrations by passing
``--integration`` flag
-with appropriate integration name when starting Breeze. You can specify
several ``--integration`` flags
-to start more than one integration at a time.
-Finally you can specify ``--integration all`` to start all integrations.
+On Linux, there is a problem with propagating ownership of created files (a
known Docker problem). The
+files and directories created in the container are not owned by the host user
(but by the root user in our
+case). This may prevent you from switching branches, for example, if files
owned by the root user are
+created within your sources. In case you are on a Linux host and have some
files in your sources created
+by the root user, you can fix the ownership of those files by running this
script:
-Once integration is started, it will continue to run until the environment is
stopped with
-``breeze stop`` command. or restarted via ``breeze restart`` command
+.. code-block::
-Note that running integrations uses significant resources - CPU and memory.
+ ./scripts/ci/ci_fix_ownership.sh
Mounting Local Sources to Breeze
-................................
+--------------------------------
Important sources of Airflow are mounted inside the ``airflow`` container that
you enter.
This means that you can continue editing your changes on the host in your
favourite IDE and have them
@@ -466,7 +936,7 @@ the directory used by airflow scheduler and webserver to
scan dags for. You can
from local sources in Airflow. If you wish to add local DAGs that can be run
by Breeze.
Port Forwarding
-...............
+---------------
When you run Airflow Breeze, the following ports are automatically forwarded:
@@ -492,9 +962,13 @@ For databases, you need to run ``airflow db reset`` at
least once (or run some t
Airflow Breeze to get the database/tables created. You can connect to
databases with IDE or any other
database client:
-.. image:: images/database_view.png
- :align: center
- :alt: Database view
+
+.. raw:: html
+
+ <div align="center">
+ <img src="images/database_view.png" width="640"
+ alt="Airflow Breeze - Database view">
+ </div>
You can change the used host port numbers by setting appropriate environment
variables:
@@ -504,95 +978,6 @@ You can change the used host port numbers by setting
appropriate environment var
If you set these variables, next time when you enter the environment the new
ports should be in effect.
-Cleaning the Environment
-........................
-
-You may need to clean up your Docker environment occasionally. The images are
quite big
-(1.5GB for both images needed for static code analysis and CI tests) and, if
you often rebuild/update
-them, you may end up with some unused image data.
-
-To clean up the Docker environment:
-
-1. Stop Breeze with ``./breeze stop``.
-
-2. Run the ``docker system prune`` command.
-
-3. Run ``docker images --all`` and ``docker ps --all`` to verify that your
Docker is clean.
-
- Both commands should return an empty list of images and containers
respectively.
-
-If you run into disk space errors, consider pruning your Docker images with
the ``docker system prune --all``
-command. You may need to restart the Docker Engine before running this command.
-
-In case of disk space errors on macOS, increase the disk space available for
Docker. See
-`Prerequisites <#prerequisites>`_ for details.
-
-Running static checks
----------------------
-
-You can run static checks via Breeze. You can also run them via pre-commit
command but with auto-completion
-Breeze makes it easier to run selective static checks. If you press <TAB>
after the static-check and if
-you have auto-complete setup you should see auto-completable list of all
checks available.
-
-.. code-block:: bash
-
- ./breeze static-check mypy
-
-The above will run mypy check for currently staged files.
-
-You can also add arbitrary pre-commit flag after ``--``
-
-.. code-block:: bash
-
- ./breeze static-check mypy -- --all-files
-
-The above will run mypy check for all files.
-
-Running Kubernetes tests in virtual environment
------------------------------------------------
-
-Breeze helps with running Kubernetes tests in the same environment/way as CI
tests are run.
-Breeze helps to setup KinD cluster for testing, setting up virtualenv and
downloads the right tools
-automatically to run the tests.
-
-This is described in `Testing Kubernetes
<TESTING.rst#running-tests-with-kubernetes>`_ in detail.
-
-Building the Documentation
---------------------------
-
-To build documentation in Breeze, use the ``build-docs`` command:
-
-.. code-block:: bash
-
- ./breeze build-docs
-
-Results of the build can be found in the ``docs/_build`` folder.
-
-Often errors during documentation generation come from the docstrings of
auto-api generated classes.
-During the docs building auto-api generated files are stored in the
``docs/_api`` folder. This helps you
-easily identify the location the problems with documentation originated from.
-
-Running Arbitrary Commands in the Breeze Environment
-----------------------------------------------------
-
-To run other commands/executables inside the Breeze Docker-based environment,
use the
-``./breeze shell`` command. You should add your command as -c "command" after
``--`` as extra arguments.
-
-.. code-block:: bash
-
- breeze shell -- -c "ls -la"
-
-Running Docker Compose Commands
--------------------------------
-
-To run Docker Compose commands (such as ``help``, ``pull``, etc), use the
-``docker-compose`` command. To add extra arguments, specify them
-after ``--`` as extra arguments.
-
-.. code-block:: bash
-
- ./breeze docker-compose pull -- --ignore-pull-failures
-
Managing Dependencies
---------------------
@@ -619,32 +1004,6 @@ asks you to confirm rebuilding the image and proceeds
with rebuilding if you con
if you do not confirm). After rebuilding is done, Breeze drops you to shell.
You may also use the
``build-image`` command to only rebuild CI image and not to go into shell.
-Generating requirements
-.......................
-
-Whenever you modify and commit setup.py, you need to re-generate requirement
files. Those requirement
-files ara stored separately for each python version in the ``requirements``
folder. Those are
-constraints rather than requirements as described in detail in the
-`CONTRIBUTING.rst <CONTRIBUTING.rst#pinned-requirement-files>`_ contributing
documentation.
-
-In case you modify setup.py you need to update the requirements - for every
python version supported.
-
-.. code-block:: bash
-
- breeze generate-requirements --python 3.6
-
-.. code-block:: bash
-
- breeze generate-requirements --python 3.7
-
-.. code-block:: bash
-
- breeze generate-requirements --python 3.8
-
-
-This bumps requirements to latest versions and stores hash of setup.py so that
we are automatically
-upgrading the requirements as we add new ones.
-
Incremental apt Dependencies in the Dockerfile.ci during development
....................................................................
@@ -656,89 +1015,6 @@ of the ``Dockerfile.ci``. This way dependencies will be
added incrementally.
Before merge, these dependencies should be moved to the appropriate ``apt-get
install`` command,
which is already in the ``Dockerfile.ci``.
-Setting Up the auto-completion
-------------------------------
-
-The ``breeze`` command comes with a built-in bash/zsh autocomplete option for
its options. When you start typing
-the command, you can use <TAB> to show all the available switches and get
autocompletion on typical
-values of parameters that you can use.
-
-You can set up the autocomplete option automatically by running:
-
-.. code-block:: bash
-
- ./breeze setup-autocomplete
-
-You get the autocompletion working when you re-enter the shell.
-
-Zsh autocompletion is currently limited to only autocomplete options. Bash
autocompletion also completes
-options values (for example, Python version or static check name).
-
-Setting default answers for User Interaction
---------------------------------------------
-
-Sometimes during the build, you are asked whether to perform an action, skip
it, or quit. This happens
-when rebuilding or removing an image - actions that take a lot of time and
could be potentially destructive.
-
-For automation scripts, you can export one of the three variables to control
the default
-interaction behaviour:
-
-.. code-block::
-
- export FORCE_ANSWER_TO_QUESTIONS="yes"
-
-If ``FORCE_ANSWER_TO_QUESTIONS`` is set to ``yes``, the images are
automatically rebuilt when needed.
-Images are deleted without asking.
-
-.. code-block::
-
- export FORCE_ANSWER_TO_QUESTIONS="no"
-
-If ``FORCE_ANSWER_TO_QUESTIONS`` is set to ``no``, the old images are used
even if rebuilding is needed.
-This is useful when you work offline. Deleting images is aborted.
-
-.. code-block::
-
- export FORCE_ANSWER_TO_QUESTIONS="quit"
-
-If ``FORCE_ANSWER_TO_QUESTIONS`` is set to ``quit``, the whole script is
aborted. Deleting images is aborted.
-
-If more than one variable is set, ``yes`` takes precedence over ``no``, which
takes precedence over ``quit``.
-
-
-Using Your Host IDE
-===================
-
-You can set up your host IDE (for example, IntelliJ's PyCharm/Idea) to work
with Breeze
-and benefit from all the features provided by your IDE, such as local and
remote debugging,
-autocompletion, documentation support, etc.
-
-To use your host IDE with Breeze:
-
-1. Create a local virtual environment as follows:
-
- ``mkvirtualenv <ENV_NAME> --python=python<VERSION>``
-
- You can use any of the following wrappers to create and manage your virtual
environemnts:
- `pyenv <https://github.com/pyenv/pyenv>`_, `pyenv-virtualenv
<https://github.com/pyenv/pyenv-virtualenv>`_,
- or `virtualenvwrapper
<https://virtualenvwrapper.readthedocs.io/en/latest/>`_.
-
- Ideally, you should have virtualenvs for all Python versions supported by
Airflow (3.5, 3.6, 3.7)
- and switch between them with the ``workon`` command.
-
-2. Use the ``workon`` command to enter the Breeze environment.
-
-3. Initialize the created local virtualenv:
-
- ``./breeze initialize-local-virtualenv``
-
-4. Select the virtualenv you created as the project's default virtualenv in
your IDE.
-
-Note that you can also use the local virtualenv for Airflow development
without Breeze.
-This is a lightweight solution that has its own limitations.
-
-More details on using the local virtualenv are available in the
`LOCAL_VIRTUALENV.rst <LOCAL_VIRTUALENV.rst>`_.
-
Breeze Command-Line Interface Reference
=======================================
@@ -874,7 +1150,8 @@ This is the current syntax for `./breeze <./breeze>`_:
If specified, installs Airflow directly from PIP released version.
This happens at
image building time in production image and at container entering
time for CI image. One of:
- 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3
1.10.2 master v1-10-test
+ 1.10.11 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4
1.10.3 1.10.2 master
+ v1-10-test
-t, --install-airflow-reference <INSTALL_AIRFLOW_REFERENCE>
If specified, installs Airflow directly from reference in GitHub.
This happens at
@@ -1634,7 +1911,8 @@ This is the current syntax for `./breeze <./breeze>`_:
If specified, installs Airflow directly from PIP released version.
This happens at
image building time in production image and at container entering
time for CI image. One of:
- 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3
1.10.2 master v1-10-test
+ 1.10.11 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4
1.10.3 1.10.2 master
+ v1-10-test
-t, --install-airflow-reference <INSTALL_AIRFLOW_REFERENCE>
If specified, installs Airflow directly from reference in GitHub.
This happens at
@@ -1759,42 +2037,3 @@ This is the current syntax for `./breeze <./breeze>`_:
Shows detailed help message for the command specified.
.. END BREEZE HELP MARKER
-
-
-Troubleshooting
-===============
-
-If you are having problems with the Breeze environment, try the steps below.
After each step you
-can check whether your problem is fixed.
-
-1. If you are on macOS, check if you have enough disk space for Docker.
-2. Restart Breeze with ``./breeze restart``.
-3. Delete the ``.build`` directory and run ``./breeze build-image
--force-pull-images``.
-4. Clean up Docker images via ``breeze cleanup-image`` command.
-5. Restart your Docker Engine and try again.
-6. Restart your machine and try again.
-7. Re-install Docker CE and try again.
-
-In case the problems are not solved, you can set the VERBOSE_COMMANDS variable
to "true":
-
-.. code-block::
-
- export VERBOSE_COMMANDS="true"
-
-
-Then run the failed command, copy-and-paste the output from your terminal to
the
-`Airflow Slack <https://apache-airflow-slack.herokuapp.com/>`_
#airflow-breeze channel and
-describe your problem.
-
-Fixing File/Directory Ownership
--------------------------------
-
-On Linux, there is a problem with propagating ownership of created files (a
known Docker problem). The
-files and directories created in the container are not owned by the host user
(but by the root user in our
-case). This may prevent you from switching branches, for example, if files
owned by the root user are
-created within your sources. In case you are on a Linux host and have some
files in your sources created
-by the root user, you can fix the ownership of those files by running this
script:
-
-.. code-block::
-
- ./scripts/ci/ci_fix_ownership.sh
diff --git a/README.md b/README.md
index def2ccc..bed73cc 100644
--- a/README.md
+++ b/README.md
@@ -43,6 +43,7 @@ Use Airflow to author workflows as directed acyclic graphs
(DAGs) of tasks. The
- [Requirements](#requirements)
- [Getting started](#getting-started)
- [Installing from PyPI](#installing-from-pypi)
+- [Building customized production
images](#building-customized-production-images)
- [Beyond the Horizon](#beyond-the-horizon)
- [Principles](#principles)
- [User Interface](#user-interface)
@@ -67,7 +68,7 @@ Apache Airflow is tested with:
* Sqlite - latest stable (it is used mainly for development purpose)
* Kubernetes - 1.16.2, 1.17.0
-### Stable version (1.10.10)
+### Stable version (1.10.11)
* Python versions: 2.7, 3.5, 3.6, 3.7
* Postgres DB: 9.6, 10
@@ -118,6 +119,26 @@ pip install apache-airflow[postgres,gcp]==1.10.10 \
--constraint
https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
```
+## Building customized production images
+
+In order to use Airlfow in Docker Compose or Kubernetes, you might need to use
or build production images
+of Apache Airflow. The community provides two types of support for the
production images:
+
+* We provide pre-build relesed version of production image in PyPI build from
released
+ sources of Apache Airflow - shortly after release. Those images are
available in the DockerHub.
+ You can pull those images via `docker pull
apache/airflow:<VERSION>-pythonX.Y` - version is the
+ version number (for example 1.10.11). Additionally `docker pull
apache/airflow` will pull latest
+ stable version of the image with default python version (currently 3.6)
+
+* In `master` branch of Airflow and in `v1-10-stable` branch we provide
Dockerfiles and accompanying
+ files that allow to build your own customized version of the Airflow
Production image. The instructions
+ on how to build your own image with additional dependencies (if needed) are
provided in the
+ [IMAGES.rst](IMAGES.rst#production-images) if you want to build it using
`docker build` command or in
+ [BREEZE.rst](BREEZE.rst#building-production-images) to use Breeze tool which
easier interface,
+ auto-complete, and accompanying screencast video. Note, that while it is
possible to use master
+ branch to build images for released Airflow versions, it might at times get
broken so you should
+ rather rely on building your own images from the v1-10-stable branch.
+
## Beyond the Horizon
Airflow **is not** a data streaming solution. Tasks do not move data from
diff --git a/breeze-complete b/breeze-complete
index 016dc5e..8cc12df 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -29,6 +29,7 @@ _BREEZE_ALLOWED_POSTGRES_VERSIONS="9.6 10"
_BREEZE_ALLOWED_KIND_OPERATIONS="start stop restart status deploy test shell"
_BREEZE_ALLOWED_INSTALL_AIRFLOW_VERSIONS=$(cat <<-EOF
+1.10.11
1.10.10
1.10.9
1.10.8
diff --git a/images/breeze/add_overlay.sh b/images/breeze/add_overlay.sh
new file mode 100755
index 0000000..80e305e
--- /dev/null
+++ b/images/breeze/add_overlay.sh
@@ -0,0 +1,49 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# shellcheck disable=SC2034
+time_breeze="44:37"
+time_breeze_installation=" 3:04"
+time_breeze_running_tests=" 2:06"
+time_breeze_select_backend_python=" 7:15"
+time_breeze_using_tmux=" 2:34"
+time_breeze_using_exec=" 1:35"
+time_breeze_cloud_tools=" 1:55"
+time_breeze_integrations=" 3:20"
+time_breeze_build_images=" 1:48"
+time_breeze_build_images_prod=" 1:30"
+time_breeze_build_images_released_versions=" 1:30"
+time_breeze_static_checks=" 1:24"
+time_breeze_build_docs=" 1:04"
+time_breeze_generate_requirements=" 1:37"
+time_breeze_initialize_virtualenv=" 2:53"
+time_breeze_kubernetes_tests=" 9:06"
+time_breeze_stop=" 1:37"
+
+for i in breeze*
+do
+ variable_name="time_${i%.*}"
+ displayed_time="${!variable_name}"
+ convert "$i" -strokewidth 0 -fill "rgba( 220, 220, 220 , 0.5 )" \
+ -draw "rectangle 1620,980 1920,1080" \
+ -fill "rgba( 255, 255, 255 , 1 )" \
+ -draw "path 'M 1650,1010 L 1650,1050 L 1690,1030 Z'" \
+ -pointsize 54 -draw "text 1745,1050 '${displayed_time}'" \
+ "overlayed_${i}"
+done
diff --git a/images/breeze/breeze.png b/images/breeze/breeze.png
new file mode 100644
index 0000000..d4c0bb4
Binary files /dev/null and b/images/breeze/breeze.png differ
diff --git a/images/breeze/breeze_build_docs.png
b/images/breeze/breeze_build_docs.png
new file mode 100644
index 0000000..0a0dfa8
Binary files /dev/null and b/images/breeze/breeze_build_docs.png differ
diff --git a/images/breeze/breeze_build_images.png
b/images/breeze/breeze_build_images.png
new file mode 100644
index 0000000..4348098
Binary files /dev/null and b/images/breeze/breeze_build_images.png differ
diff --git a/images/breeze/breeze_build_images_prod.png
b/images/breeze/breeze_build_images_prod.png
new file mode 100644
index 0000000..4a3a94d
Binary files /dev/null and b/images/breeze/breeze_build_images_prod.png differ
diff --git a/images/breeze/breeze_build_images_released_versions.png
b/images/breeze/breeze_build_images_released_versions.png
new file mode 100644
index 0000000..2b49c1f
Binary files /dev/null and
b/images/breeze/breeze_build_images_released_versions.png differ
diff --git a/images/breeze/breeze_cloud_tools.png
b/images/breeze/breeze_cloud_tools.png
new file mode 100644
index 0000000..9cffe97
Binary files /dev/null and b/images/breeze/breeze_cloud_tools.png differ
diff --git a/images/breeze/breeze_generate_requirements.png
b/images/breeze/breeze_generate_requirements.png
new file mode 100644
index 0000000..2a23f63
Binary files /dev/null and b/images/breeze/breeze_generate_requirements.png
differ
diff --git a/images/breeze/breeze_initialize_virtualenv.png
b/images/breeze/breeze_initialize_virtualenv.png
new file mode 100644
index 0000000..93cbe0b
Binary files /dev/null and b/images/breeze/breeze_initialize_virtualenv.png
differ
diff --git a/images/breeze/breeze_installation.png
b/images/breeze/breeze_installation.png
new file mode 100644
index 0000000..dfe8bfa
Binary files /dev/null and b/images/breeze/breeze_installation.png differ
diff --git a/images/breeze/breeze_integrations.png
b/images/breeze/breeze_integrations.png
new file mode 100644
index 0000000..4877cf3
Binary files /dev/null and b/images/breeze/breeze_integrations.png differ
diff --git a/images/breeze/breeze_kubernetes_tests.png
b/images/breeze/breeze_kubernetes_tests.png
new file mode 100644
index 0000000..8e2f001
Binary files /dev/null and b/images/breeze/breeze_kubernetes_tests.png differ
diff --git a/images/breeze/breeze_running_tests.png
b/images/breeze/breeze_running_tests.png
new file mode 100644
index 0000000..08a10631
Binary files /dev/null and b/images/breeze/breeze_running_tests.png differ
diff --git a/images/breeze/breeze_select_backend_python.png
b/images/breeze/breeze_select_backend_python.png
new file mode 100644
index 0000000..6e6f380
Binary files /dev/null and b/images/breeze/breeze_select_backend_python.png
differ
diff --git a/images/breeze/breeze_static_checks.png
b/images/breeze/breeze_static_checks.png
new file mode 100644
index 0000000..e46f1a5
Binary files /dev/null and b/images/breeze/breeze_static_checks.png differ
diff --git a/images/breeze/breeze_stop.png b/images/breeze/breeze_stop.png
new file mode 100644
index 0000000..af5e8f9
Binary files /dev/null and b/images/breeze/breeze_stop.png differ
diff --git a/images/breeze/breeze_using_exec.png
b/images/breeze/breeze_using_exec.png
new file mode 100644
index 0000000..271ef61
Binary files /dev/null and b/images/breeze/breeze_using_exec.png differ
diff --git a/images/breeze/breeze_using_tmux.png
b/images/breeze/breeze_using_tmux.png
new file mode 100644
index 0000000..7ef09ac
Binary files /dev/null and b/images/breeze/breeze_using_tmux.png differ
diff --git a/images/breeze/overlayed_breeze.png
b/images/breeze/overlayed_breeze.png
new file mode 100644
index 0000000..1c4c505
Binary files /dev/null and b/images/breeze/overlayed_breeze.png differ
diff --git a/images/breeze/overlayed_breeze_build_docs.png
b/images/breeze/overlayed_breeze_build_docs.png
new file mode 100644
index 0000000..6c9146a
Binary files /dev/null and b/images/breeze/overlayed_breeze_build_docs.png
differ
diff --git a/images/breeze/overlayed_breeze_build_images.png
b/images/breeze/overlayed_breeze_build_images.png
new file mode 100644
index 0000000..3e6fa41
Binary files /dev/null and b/images/breeze/overlayed_breeze_build_images.png
differ
diff --git a/images/breeze/overlayed_breeze_build_images_prod.png
b/images/breeze/overlayed_breeze_build_images_prod.png
new file mode 100644
index 0000000..7eb2147
Binary files /dev/null and
b/images/breeze/overlayed_breeze_build_images_prod.png differ
diff --git a/images/breeze/overlayed_breeze_build_images_released_versions.png
b/images/breeze/overlayed_breeze_build_images_released_versions.png
new file mode 100644
index 0000000..8fc2bba
Binary files /dev/null and
b/images/breeze/overlayed_breeze_build_images_released_versions.png differ
diff --git a/images/breeze/overlayed_breeze_cloud_tools.png
b/images/breeze/overlayed_breeze_cloud_tools.png
new file mode 100644
index 0000000..85ffc55
Binary files /dev/null and b/images/breeze/overlayed_breeze_cloud_tools.png
differ
diff --git a/images/breeze/overlayed_breeze_generate_requirements.png
b/images/breeze/overlayed_breeze_generate_requirements.png
new file mode 100644
index 0000000..69b32ff
Binary files /dev/null and
b/images/breeze/overlayed_breeze_generate_requirements.png differ
diff --git a/images/breeze/overlayed_breeze_initialize_virtualenv.png
b/images/breeze/overlayed_breeze_initialize_virtualenv.png
new file mode 100644
index 0000000..bda0b8e
Binary files /dev/null and
b/images/breeze/overlayed_breeze_initialize_virtualenv.png differ
diff --git a/images/breeze/overlayed_breeze_installation.png
b/images/breeze/overlayed_breeze_installation.png
new file mode 100644
index 0000000..c3d308b
Binary files /dev/null and b/images/breeze/overlayed_breeze_installation.png
differ
diff --git a/images/breeze/overlayed_breeze_integrations.png
b/images/breeze/overlayed_breeze_integrations.png
new file mode 100644
index 0000000..4773a0b
Binary files /dev/null and b/images/breeze/overlayed_breeze_integrations.png
differ
diff --git a/images/breeze/overlayed_breeze_kubernetes_tests.png
b/images/breeze/overlayed_breeze_kubernetes_tests.png
new file mode 100644
index 0000000..cd74b2d
Binary files /dev/null and
b/images/breeze/overlayed_breeze_kubernetes_tests.png differ
diff --git a/images/breeze/overlayed_breeze_running_tests.png
b/images/breeze/overlayed_breeze_running_tests.png
new file mode 100644
index 0000000..f6e0092
Binary files /dev/null and b/images/breeze/overlayed_breeze_running_tests.png
differ
diff --git a/images/breeze/overlayed_breeze_select_backend_python.png
b/images/breeze/overlayed_breeze_select_backend_python.png
new file mode 100644
index 0000000..fe6736a
Binary files /dev/null and
b/images/breeze/overlayed_breeze_select_backend_python.png differ
diff --git a/images/breeze/overlayed_breeze_static_checks.png
b/images/breeze/overlayed_breeze_static_checks.png
new file mode 100644
index 0000000..c61d27c
Binary files /dev/null and b/images/breeze/overlayed_breeze_static_checks.png
differ
diff --git a/images/breeze/overlayed_breeze_stop.png
b/images/breeze/overlayed_breeze_stop.png
new file mode 100644
index 0000000..8621a71
Binary files /dev/null and b/images/breeze/overlayed_breeze_stop.png differ
diff --git a/images/breeze/overlayed_breeze_using_exec.png
b/images/breeze/overlayed_breeze_using_exec.png
new file mode 100644
index 0000000..95c1fd0
Binary files /dev/null and b/images/breeze/overlayed_breeze_using_exec.png
differ
diff --git a/images/breeze/overlayed_breeze_using_tmux.png
b/images/breeze/overlayed_breeze_using_tmux.png
new file mode 100644
index 0000000..92feae6
Binary files /dev/null and b/images/breeze/overlayed_breeze_using_tmux.png
differ
