mik-laj commented on a change in pull request #9768: URL: https://github.com/apache/airflow/pull/9768#discussion_r453210136
########## File path: BREEZE.rst ########## @@ -388,8 +447,341 @@ Also - in case you run several different Breeze containers in parallel (from dif with different versions) - they docker images for CLI Cloud Providers tools are shared so if you update it for one Breeze container, they will also get updated for all the other containers. -Stopping Interactive environment -................................ +.. raw:: html + + <div align="center"> + <a href="https://youtu.be/4MCTXq-oF68?t=1072";> + <img src="images/breeze/overlayed_breeze_cloud_tools.png" width="640" + alt="Airflow Breeze - Cloud tools"> + </a> + </div> + + +Launching Breeze integrations +----------------------------- + +When Breeze starts, it can start additional integrations. Those are additional docker containers +that are started in the same docker-compose command. Those are required by some of the tests +as described in `TESTING.rst <TESTING.rst#airflow-integration-tests>`_. + +By default Breeze starts only airflow container without any integration enabled. If you selected +``postgres`` or ``mysql`` backend, the container for the selected backend is also started (but only the one +that is selected). You can start the additional integrations by passing ``--integration`` flag +with appropriate integration name when starting Breeze. You can specify several ``--integration`` flags +to start more than one integration at a time. +Finally you can specify ``--integration all`` to start all integrations. + +Once integration is started, it will continue to run until the environment is stopped with +``breeze stop`` command. or restarted via ``breeze restart`` command + +Note that running integrations uses significant resources - CPU and memory. + +.. raw:: html + + <div align="center"> + <a href="https://youtu.be/4MCTXq-oF68?t=1187";> + <img src="images/breeze/overlayed_breeze_integrations.png" width="640" + alt="Airflow Breeze - Integrations"> + </a> + </div> + +Building CI images +------------------ + +With Breeze you can build images that are used by Airflow CI and production ones. + +For all development tasks, unit tests, integration tests, and static code checks, we use the +**CI image** maintained on the DockerHub in the ``apache/airflow`` repository. +This Docker image contains a lot of test-related packages (size of ~1GB). +Its tag follows the pattern of ``<BRANCH>-python<PYTHON_MAJOR_MINOR_VERSION>-ci`` +(for example, ``apache/airflow:master-python3.6-ci`` or ``apache/airflow:v1-10-test-python3.6-ci``). +The image is built using the `<Dockerfile.ci>`_ Dockerfile. + +The CI image is built automatically as needed, however it can be rebuilt manually with +``build-image`` command. The production +image should be built manually - but also a variant of this image is built automatically when +kubernetes tests are executed see `Running Kubernetes tests <#running-kubernetes-tests>`_ + +.. raw:: html + + <div align="center"> + <a href="https://youtu.be/4MCTXq-oF68?t=1387";> + <img src="images/breeze/overlayed_breeze_build_images.png" width="640" + alt="Airflow Breeze - Building images"> + </a> + </div> + +Building the image first time pulls a pre-built version of images from the Docker Hub, which may take some +time. But for subsequent source code changes, no wait time is expected. +However, changes to sensitive files like ``setup.py`` or ``Dockerfile.ci`` will trigger a rebuild +that may take more time though it is highly optimized to only rebuild what is needed. + +Breeze has built in mechanism to check if your local image has not diverged too much from the +latest image build on CI. This might happen when for example latest patches have been released as new +Python images or when significant changes are made in the Dockerfile. In such cases, Breeze will +download the latest images before rebuilding because this is usually faster than rebuilding the image. + +In most cases, rebuilding an image requires network connectivity (for example, to download new +dependencies). If you work offline and do not want to rebuild the images when needed, you can set the +``FORCE_ANSWER_TO_QUESTIONS`` variable to ``no`` as described in the +`Setting default behaviour for user interaction <#setting-default-behaviour-for-user-interaction>`_ section. + +Building Production images +-------------------------- + +The **Production image** is also maintained on the DockerHub in the +```apache/airflow`` repository. This Docker image (and Dockerfile) contains size-optimised Airflow +installation with selected extras and dependencies. Its tag follows the pattern of +``<BRANCH>-python<PYTHON_MAJOR_MINOR_VERSION>`` (for example, ``apache/airflow:master-python3.6`` +or ``apache/airflow:v1-10-test-python3.6``). + +However in many cases you want to add your own custom version of the image - with added apt dependencies, +python dependencies, additional Airflow extras. Breeze's ``build-image`` command helps to build your own, +customised variant of the image that contains everything you need. + +You can switch to building the production image by adding ``--production-image``flag to the ``build_image`` Review comment: ```suggestion You can switch to building the production image by adding ``--production-image`` flag to the ``build_image`` ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
