efedotova commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to
Breeze documentation from GSOD
URL: https://github.com/apache/airflow/pull/6285#discussion_r334251004
##########
File path: BREEZE.rst
##########
@@ -290,165 +264,170 @@ It is as easy as copy&pasting this line into your code:
import ipdb; ipdb.set_trace()
-Once you hit the line you will be dropped into interactive ipdb debugger
where you have colors
-and auto-completion to guide your debugging. This works from the console where
you started your program.
-Note that in case of ``nosetest`` you need to provide ``--nocapture`` flag to
avoid nosetests
+Once you hit the line, you will be dropped into an interactive ``ipdb``
debugger where you have colors
+and autocompletion to guide your debugging. This works from the console where
you started your program.
+Note that in case of ``nosetest`` you need to provide the ``--nocapture`` flag
to avoid nosetests
capturing the stdout of your process.
-Airflow directory structure inside Docker
------------------------------------------
-
-When you are in the container note that following directories are used:
-
-.. code-block:: text
-
- /opt/airflow - here sources of Airflow are mounted from the host
(AIRFLOW_SOURCES)
- /root/airflow - all the "dynamic" Airflow files are created here:
(AIRFLOW_HOME)
- airflow.db - sqlite database in case sqlite is used
- dags - folder where non-test dags are stored (test dags are in
/opt/airflow/tests/dags)
- logs - logs from airflow executions are created there
- unittest.cfg - unit test configuration generated when entering the
environment
- webserver_config.py - webserver configuration generated when running
airflow in the container
-Note that when run in your local environment ``/root/airflow/logs`` folder is
actually mounted from your
-``logs`` directory in airflow sources, so all logs created in the container
are automatically visible in the host
-as well. Every time you enter the container the logs directory is cleaned so
that logs do not accumulate.
-
-Port forwarding
+Port Forwarding
---------------
When you run Airflow Breeze, the following ports are automatically forwarded:
-* 28080 -> forwarded to airflow webserver -> airflow-testing:8080
-* 25433 -> forwarded to postgres database -> postgres:5432
-* 23306 -> forwarded to mysql database -> mysql:3306
+* 28080 -> forwarded to Airflow webserver -> airflow-testing:8080
+* 25433 -> forwarded to Postgres database -> postgres:5432
+* 23306 -> forwarded to Mysql database -> mysql:3306
-You can connect to those ports/databases using:
+You can connect to these ports/databases using:
* Webserver: ``http://127.0.0.1:28080``
* Postgres:
``jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow``
* Mysql: ``jdbc:mysql://localhost:23306/airflow?user=root``
-Note that you need to start the webserver manually with ``airflow webserver``
command if you want to connect
-to the webserver (you can use ``tmux`` to multiply terminals).
+Start the webserver manually with the ``airflow webserver`` command if you
want to connect
+to the webserver. You can use ``tmux`` to multiply terminals.
-For databases you need to run ``airflow db reset`` at least once (or run some
tests) after you started
-Airflow Breeze to get the database/tables created. You can connect to databases
-with IDE or any other Database client:
+For databases, you need to run ``airflow db reset`` at least once (or run some
tests) after you started
+Airflow Breeze to get the database/tables created. You can connect to
databases with IDE or any other database client:
.. image:: images/database_view.png
:align: center
:alt: Database view
-You can change host port numbers used by setting appropriate environment
variables:
+You can change the used host port numbers by setting appropriate environment
variables:
* ``WEBSERVER_HOST_PORT``
* ``POSTGRES_HOST_PORT``
* ``MYSQL_HOST_PORT``
-When you set those variables, next time when you enter the environment the new
ports should be in effect.
+If you set these variables, next time when you enter the environment the new
ports should be in effect.
-Cleaning up the images
-----------------------
+Building the Documentation
+--------------------------
-You might need to cleanup your Docker environment occasionally. The images are
quite big
-(1.5GB for both images needed for static code analysis and CI tests). And if
you often rebuild/update
-images you might end up with some unused image data.
+To build documentation in Breeze, use the ``-O``, ``--build-docs`` command:
-Cleanup can be performed with ``docker system prune`` command.
-Make sure to `Stop Breeze <#stopping-breeze>`_ first with ``./breeze
--stop-environment``.
+.. code-block:: bash
-If you run into disk space errors, we recommend you prune your docker images
using the
-``docker system prune --all`` command. You might need to restart the docker
-engine before running this command.
+ ./breeze --build-docs
-You can check if your docker is clean by running ``docker images --all`` and
``docker ps --all`` - both
-should return an empty list of images and containers respectively.
+Results of the build can be found in the ``docs/_build`` folder.
-If you are on Mac OS and you end up with not enough disk space for Docker you
should increase disk space
-available for Docker. See `Prerequsites <#prerequisites>`_.
+Often errors during documentation generation come from the docstrings of
auto-api generated classes.
+During the docs building auto-api generated files are stored in the
``docs/_api`` folder. This helps you easily identify the location the problems
with documentation originated from.
Troubleshooting
---------------
-If you are having problems with the Breeze environment - try the following
(after each step you
-can check if your problem is fixed)
+If you are having problems with the Breeze environment, try the steps below.
After each step you
+can check whether your problem is fixed.
-1. Check if you have enough disks space in Docker if you are on MacOS.
-2. Stop Breeze - use ``./breeze --stop-environment``
-3. Delete ``.build`` directory and run ``./breeze --force-pull-images``
-4. `Clean up docker images <#cleaning-up-the-images>`_
-5. Restart your docker engine and try again
-6. Restart your machine and try again
-7. Remove and re-install Docker CE and try again
+1. If you are on macOS, check if you have enough disk space for Docker.
+2. Stop Breeze with ``./breeze --stop-environment``.
+3. Delete the ``.build`` directory and run ``./breeze --force-pull-images``.
+4. `Clean up Docker images <#cleaning-up-the-images>`_.
+5. Restart your Docker Engine and try again.
+6. Restart your machine and try again.
+7. Re-install Docker CE and try again.
-In case the problems are not solved, you can set VERBOSE variable to "true"
(`export VERBOSE="true"`)
-and rerun failing command, and copy & paste the output from your terminal,
describe the problem and
-post it in [Airflow Slack](https://apache-airflow-slack.herokuapp.com/)
#troubleshooting channel.
+In case the problems are not solved, you can set the VERBOSE variable to
"true" (``export VERBOSE="true"``),
+rerun the failed command, copy-and-paste the output from your terminal to the
_`Airflow Slack <https://apache-airflow-slack.herokuapp.com/>`_
#troubleshooting channel and add the problem description.
-Using Breeze for other tasks
-============================
+Testing in Breeze
Review comment:
to me it sounds like we have lots of tests/checks/hooks for Airflow and all
of them are chaotically described in different places now. I would recommend
aggregating this info, for example, in the contribution guide and providing
links to corresponding chapters for details. what do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
