efedotova commented on a change in pull request #6450: Further fixes for GSOD URL: https://github.com/apache/airflow/pull/6450#discussion_r341990421
########## File path: LOCAL_VIRTUALENV.rst ########## @@ -18,115 +18,156 @@ .. contents:: :local: +About Local Virtual Environment (virtualenv) +============================================ -Local virtualenv environment -============================ +Use the local virtualenv development option in the combination with the _`Breeze <BREEZE.rst#aout-airflow-breeze>`_ +development environment. This option helps you benefit from the infrastructure provided +by your IDE and work in the enviroment where all necessary dependencies and tests are +available and set up within Docker images. -Installation ------------- +But you can also use the local virtualenv as a standalone development option of +you develop Airflow functionality that does not incur large external dependencies and +CI test coverage. -Install Python (3.5 or 3.6), MySQL, and libxml by using system-level -package managers like yum, apt-get for Linux, or Homebrew for Mac OS at -first. Refer to the `Dockerfile <Dockerfile>`__ for a comprehensive list +These are examples of the development options available with the local virtualenv in your IDE: + +* local debugging; +* Airflow source view; +* autocompletion; +* documentation support; +* unit tests. + +This document describes minimum requirements and insructions for using a standalone version of the local virtualenv. + +Prerequisites +============= + +Required Software Packages +-------------------------- + +Use system-level package managers like yum, apt-get for Linux, or +Homebrew for macOS to install required software packages: + +* Python (3.5 or 3.6) +* MySQL +* libxml + +Refer to the `Dockerfile <Dockerfile>`__ for a comprehensive list of required packages. -In order to use your IDE you need you can use the virtual environment. -Ideally you should setup virtualenv for all python versions that Airflow -supports (3.5, 3.6). An easy way to create the virtualenv is to use -`virtualenvwrapper <https://virtualenvwrapper.readthedocs.io/en/latest/>`__ -- it allows you to easily switch between virtualenvs using ``workon`` -command and mange your virtual environments more easily. Typically -creating the environment can be done by: +Extra Packages +-------------- -.. code:: bash +You can also install extra packages (like ``[gcp]``, etc) via +``pip install -e [EXTRA1,EXTRA2 ...]``. However, some of them may +have additional install and setup requirements for your local system. - mkvirtualenv <ENV_NAME> --python=python<VERSION> +For example, if you have a trouble installing the mysql client on macOS and get +an error as follows: -Then you need to install python PIP requirements. Typically it can be -done with: ``pip install -e ".[devel]"``. +.. code:: text -After creating the virtualenv, run this command to create the Airflow -sqlite database: + ld: library not found for -lssl + +you should set LIBRARY\_PATH before running ``pip install``: .. code:: bash - airflow db init + export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/opt/openssl/lib/ +You are STRONGLY encouraged to also install and use `pre-commit hooks <CONTRIBUTING.rst#pre-commit-hooks>`_ +for your local virtualenv development environment. Pre-commit hooks can speed up your +development cycle a lot. -Creating virtualenv can be automated with `Breeze environment <BREEZE.rst#configuring-local-virtualenv>`_ +The full list of extras is available in `<setup.py>`_. -Once initialization is done, you should select the virtualenv you -initialized as the project's default virtualenv in your IDE. +Creating a Local virtualenv +=========================== -After setting it up - you can use the usual "Run Test" option of the IDE -and have the autocomplete and documentation support from IDE as well as -you can debug and view the sources of Airflow - which is very helpful -during development. +To use your IDE for Airflow development and testing, you need to configure a virtual +environment. Ideally you should set up virtualenv for all Python versions that Airflow +supports (3.5, 3.6). -Installing other extras ------------------------ +Consider using the +`virtualenvwrapper <https://virtualenvwrapper.readthedocs.io/en/latest/>`__ +that allows you to create virtual environments and easily switch between them with the +``workon`` command. -You can also other extras (like ``[mysql]``, ``[gcp]`` etc. via -``pip install -e [EXTRA1,EXTRA2 ...]``. However some of the extras have additional -system requirements and you might need to install additional packages on your -local machine. +1. Create an environment as follows: -For example if you have trouble installing mysql client on MacOS and you have -an error similar to + ``mkvirtualenv <ENV_NAME> --python=python<VERSION>`` -.. code:: text +2. Install Python PIP requirements: - ld: library not found for -lssl + ``pip install -e ".[devel]"`` -you should set LIBRARY\_PATH before running ``pip install``: +3. Create the Airflow sqlite database: -.. code:: bash + ``airflow db init`` - export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/opt/openssl/lib/ +4. Select the virtualenv you created as the project's default virtualenv in your IDE. + +Note that if you have the Breeze development environment installed, the ``breeze`` +script can _`automate creating a virtualenv <BREEZE.rst#using-your-host-ide-with-breeze>`_ +(steps 2 and 3). -The full list of extras is available in `<setup.py>`_ +Debugging and Running Tests +=========================== -Running individual tests +Local and Remote Debugging +-------------------------- + +One of the great benefits of using the local virtualenv is an option to run +local debugging in your IDE graphical interface. You can also use ``ipdb`` +if you prefer _`console debugging <BREEZE.rst#debugging-with-ipdb>`_. + +You can also benefit from the _`remote debugging <BREEZE.rst#debugging-airflow-breeze-tests-in-ide>`_ +option available in IntelliJ/PyCharm (paid version) and debug example DAGs +running in a separate container. Review comment: I'd like to move the section BREEZE.rst#debugging-airflow-breeze-tests-in-ide to this doc. Do you think it makes sense? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
