j143 commented on a change in pull request #19756: URL: https://github.com/apache/airflow/pull/19756#discussion_r756838115
########## File path: CONTRIBUTORS_QUICK_START.rst ########## @@ -1818,3 +1818,192 @@ describes how to do it. .. raw:: html </details> + + +Setup and develop using Gitpod online workspaces +################################################ + +.. raw:: html + + <details> + <summary>Setup and develop using Gitpod online workspaces</summary> + + + +Setup Airflow with Breeze +------------------------- + + +Forking and cloning Project +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +1. Goto |airflow_github| and fork the project. + + .. |airflow_github| raw:: html + + <a href="https://github.com/apache/airflow/"; target="_blank">https://github.com/apache/airflow/</a> + + .. raw:: html + + <div align="center" style="padding-bottom:10px"> + <img src="images/quick_start/airflow_fork.png" + alt="Forking Apache Airflow project"> + </div> + +2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link. + + .. raw:: html + + <div align="center" style="padding-bottom:10px"> + <img src="images/quick_start/airflow_clone.png" + alt="Cloning github fork of Apache airflow"> + </div> + +3. Add goto https://gitpod.io/#<copied-url> as shown. + + .. raw:: html + + <div align="center" style="padding-bottom:10px"> + <img src="images/quick_start/airflow_gitpod_url.png" + alt="Open personal airflow clone with Gitpod"> + </div> + +Setting up Breeze +~~~~~~~~~~~~~~~~~ + +1. Breeze is already initialized in one of the terminals in Gitpod + +2. Once the breeze environment is initialized, create airflow tables and users from the breeze CLI. ``airflow db reset`` + is required to execute at least once for Airflow Breeze to get the database/tables created. + +.. code-block:: bash + + root@b76fcb399bb6:/opt/airflow# airflow db reset + root@b76fcb399bb6:/opt/airflow# airflow users create --role Admin --username admin --password admin \ + --email [email protected] --firstname foo --lastname bar + + +3. Closing Breeze environment. After successfully finishing above command will leave you in container, + type ``exit`` to exit the container + +.. code-block:: bash + + root@b76fcb399bb6:/opt/airflow# + root@b76fcb399bb6:/opt/airflow# exit + +.. code-block:: bash + + $ ./breeze stop + + +Installing airflow with breeze. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Gitpod default image have all the required packages installed. + +1. Add following line to ~/.bashrc in order to call breeze command from anywhere. + +.. code-block:: bash + + export PATH=${PATH}:"/home/${USER}/Projects/airflow" + source ~/.bashrc + + +Starting development +-------------------- + + +Creating a branch +~~~~~~~~~~~~~~~~~ + +1. Click on the branch symbol in the status bar + + .. raw:: html + + <div align="center" style="padding-bottom:10px"> + <img src="images/quick_start/vscode_creating_branch_1.png" + alt="Creating a new branch"> + </div> + +2. Give a name to a branch and checkout + + .. raw:: html + + <div align="center" style="padding-bottom:10px"> + <img src="images/quick_start/vscode_creating_branch_2.png" + alt="Giving a name to a branch"> + </div> + + + +Testing +~~~~~~~ + +All Tests are inside ./tests directory. + +- Running Unit tests inside Breeze environment. + + Just run ``pytest filepath+filename`` to run the tests. + +.. code-block:: bash + + root@4a2143c17426:/opt/airflow# pytest tests/utils/test_session.py + ======================================= test session starts ======================================= + platform linux -- Python 3.7.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/local/bin/python + cachedir: .pytest_cache + rootdir: /opt/airflow, configfile: pytest.ini + plugins: anyio-3.3.4, flaky-3.7.0, asyncio-0.16.0, cov-3.0.0, forked-1.3.0, httpx-0.15.0, instafail-0.4.2, rerunfailures-9.1.1, timeouts-1.2.1, xdist-2.4.0, requests-mock-1.9.3 + setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s + collected 4 items + + tests/utils/test_session.py::TestSession::test_raised_provide_session PASSED [ 25%] + tests/utils/test_session.py::TestSession::test_provide_session_without_args_and_kwargs PASSED [ 50%] + tests/utils/test_session.py::TestSession::test_provide_session_with_args PASSED [ 75%] + tests/utils/test_session.py::TestSession::test_provide_session_with_kwargs PASSED [100%] + +====================================== 4 passed, 11 warnings in 33.14s ====================================== + +- Running All the test with Breeze by specifying required python version, backend, backend version + +.. code-block:: bash + + $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests + + +- Running specific test in container using shell scripts. Testing in container scripts are located in + ``./scripts/in_container`` directory. + +.. code-block:: bash + + root@4a2143c17426:/opt/airflow# ls ./scripts/in_container/ + _in_container_script_init.sh quarantine_issue_header.md run_mypy.sh + _in_container_utils.sh run_anything.sh run_prepare_airflow_packages.sh + airflow_ci.cfg run_ci_tests.sh run_prepare_provider_documentation.sh + bin run_docs_build.sh run_prepare_provider_packages.sh + check_environment.sh run_extract_tests.sh run_resource_check.sh + check_junitxml_result.py run_fix_ownership.sh run_system_tests.sh + configure_environment.sh run_flake8.sh run_tmux_welcome.sh + entrypoint_ci.sh run_generate_constraints.sh stop_tmux_airflow.sh + entrypoint_exec.sh run_init_script.sh update_quarantined_test_status.py + prod run_install_and_test_provider_packages.sh + + root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh + +- Running specific type of test + + - Types of tests + + - Running specific type of test + + .. code-block:: bash + + $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core Review comment: This command gives an error for me. <details> <summary>error log</summary> ```c gitpod /workspace/airflow $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core Good version of docker 20.10.8. Backend: mysql MySQL version: 5.7 Python version: 3.8 Resetting the DB! Selected test type: Core mkdir: created directory '/workspace/airflow/.build/main/3.8' mkdir: created directory '/workspace/airflow/.build/main/3.8/CI' a69c30ae03621a0e7051da64fcf34eff62de9166c8538ef408793a4caa5af362 a69c30ae03621a0e7051da64fcf34eff62de9166c8538ef408793a4caa5af362 Use CI image. Branch name: main Docker image: ghcr.io/apache/airflow/main/ci/python3.8:latest Airflow source version: 2.3.0.dev0 Python version: 3.8 Backend: mysql 5.7 #################################################################################################### Airflow Breeze CHEATSHEET /workspace/airflow/breeze #################################################################################################### Port forwarding: Ports are forwarded to the running docker containers for webserver and database * 12322 -> forwarded to Airflow ssh server -> airflow:22 * 28080 -> forwarded to Airflow webserver -> airflow:8080 * 25555 -> forwarded to Flower dashboard -> airflow:5555 * 25433 -> forwarded to Postgres database -> postgres:5432 * 23306 -> forwarded to MySQL database -> mysql:3306 * 21433 -> forwarded to MSSQL database -> mssql:1443 * 26379 -> forwarded to Redis broker -> redis:6379 Here are links to those services that you can use on host: * ssh connection for remote debugging: ssh -p 12322 [email protected] pw: airflow * Webserver: http://127.0.0.1:28080 * Flower: http://127.0.0.1:25555 * Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow * Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root * Redis: redis://127.0.0.1:26379/0 #################################################################################################### You can setup autocomplete by running 'breeze setup-autocomplete' #################################################################################################### You can toggle ascii/cheatsheet by running: * breeze toggle-suppress-cheatsheet * breeze toggle-suppress-asciiart #################################################################################################### Unable to find image 'ghcr.io/apache/airflow/main/ci/python3.8:latest' locally ylatest: Pulling from apache/airflow/main/ci/python3.8 a10c77af2613: Already exists 811275ca69b8: Already exists a6e65fb20e69: Pulling fs layer fc5582d0981c: Pulling fs layer 9c7a8f0f8e58: Pulling fs layer 8441add4ab92: Pulling fs layer ebda9c5a1db7: Pulling fs layer dbad869b28d4: Pulling fs layer 2a66638f1e61: Pulling fs layer 41837b664714: Pulling fs layer a771646b27d5: Pulling fs layer afb4979590d9: Pulling fs layer 428b7c9025e2: Pulling fs layer bb98aff13f15: Pulling fs layer 9cd5fdf5c4d5: Pulling fs layer 6899d37becd2: Pulling fs layer 58f7721b779c: Pulling fs layer b20ca6be30aa: Pulling fs layer 8441add4ab92: Waiting ebda9c5a1db7: Waiting 6f8d52771bf7: Pulling fs layer 9a075d558b4b: Pulling fs layer 428b7c9025e2: Waiting 6ba6b39b87e7: Pulling fs layer fa5244c3e2fa: Pulling fs layer 72edde940830: Pulling fs layer 2a66638f1e61: Waiting afb4979590d9: Waiting 41837b664714: Waiting c618f323bf15: Pulling fs layer 6f8d52771bf7: Waiting 9cd5fdf5c4d5: Waiting 6ba6b39b87e7: Waiting bb98aff13f15: Waiting 478e02023d40: Pulling fs layer 58f7721b779c: Waiting d0e12a8666f5: Pulling fs layer fa5244c3e2fa: Waiting 9ba82d4ac0e8: Pulling fs layer c618f323bf15: Waiting 478e02023d40: Waiting 83f21b096d92: Pulling fs layer b20ca6be30aa: Waiting 72edde940830: Waiting d0e12a8666f5: Waiting 6899d37becd2: Waiting 7e847a87dea6: Pulling fs layer d0bd026e9517: Pulling fs layer 9a075d558b4b: Waiting f6b0b6c17afd: Pulling fs layer 7e847a87dea6: Waiting f5d244d620fb: Pulling fs layer d0bd026e9517: Waiting f6b0b6c17afd: Waiting 3f16cf0cdc03: Pulling fs layer f5d244d620fb: Waiting fc5582d0981c: Verifying Checksum fc5582d0981c: Download complete 9c7a8f0f8e58: Verifying Checksum 9c7a8f0f8e58: Download complete a6e65fb20e69: Verifying Checksum a6e65fb20e69: Download complete 8441add4ab92: Verifying Checksum 8441add4ab92: Download complete dbad869b28d4: Verifying Checksum dbad869b28d4: Download complete 41837b664714: Verifying Checksum 41837b664714: Download complete 2a66638f1e61: Verifying Checksum 2a66638f1e61: Download complete a6e65fb20e69: Pull complete fc5582d0981c: Pull complete afb4979590d9: Verifying Checksum afb4979590d9: Download complete 428b7c9025e2: Verifying Checksum 428b7c9025e2: Download complete 9c7a8f0f8e58: Pull complete bb98aff13f15: Verifying Checksum bb98aff13f15: Download complete a771646b27d5: Verifying Checksum a771646b27d5: Download complete 6899d37becd2: Verifying Checksum 6899d37becd2: Download complete 8441add4ab92: Pull complete 58f7721b779c: Verifying Checksum 58f7721b779c: Download complete ebda9c5a1db7: Download complete b20ca6be30aa: Download complete 9a075d558b4b: Verifying Checksum 9a075d558b4b: Download complete 6ba6b39b87e7: Download complete fa5244c3e2fa: Verifying Checksum fa5244c3e2fa: Download complete 6f8d52771bf7: Verifying Checksum 6f8d52771bf7: Download complete 72edde940830: Verifying Checksum 72edde940830: Download complete c618f323bf15: Verifying Checksum c618f323bf15: Download complete 478e02023d40: Verifying Checksum 478e02023d40: Download complete d0e12a8666f5: Verifying Checksum d0e12a8666f5: Download complete 83f21b096d92: Download complete 9ba82d4ac0e8: Verifying Checksum 9ba82d4ac0e8: Download complete 7e847a87dea6: Download complete d0bd026e9517: Verifying Checksum d0bd026e9517: Download complete f6b0b6c17afd: Verifying Checksum f6b0b6c17afd: Download complete f5d244d620fb: Verifying Checksum f5d244d620fb: Download complete 3f16cf0cdc03: Verifying Checksum 3f16cf0cdc03: Download complete 9cd5fdf5c4d5: Verifying Checksum 9cd5fdf5c4d5: Download complete ebda9c5a1db7: Pull complete dbad869b28d4: Pull complete 2a66638f1e61: Pull complete 41837b664714: Pull complete a771646b27d5: Pull complete afb4979590d9: Pull complete 428b7c9025e2: Pull complete bb98aff13f15: Pull complete 9cd5fdf5c4d5: Pull complete 6899d37becd2: Pull complete 58f7721b779c: Pull complete b20ca6be30aa: Pull complete 6f8d52771bf7: Pull complete 9a075d558b4b: Pull complete 6ba6b39b87e7: Pull complete fa5244c3e2fa: Pull complete 72edde940830: Pull complete c618f323bf15: Pull complete 478e02023d40: Pull complete d0e12a8666f5: Pull complete 9ba82d4ac0e8: Pull complete 83f21b096d92: Pull complete 7e847a87dea6: Pull complete d0bd026e9517: Pull complete f6b0b6c17afd: Pull complete f5d244d620fb: Pull complete 3f16cf0cdc03: Pull complete Digest: sha256:e19ff75603a5e74a82f39f897426119ad4c61cbbb5f0035b00c216d83f39190e Status: Downloaded newer image for ghcr.io/apache/airflow/main/ci/python3.8:latest Checking resources. * Memory available 63G. OK. * CPUs available 16. OK. WARNING!!!: Not enough Disk space available for Docker. At least 40 GBs recommended. You have 23G WARNING!!!: You have not enough resources to run Airflow (see above)! Please follow the instructions to increase amount of resources available: Please check https://github.com/apache/airflow/blob/main/BREEZE.rst#resources-required for details Good version of docker-compose: 1.29.2 WARNING: The ENABLE_TEST_COVERAGE variable is not set. Defaulting to a blank string. Pulling mysql (mysql:5.7)... 5.7: Pulling from library/mysql a10c77af2613: Already exists b76a7eb51ffd: Pull complete 258223f927e4: Pull complete 2d2c75386df9: Pull complete 63e92e4046c9: Pull complete f5845c731544: Pull complete bd0401123a9b: Pull complete 2724b2da64fd: Pull complete d10a7e9e325c: Pull complete 1c5fd9c3683d: Pull complete 2e35f83a12e9: Pull complete Digest: sha256:7a3a7b7a29e6fbff433c339fc52245435fa2c308586481f2f92ab1df239d6a29 Status: Downloaded newer image for mysql:5.7 Creating docker-compose_mysql_1 ... done Creating docker-compose_airflow_run ... done Error response from daemon: driver failed programming external connectivity on endpoint docker-compose_airflow_run_f081fd6ac899 (0d04b319ee1c4e4ceaed9d626bc70df0cb1ae0699efdf9ccead83aaed72ca420): Bind for 0.0.0.0:28080 failed: port is already allocated ERROR: 1 ``` </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
