mik-laj commented on a change in pull request #5459: [AIRFLOW-XXX] Rewrite the contributing guide URL: https://github.com/apache/airflow/pull/5459#discussion_r296492511
########## File path: CONTRIBUTING.md ########## @@ -19,422 +19,442 @@ under the License. # Contributing -Contributions are welcome and are greatly appreciated! Every -little bit helps, and credit will always be given. +Contributions are welcome and are greatly appreciated! Every little bit helps, and credit will always be +given. # Table of Contents - * [TOC](#table-of-contents) - * [Types of Contributions](#types-of-contributions) - - [Report Bugs](#report-bugs) - - [Fix Bugs](#fix-bugs) - - [Implement Features](#implement-features) - - [Improve Documentation](#improve-documentation) - - [Submit Feedback](#submit-feedback) - * [Documentation](#documentation) - * [Development and Testing](#development-and-testing) - - [Setting up a development environment](#setting-up-a-development-environment) - - [Running unit tests](#running-unit-tests) - * [Pull requests guidelines](#pull-request-guidelines) - * [Changing the Metadata Database](#changing-the-metadata-database) -## Types of Contributions +* [Development environment](#development-environment) + - [Tools used](#tools-used) + - [Setting up a development environment](#setting-up-a-development-environment) + - [Building front-end assets](#building-front-end-assets) +* [Developing Airflow](#developing-airflow) + - [Running unit tests](#running-unit-tests) + - [Setting up your own Travis CI](#setting-up-your-own-travis-ci) + - [Writing documentation](#writing-documentation) + - [Changing the metastore schema](#changing-the-metastore-schema) + - [Conventions](#conventions) +* [Pull request guidelines](#pull-request-guidelines) +* [Types of contributions](#types-of-contributions) + - [Documentation](#documentation) + - [Bug fixes & new small features](#bug-fixes--new-small-features) + - [Large changes](#large-changes) + - [Breaking changes & deprecation warnings](#breaking-changes--deprecation-warnings) + - [Other feedback](#other-feedback) +* [Tool specific tips & tricks](#tool-specific-tips--tricks) + - [reStructuredText](#restructuredtext) + - [Pylint](#pylint) -### Report Bugs +# Development environment -Report bugs through [Apache Jira](https://issues.apache.org/jira/browse/AIRFLOW) +This section explains how to set up a development environment for contributing to Apache Airflow. -Please report relevant information and preferably code that exhibits -the problem. +## Tools used -### Fix Bugs +We use the following tools (in the CI): -Look through the Jira issues for bugs. Anything is open to whoever wants -to implement it. +For code quality: +- [Travis](https://travis-ci.org/apache/airflow) for CI +- [Flake8](http://flake8.pycqa.org) for code linting +- [Pylint](https://www.pylint.org) for more code linting +- [Mypy](http://mypy-lang.org) for static type checking +- [Apache Rat](https://creadur.apache.org) for checking Apache license headers +- [Sphinx](http://www.sphinx-doc.org) for documentation building +- [Read the Docs](https://readthedocs.org/projects/airflow) for documentation hosting -### Implement Features +All tools can be called separate from the CI pipeline. See `.travis.yml` how they are called to run them manually. -Look through the [Apache Jira](https://issues.apache.org/jira/browse/AIRFLOW) for features. Any unassigned "Improvement" issue is open to whoever wants to implement it. +A few major libraries we depend on: -We've created the operators, hooks, macros and executors we needed, but we -made sure that this part of Airflow is extensible. New operators, -hooks, macros and executors are very welcomed! +- [SQLAlchemy](https://www.sqlalchemy.org) for ORM (object-relational mapping) +- [Flask Application Builder (FAB)](https://flask-appbuilder.readthedocs.io) for the UI +- [Alembic](https://alembic.sqlalchemy.org) for database migrations -### Improve Documentation +## Setting up a development environment -Airflow could always use better documentation, -whether as part of the official Airflow docs, -in docstrings, `docs/*.rst` or even on the web as blog posts or -articles. +There are three ways to install an Apache Airflow development environment: -### Submit Feedback +1. Using tools and libraries installed natively on your system +1. Using a single Docker container +1. Using Docker Compose and Airflow's CI scripts -The best way to send feedback is to open an issue on [Apache Jira](https://issues.apache.org/jira/browse/AIRFLOW) +### 1. Using tools and libraries installed natively on your system +Install Python (>=3.5.*), MySQL, and libxml by using system-level package managers like yum, apt-get for Linux, or Homebrew for Mac OS at first. Refer to the base CI Dockerfile for a comprehensive list of required packages. -If you are proposing a feature: +Then install Python development requirements. It is usually best to work in a virtualenv: -- Explain in detail how it would work. -- Keep the scope as narrow as possible, to make it easier to implement. -- Remember that this is a volunteer-driven project, and that contributions are welcome :) +```bash +cd $AIRFLOW_HOME +virtualenv env +source env/bin/activate +pip install -e '.[devel]' +``` -## Documentation +### 2. Using a single Docker container +Docker isolates everything from the rest of your system which you might prefer to avoid mixing with other development environments. -The latest API documentation is usually available -[here](https://airflow.apache.org/). To generate a local version, -you need to have set up an Airflow development environment (see below). Also -install the `doc` extra. +```bash +# Start docker in your Airflow directory +docker run -t -i -v `pwd`:/airflow/ -w /airflow/ python:3 bash -``` -pip install -e '.[doc]' -``` +# To install all of Airflows dependencies to run all tests (this is a lot) +pip install -e . -Generate and serve the documentation by running: +# To run only certain tests install the devel requirements and whatever is required +# for your test. See setup.py for the possible requirements. For example: +pip install -e '.[gcp,devel]' -``` -cd docs -./build.sh -./start_doc_server.sh +# Init the database +airflow initdb + +nosetests -v tests/hooks/test_druid_hook.py + + test_get_first_record (tests.hooks.test_druid_hook.TestDruidDbApiHook) ... ok + test_get_records (tests.hooks.test_druid_hook.TestDruidDbApiHook) ... ok + test_get_uri (tests.hooks.test_druid_hook.TestDruidDbApiHook) ... ok + test_get_conn_url (tests.hooks.test_druid_hook.TestDruidHook) ... ok + test_submit_gone_wrong (tests.hooks.test_druid_hook.TestDruidHook) ... ok + test_submit_ok (tests.hooks.test_druid_hook.TestDruidHook) ... ok + test_submit_timeout (tests.hooks.test_druid_hook.TestDruidHook) ... ok + test_submit_unknown_response (tests.hooks.test_druid_hook.TestDruidHook) ... ok + + ---------------------------------------------------------------------- + Ran 8 tests in 3.036s + + OK ``` -Only a subset of the API reference documentation builds. Install additional -extras to build the full API reference. +The Airflow code is mounted inside of the Docker container and installed "editable" (with `-e`), so if you change any code, it is directly available in the environment in your container. -## Development and Testing +### 3. Using Docker Compose and Airflow's CI scripts -### Setting up a development environment +If you prefer a bit more full-fledged development environment, where you can test e.g. against a Postgres database, you will need multiple Docker containers. Docker Compose is a convenient tool to manage multiple containers. Start a Docker container with Compose for development to avoid installing the packages directly on your system. The following will give you a shell inside a container, run all required service containers (MySQL, PostgresSQL, krb5 and so on) and install all the dependencies: -There are three ways to setup an Apache Airflow development environment. +```bash +docker-compose -f scripts/ci/docker-compose.yml run airflow-testing bash +# From the container +export TOX_ENV=py35-backend_mysql-env_docker +/app/scripts/ci/run-ci.sh +``` -1. Using tools and libraries installed directly on your system +If you wish to run individual tests inside of Docker environment you can do as follows: - Install Python (2.7.x or 3.5.x), MySQL, and libxml by using system-level package - managers like yum, apt-get for Linux, or Homebrew for Mac OS at first. Refer to the [base CI Dockerfile](https://github.com/apache/airflow-ci/blob/master/Dockerfile) for - a comprehensive list of required packages. +```bash +# From the container (with your desired environment) with druid hook +export TOX_ENV=py35-backend_mysql-env_docker +/app/scripts/ci/run-ci.sh -- tests/hooks/test_druid_hook.py +``` - Then install python development requirements. It is usually best to work in a virtualenv: +## Building front-end assets - ```bash - cd $AIRFLOW_HOME - virtualenv env - source env/bin/activate - pip install -e '.[devel]' - ``` +### Setting up the Node/npm JavaScript environment -2. Using a Docker container +`airflow/www/` contains all npm-managed, front end assets. Flask-Appbuilder itself comes bundled with jQuery +and bootstrap. While these may be phased out over time, these packages are currently not managed with npm. - Go to your Airflow directory and start a new docker container. You can choose between Python 2 or 3, whatever you prefer. +### Node/npm versions - ``` - # Start docker in your Airflow directory - docker run -t -i -v `pwd`:/airflow/ -w /airflow/ python:3 bash +Make sure you are using recent versions of node and npm. No problems have been found with node>=8.11.3 and +npm>=6.1.3. - # To install all of airflows dependencies to run all tests (this is a lot) - pip install -e . - - # To run only certain tests install the devel requirements and whatever is required - # for your test. See setup.py for the possible requirements. For example: - pip install -e '.[gcp,devel]' +### Using npm to generate bundled files - # Init the database - airflow initdb +#### npm - nosetests -v tests/hooks/test_druid_hook.py +First, npm must be available in your environment. If it is not you can run the following commands (taken from +[this source](https://gist.github.com/DanHerbert/9520689)): - test_get_first_record (tests.hooks.test_druid_hook.TestDruidDbApiHook) ... ok - test_get_records (tests.hooks.test_druid_hook.TestDruidDbApiHook) ... ok - test_get_uri (tests.hooks.test_druid_hook.TestDruidDbApiHook) ... ok - test_get_conn_url (tests.hooks.test_druid_hook.TestDruidHook) ... ok - test_submit_gone_wrong (tests.hooks.test_druid_hook.TestDruidHook) ... ok - test_submit_ok (tests.hooks.test_druid_hook.TestDruidHook) ... ok - test_submit_timeout (tests.hooks.test_druid_hook.TestDruidHook) ... ok - test_submit_unknown_response (tests.hooks.test_druid_hook.TestDruidHook) ... ok +```bash +brew install node --without-npm +echo prefix=~/.npm-packages >> ~/.npmrc +curl -L https://www.npmjs.com/install.sh | sh +``` - ---------------------------------------------------------------------- - Ran 8 tests in 3.036s +The final step is to add `~/.npm-packages/bin` to your `PATH` so commands you install globally are usable. Add +something like this to your `.bashrc` file, then `source ~/.bashrc` to reflect the change: - OK - ``` +```bash +export PATH="$HOME/.npm-packages/bin:$PATH" +``` - The Airflow code is mounted inside of the Docker container, so if you change something using your favorite IDE, you can directly test it in the container. +#### npm packages -3. Using [Docker Compose](https://docs.docker.com/compose/) and Airflow's CI scripts +To install third party libraries defined in `package.json`, run the following within the `airflow/www/` +directory which will install them in a new `node_modules/` folder within `www/`: - Start a docker container through Compose for development to avoid installing the packages directly on your system. The following will give you a shell inside a container, run all required service containers (MySQL, PostgresSQL, krb5 and so on) and install all the dependencies: +```bash +# from the root of the repository, move to where our JS package.json lives +cd airflow/www/ +# run npm install to fetch all the dependencies +npm install +``` - ```bash - docker-compose -f scripts/ci/docker-compose.yml run airflow-testing bash - # From the container - export TOX_ENV=py35-backend_mysql-env_docker - /app/scripts/ci/run-ci.sh - ``` +To parse and generate bundled files for airflow, run either of the following commands. The `dev` flag will +keep the npm script running and re-run it upon any changes within the assets directory. - If you wish to run individual tests inside of Docker environment you can do as follows: +```bash +# Compiles the production / optimized js & css +npm run prod - ```bash - # From the container (with your desired environment) with druid hook - export TOX_ENV=py35-backend_mysql-env_docker - /app/scripts/ci/run-ci.sh -- tests/hooks/test_druid_hook.py - ``` +# Start a web server that manages and updates your assets as you modify them +npm run dev +``` +#### Upgrading npm packages -### Running unit tests +Should you add or upgrade a npm package, which involves changing `package.json`, you'll need to re-run +`npm install` and push the newly generated `package-lock.json` file so we get the reproducible build. -To run tests locally, once your unit test environment is setup (directly on your -system or through our Docker setup) you should be able to simply run -``./run_unit_tests.sh`` at will. +#### JavaScript style guide -For example, in order to just execute the "core" unit tests, run the following: +We try to enforce a more consistent style and try to follow the JS community guidelines. Once you add or +modify any JavaScript code in the project, please make sure it follows the guidelines defined in +[Airbnb JavaScript Style Guide](https://github.com/airbnb/javascript). Apache Airflow uses +[ESLint](https://eslint.org/) as a tool for identifying and reporting on patterns in JavaScript, which can be +used by running any of the following commands: +```bash +# Check JS code in .js and .html files, and report any errors/warnings +npm run lint + +# Check JS code in .js and .html files, report any errors/warnings and fix them if possible +npm run lint:fix ``` -./run_unit_tests.sh tests.core:CoreTest -s --logging-level=DEBUG -``` -or a single test method: +# Developing Airflow + +## Running unit tests + +To run tests locally, once your unit test environment is set up you should be able to run +``./run_unit_tests.sh`` at will. For example, in order to just execute the "core" unit tests, run the +following: +```bash +./run_unit_tests.sh tests.core:CoreTest -s --logging-level=DEBUG ``` + +Or a single test: +```bash ./run_unit_tests.sh tests.core:CoreTest.test_check_operators -s --logging-level=DEBUG ``` -or another example: -``` + +Another example: +```bash ./run_unit_tests.sh tests.contrib.operators.test_dataproc_operator:DataprocClusterCreateOperatorTest.test_create_cluster_deletes_error_cluster -s --logging-level=DEBUG ``` To run the whole test suite with Docker Compose, do: - -``` +```bash # Install Docker Compose first, then this will run the tests docker-compose -f scripts/ci/docker-compose.yml run airflow-testing /app/scripts/ci/run-ci.sh ``` -Alternatively, you can also set up [Travis CI](https://travis-ci.org/) on your repo to automate this. -It is free for open source projects. +## Setting up your own Travis CI -Another great way of automating linting and testing is to use [Git Hooks](https://git-scm.com/book/uz/v2/Customizing-Git-Git-Hooks). For example you could create a `pre-commit` file based on the Travis CI Pipeline so that before each commit a local pipeline will be triggered and if this pipeline fails (returns an exit code other than `0`) the commit does not come through. -This "in theory" has the advantage that you can not commit any code that fails that again reduces the errors in the Travis CI Pipelines. +Every PR made to Airflow is automatically passed through the Airflow CI pipeline, which runs on Travis. If you +want to run the same pipeline without having to submit a PR, you can set up your own Travis CI. -Since there are a lot of tests the script would last very long so you probably only should test your new feature locally. +Register your Airflow fork in Travis ([guide](https://docs.travis-ci.com/user/tutorial)) and the pipeline will +run on every push. -The following example of a `pre-commit` file allows you.. -- to lint your code via flake8 -- to test your code via nosetests in a docker container based on python 2 -- to test your code via nosetests in a docker container based on python 3 +Note that travis-ci.org is legacy; make sure you register your fork on +[travis-ci.com](https://travis-ci.com). -``` -#!/bin/sh - -GREEN='\033[0;32m' -NO_COLOR='\033[0m' - -setup_python_env() { - local venv_path=${1} - - echo -e "${GREEN}Activating python virtual environment ${venv_path}..${NO_COLOR}" - source ${venv_path} -} -run_linting() { - local project_dir=$(git rev-parse --show-toplevel) - - echo -e "${GREEN}Running flake8 over directory ${project_dir}..${NO_COLOR}" - flake8 ${project_dir} -} -run_testing_in_docker() { - local feature_path=${1} - local airflow_py2_container=${2} - local airflow_py3_container=${3} - - echo -e "${GREEN}Running tests in ${feature_path} in airflow python 2 docker container..${NO_COLOR}" - docker exec -i -w /airflow/ ${airflow_py2_container} nosetests -v ${feature_path} - echo -e "${GREEN}Running tests in ${feature_path} in airflow python 3 docker container..${NO_COLOR}" - docker exec -i -w /airflow/ ${airflow_py3_container} nosetests -v ${feature_path} -} - -set -e -# NOTE: Before running this make sure you have set the function arguments correctly. -setup_python_env /Users/feluelle/venv/bin/activate -run_linting -run_testing_in_docker tests/contrib/hooks/test_imap_hook.py dazzling_chatterjee quirky_stallman +## Writing documentation -``` +The latest API documentation is usually available [here](https://airflow.apache.org/). To generate a local Review comment: Documentation of GitHub master (latest development branch) is available on ReadTheDocs: https://airflow.readthedocs.io/en/latest/ ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
