potiuk commented on a change in pull request #7841: Requirements now depend on python version URL: https://github.com/apache/airflow/pull/7841#discussion_r397602871
########## File path: CONTRIBUTING.rst ########## @@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in The problem is that Apache Airflow is a bit of both - application to install and library to be used when you are developing your own operators and DAGs. -This - seemingly unsolvable - puzzle is solved as follows: +This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available +as of airflow 1.10.10. -* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while - still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to - install in case a direct or transitive dependency is released that breaks the installation. In such case - when installing ``apache-airflow``, you might need to provide additional constraints (for - example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``) +Pinned requirement files +------------------------ -* we have ``requirements.txt`` file generated automatically based on the set of all latest working - and tested requirement versions. You can also use that file as a constraints file when installing - apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or - from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that - this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or - ``pip install apache-airflow[gcp] --constraint requirements.txt`` +By default when you install ``apache-airflow`` package - the dependencies are as open as possible while +still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to +install in case a direct or transitive dependency is released that breaks the installation. In such case +when installing ``apache-airflow``, you might need to provide additional constraints (for +example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``) -The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies -It reflects the current set of dependencies installed in the CI image of Apache Airflow. -The same set of requirements will be used to produce the production image. +However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated +automatically and committd in the requirements folder based on the set of all latest working and tested +requirement versions. Those requirement.txt files can be used as constraints file when installing apache +airflow - either from the sources ``pip install -e . --constraint requirements/requirements-python3.7.txt`` +or from the pypi package ``pip install apache-airflow --constraint requirements/requirements-python3.7.txt``. +This work also with extras - ``pip install .[gcp] --constraint requirements/requirements-python3.7.txt``. +It is also possible to use constraints directly from github (note that it only works starting from +airflow 1.10.10 version: -If you do not use pre-commits and the CI builds fails / you need to regenerate it, you can do it manually: -``pre-commit run generate-requirements --all-files`` or via script -``./scripts/ci/ci_generate_requirements.sh``. -This will try to regenerate the requirements.txt file with the latest requirements matching -the setup.py constraints. +.. code-block:: bash + + pip install apache-airflow[gcp]==1.10.10 \ + --constraint https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt + +There are different set of fixed requirements for different python major/minor versions and you should +use the right python version file for the right python version. + +The ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file MUST be regenerated every time after setup.py +is updated. This is checked automatically in Travis CI build there are separate jobs for each +python version that checks if the requirements should be updated. + +If they are not you should regenerate the requirements locally using one of the methods described below. + +Generating requirement files +---------------------------- + +This shuould be done every time after you modify setup.py file. You can generate requirement files Review comment: Fixed ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
