potiuk edited a comment on issue #13336: URL: https://github.com/apache/airflow/issues/13336#issuecomment-751638597
Unfortunately we do not support `pip-compile` installation. Airflow has fairly complex set of dependencies and the official way of installation is via PIP using constraint files: http://airflow.apache.org/docs/apache-airflow/stable/installation.html#getting-airflow For example for python 3.6 this is the right way of installing airflow (and it is stable to install): ``` pip install "apache-airflow==1.10.14" \ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt"; ``` In the constraint file for 1.10.14 you will find that we have (for python 3.6 - you can check other versions by changing python version in the URL): * argcomplete==1.12.2 * importlib-metadata==2.1.0 We have a whole CI system implemented that verifies and automatically updates the constrainfts for upcoming Airflow relases and makes sure that those constraints are consistent (they pass `pip check`) and that all the 4000+ unit tests works with those constraints. Additionally those are constraints not requirements, which makes them stable to install, but at the same time they do not limit you from individually upgrading dependencies if needed (within the boundaries limited by Airflow|). We try to limit as little as possible because Airflow is both - application and library (there are people who need to upgrade their dependencies for the custom DAGs/Operators they write). You can read more about Airflow's dependency management here and the reasoning why we are doing it: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#dependency-management Unfortunately neither `poetry` (we disscussed it recently) nor (as far as I know) `pip-compile` is flexible enough to give us what we want. They are opinionated in the way they treat dependencies and the models they introduce are not flexible enough to handle the case where you want to keep stable installation and leave the freedom to upgrade particular dependencies, that's why we have the `pip` + CI generated constraints solution which handle all the cases for us. @ryansiu1995: If you know `pip-compile` well enough and want to continue using it and replicate what we have done with constraints, you are most welcome to describe and add alternative installation method also for others to use. However the per-python-version constratint files automatically generated in our repo should be the starting point for that, because we have quite sophisticated engineering behind to make it both consistent and automatically upgraded when new versions of packages are released. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
