potiuk edited a comment on pull request #9684: URL: https://github.com/apache/airflow/pull/9684#issuecomment-654163965
> @potiuk I made the update. > > Not sure why I had to do this though, isn't is possible to declare depencies in a pyproject.toml with python version ^3.6 ? > Wouldn'it do the same ? It is explained few paragraphs above: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#airflow-dependencies Airflow is not a standard project - it's both an "application" (for people installing it) and "library" (for people writing DAGs). Usually applications have pinned dependencies and libraries have dependencies as opened as possible. This article is one of many that explains why: http://blog.chrisgorgolewski.org/2017/12/to-pin-or-not-to-pin-dependencies.html . Since we are a bit of both, we had to come up with a clever handling of deps: * In setup.py we have as open deps as possible so that you can upgrade to newer version of depended library when it is released so that you can develop DAGs using latest compatible versions of libraries * In "requirements.txt" we keep the pinned requirement file that you can use as --constraint when installing airflow as "application" (via pip install) - see [INSTALL](INSTALL) for recommended way of installing airflow and some more options in CONTIBUTING.rst where you can get the constraint files directly from GitHub. This way you can be sure that you can have working airflow installed no matter if someone released a new version of their libraries that breaks it (happened many times in the past) The "Genereate requirements" process is our way to keep the pinned dependencies updated regularly and our test suite makes sure that when you generated a new dependencies, those are used during tests and we know that those new dependencies did not break airflow (if we find that they do, we either fix it or limit versions in setup.py - it happened many time in the past that we experienced this problem). There are subtle differences between different python versions for those - you can compare those generated deps. - that's why have separate constraint file for each version of python. I hope that explains why our process is a bit complex than with a regular library or application (yes, I'd love to use [poetry](https://python-poetry.org/) for that but poetry does not support this application <> library dualism we have). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
