potiuk commented on pull request #11950: URL: https://github.com/apache/airflow/pull/11950#issuecomment-826546862
For Python 3.9 I think I will simply go with the #15515 as in order to merge it there will be one more "tricky" point which is to make sure that Python 3.9 is also built in the "build images" workflow. Once this will be merged, I think it will have all the changes from that PR so I think we will be able to close this one (I will be happy to make you co-author of the #15515 :). > Ty for the background; Given that I understand why airflow won't switch off of pip. That said what do you think of instead introducing tests that use poetry or pipenv to ensure that the requirements we have are always valid. i.e. the tests would run lock to make sure no conflicting requirements slip into airflow. PIP 21 (PR to be merged in #15513) already has a working resolver that is detecting conflicts much better and we will continue using it to prevent conflicts once it is merged. So this problem will be solved. I think once teething problems of the new resolver have been solved and we will be able to solve all the current problems (with #15513). I think the new PIP removes any need for any extra tools like poetry or pipenv (and is much more versatile IMHO). @uranusjr maybe you can comment on that one. The problem we have (and it is also described in https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#airflow-dependencies, Airflow is both library and application. And constraint mechanism allows us to handle it well. * On one hand we can keep the dependencies open in `setup.py` (so that the users of airflow are not limited and can upgrade dependencies after Airflow is released) * At the same time, with the --constraint switch and publishing "golden" set of constraints we are able to make sure that users installing released version of Airflow can also install it in the future. AFAIK (and please correct me if I am wrong) neither Poetry nor PIPEnv bring anything more, actually the workflows they support are limiting. They are more opinionated and allow either the library approach (everything as open as possible) or the application one (everything fixed with poetry/pipenv .lock file). From what I understand, the information about dependency versions from those .lock files is not available in the PyPI after packages are released. You either "bake it" into the package (for application) or strip it (for library). Unlike PIP, as far as I know neither Poetry nor Pyenv has the remote URL for lock-files capability that `--constraint` flag of PIP gives us - we can take "open-dependency" Airflow and point to the "fixed" constraints that we store in an orphaned branch in our GitHub Repo and voila - open-dependency "Airflow" is installed in a repeatable way with "golden" set of dependencies. However I am really open to add "support" for people who are using Poetry or PIPenv. I know there are many users of poetry/pipenv out there. If you are one of the poetry/pipenv users and could write a nice HowTo - how to take Airflow constraints and install airflow with Poetry and Pipenv I am all ears. It would have to use somehow the constraint file we have - otherwise it will not work in the future with the open-ended approach of Airlfow dependencies. I looked at it briefly, but I could not find a one-line equivalent of `pip install apache-airflow[extras]==2.0.2 --constraints https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.7.txt` that we have for PIP installation. If this means that we will have to store poetry.lock file in our orphan branch next to constraints, I am more than happy to help with automating it. Would you be willing to help with that and prepare/maintain such "instructions" ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
