potiuk commented on issue #10038: URL: https://github.com/apache/airflow/issues/10038#issuecomment-665131906
Hello Noel, Indeed the problem you describe is very true for Applications but it is quite the opposite for libraries. Unfortunately Airflow is a bit of both, that's why we had to find a bit different solution - keeping the setup.py dependencies as open as possible but allowing Airflow as application to be installed with frozen set of dependencies. Those are frozen both - for master (we automatically test and re-freeze them continuously after testing during our CI process) but also we freeze the fixed set of dependencies whenever we release a stable version of Airflow (and you can very easily use either the latest or the 1.10.x frozen version at any time by providing the right `--constraint` URL. You can take a look at our 'Installimg from PyPI' chapter of README.md - https://github.com/apache/airflow#installing-from-pypi where it is explained how to install Airflow in repeatable way (using constraint files we maintain automatically with all the dependencies frozen). Also INSTALL.txt contains the right instructions (note that they changed a bit in the upcoming version but you can find the corresponding instructions in 1.10.10 and 1.10.11 tags of the same files - for 1.10.10 and 1.10.12. the current set of instructions will be valid as of 1.10.12) For detailed instructions and explanation why we had gone that route, you can see https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#airflow-dependencies (also the following chapter 'Pinned constraint files' where we explained how we addressed it). As mentioned - it's the second iteration of approach to fix it. The first one in 1.10.10 and 1.10.11 has proven to be too 'heavy' for maintenance by contributors, so we improved it, simplified it, and we fully automated the process of dependency freezing last week (and it will be released in 1.10.12). I hope this addresses your concerns. Please read those links I sent and let us know if it does or does not address your concerns (and let Is know please why - if not). Also if you have any concrete proposals how we can improve the current setup without breaking our 'both library and app' requirement - we are happy to hear it. Whether it is the documentation or the process improvement proposals. I am particularly interested in how we can surface information about it better. It seems that despite all our efforts to explain it in INSTALL.txt, README.md and CONTRIBUTING.rst - it's not enough as people sometimes do not find it easily, so I would love to hear some suggestions and proposals how we could improve it - especially from person not familiar with the process and whole reasoning. The most welcome is a PR improving it that we can review and merge :). J wt., 28 lip 2020, 16:57 użytkownik Noël Bardelot <[email protected]> napisał: > From an ops point of view, packaging Airflow with about 150 dependencies > not strictly tied to a version is a bad practice. In setup.py there are > about 20 dependencies using =~ versions and about 130 dependencies using > >= versions. This means there is no guarantee that two instances of > Airflow run the same verions of python libraries. > > Being forced to manually compute dozens of dependency versions and force > them in requirements.txt is not a viable solution. > > Airflow should freeze its dependency tree when a new stable version comes > out. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <https://github.com/apache/airflow/issues/10038>, or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAERMI3VCH6PZ5ASZEYDYV3R53RMNANCNFSM4PKTPFRQ> > f > ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
