potiuk commented on issue #11433: URL: https://github.com/apache/airflow/issues/11433#issuecomment-711167926
As part of the #11546 fixing I prepared a PR #11630 where I think I got it really sorted out very well: 1) we generate setup.py and config.py dynamically based on provider and information found in the Airflow's setup.py (dependencies etrc.). Those are automatically generated from JINJA templates for those files the same way we generate README.md and CHANGES files. 2) In the procedure of preparing such provider releases, those files are regenerated (using ./breeze prepare-provider-readme) and committed to the repository in the appropriate "airfllow/providers/<PROVIDER>" dir. We have (for now) two versions of those setup.py, setup.cfg, README, CHANGES files (one for backports, and one for regular providers). Whe the backports reach end-of-life we will be able to get rid of the BACKPORT)ORIVDERS_ ones. 3) when prepping the packages, I copy the sources to "provider_packages" - one, because it is needed to do refactor for backports and two - because the "setuptools" only works when setup.py, setup,cfg, README.md, CHANGELOG,txt are named as such. If we do it in main directory, we will be overwriting airflow ones. And we cannot do it easily in "providers" folder, because also setuptools require "airflow/providers" subdir for python files. I think this can stay like that permanently. The nice thing about the final result (sdist packages) is that those .tar.gz packages when uncompressed, only contain sources of the provider they relate to + setup.py + setup.cfg + README.md + CHANGELOG.txt + LICENCE + NOTICE that makes sense for that provider, so we are now generating packages that "look like" and "behave like" packages prepared from separate repositories. Which is cool. I think - if everyone else agrees with me - that this one can be closed once we merge #11630 WDYT? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
