potiuk edited a comment on issue #11546: URL: https://github.com/apache/airflow/issues/11546#issuecomment-711156321
I already have a working version of this. @mjpieters (but also @kaxil and @ashb ). And I have one question on what you think Kaxil and Ash regarding releasing the providers officially. We've agreed that we will release the sources for each provider separately, and while I implemented this issue I think I achieved exactly this by fixing the broken sdist package (as pointed out by @mjpieters - thanks again!). According to ASF release policies, we need to make sure to release a source package that must be sufficient for the user to rebuild the packages. > SOURCE PACKAGES Every ASF release MUST contain one or more source packages, which MUST be sufficient for a user to build and test the release provided they have access to the appropriate platform and tools. And when you look at the sdist packages (with fixed setup.py) for each provider does exactly this. If you see at the artifacts produced during the build here - my test CI build (artifacts will appear soon in https://github.com/potiuk/airflow/actions/runs/313648146), both providers and backport provider's sdist packages are exactly this - they allow anyone with the appropriate platform and sources to rebuild those provider packages. The PR for that change is opened now: https://github.com/apache/airflow/pull/11630 - I am still running some final reviews and fixes with the generated packages, but it should be fine. Now. The question I have: for Airflow, we have a separate -src.tar.gz file, which contains quite a bit more than the sdist .tar.gz package: Dockerfile, scripts, open API specification, and all the auxiliary stuff that we want to also release. I believe for providers, we do not need any of those, and we do not need anything more than the sdist tar.gz file. So IMHO we do not need to release any additional src-tar.gz file. Let me know what you think @ashb @kaxil! BTW: I think I did quite a nice refactor and simplification of package generation with those changes. I still generate the packages dynamically - (the setup.py files are generated from a JINJA template for each provider (separate setup.py for backports and separately for the regular providers). Those setup.py will be updated and stored in the repo - separate setup.py for each provider in the relevant packages - this way we will keep track of those setup.py files via git history, which is pretty nice I think. I also test installation of all the generated packages (both .whl and sdist now - not only .whl as it used to be) on 1.10.12 for backport and on the latest "bare" airflow from the master for the regular provider packages (separate jobs in CI for those). This is all on top of verifying if all the operators/hooks/ etc. are importable (in 1.10.12 and master respectively). I think we are getting to a nice solution there. Once we move to betas and implement extras dependencies #11464 and later implement all the discovery features so that Airflow can automatically use those provider's extra links etc. (TODOs in https://github.com/apache/airflow/projects/5) - I think it's a really nice approach we are going to have. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
