potiuk commented on pull request #13767: URL: https://github.com/apache/airflow/pull/13767#issuecomment-763025828
Hey - especially @kaxil and @ashb - you've been waiting for this one and I think it is finally ready for review. I've refactored the original code of 'prepare_provider_packages.py" to simplify it a bit and to prepare for removal of backports in a few month's time. It should still work fine for backports (we will see it soon in CI) - but I mainly focused on getting the workflow agreed before in https://github.com/apache/airflow/issues/11425#issuecomment-749647496 I separated it out into two ommits: * First commit smply reshufles the generated documentation - it removes the old one generated and replaces them with just CHANGELOG.rst that will be kept per-provider. * The second one implements the logic that allows to prepare and release the providers in bulk. I described all the process in the dev/readme\s as usual, but this time iI also explained how you can easily debug the process step by step by manually running all the scripts. You do not have to use `breeze` to prepare provider's documentation you can simply follow the "debugging" steps explained in dev/README.md: 1) import check - to check if all provider classes can be imported 2) verification if all provider classes are properly named 3) preparing the documentation - either for selected or for all providers that are ready for it 3) generating setup/manifest/provider_info fikes - either for selected or for all providers that are ready for it 4) preparing the packages Breeze encapsulates all of that in two commands, and has support for "bulk" operations on multiple providers. So for the "release" process `breeze` should be rather useful. More about how the process looks like from the release manager perspective: 1) Preparing the release documentation and reviewing providers to release First, you can quickly see which packages should be released. When running `./breeze prepare-provider-documentation` without any parameters, it will loop through all providers and prints changes vs. the last release. Additio naly, if already there is a new version set for a package in 'provider.yaml' it will update the documentation for that provider and update the index.rst of the provider with the appropriate information: This is what you will see for packages that do not have a new version set in `provider.yaml` :  This way you will see the summary of all those changes and providers that might be candidates for releases. The summary of changes has all the links to commits - you can click through from the terminal directly. Additionally, you will see information if all the provider classes are properly named (the step will fail here and tell you if they are not):  For packages that already have a new version in `provider.yaml` (in our case I set 2.0.0 for the upcoming release of Google provider) you will see the information that the index has been updated and "diff" of the index.rst vs. the last time it was generated:  As a release manager, you can re-run this after merging new commits and it will be correctly updated (and you will see the incremental diffs every time you update). it's safe to commit those updated index.rst files. At the end of such run, you will see a summary showing for which packages documentation was generated and for which it was skipped (or errors). Skipped is when no newer version was added to `provider.yaml` vs. the already released and tagged one:  The documentation generated is added to index.rst in the "doc" folder, so when we release the docs, we can simply point to it from the README (rather than add the changelog to the package README). The documentation in the provider's index contains both - High Level CHANGELOG.rst content and detailed per-commit changelog. You can see example Google Provider 2.0 docs generated this way: http://gabby-cough.surge.sh/ Once you (as release manager) update versions of the providers in other provider.yaml files this `./breeze prepare-provider-documentation` command can be rerun and documentation for all the providers ready to released will be updated automatically in-bulk (and only for those). All the providers that have the same version as already released tags, will be skipped. 2) Generating the providers Similar to generating documentation, provider packages can be generated in-bulk: `./breeze prepare-provider-packages --package-format both` The above will go through all providers and generate the providers which are ready to be released (i.e those that have updated version in provider.yaml (and hopefully documentation generated and committed in the previous step). You will see the information about skipped packages (and why they are skipped) and green confirmation for those packages that are generated: Google one:  and neo4j (it has never been generated so this is the first time it is generated):  At the end you will see the summary showing the generated/skipped/errored packages:  I think it is quite robust already, and the same code is run in the CI, so I hope it will be green soon, in the meantime - that's quite a lot of code, so probably good time to start reviewing. Unfortunately the change is significant and it's hard to split it , the best split I could do is to separate out the "docs" from the code (which I did - in separate commits). But I am also open if you have other suggestions on how to split it (but i think it ain't easy). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
