potiuk opened a new pull request, #35586:
URL: https://github.com/apache/airflow/pull/35586
This PR moves the functionality of preparing provider packages from a python
script inside Breeze CI image to breeze Python package.
This is the first of the series of moves that will simplify the way how
provider packages are being built and prepared with the aim of improving
security of the supply chain and make it easier to debug and modify the release
process.
Historically, release process has been run inside of Breeze for several
reasons: isolation of running package preparation from Host environment, the
need to keep separate virtualenv and because we run verification of provider
packages during release process - which requires the CI environment with all
its dependencies.
So far the process looked like this:
* bash breeze parsed the arguments
* bash breeze started the docker bash script with packages as parameters
* the bash script in CI image looped over the packages and run python
prepare_provider_packages.py (twice) to generate docs and update changelog
(this is interactive process where release manager makes decision on bumping
versions). Those python script performed verification on provider.yaml files
* the bash script summarized the packages and displayed status of preparation
However after moving to Python based breeze, we can simplify it all and run
all those steps in Python breeze internal code - no need to go to docker and
use bash scripts. We also do not have to do verification of provider.yaml files
is done already extensively in pre-commit.
This PR moves all this logic to inside Breeze.
There is stil remainig duplicated code in the original in-container
`prepare_provider_packages.py` remaining, this duplication will be removed by
subsequent PRs where other release management commands for provider packages
will also be moved to Breeze as follow-up of this PR.
This PR has the following changes:
* move the provider decumentation code from `dev/provider_packages` to
`dev/breeze/` (and from in-container to in-breeze-venv execution)
* completely removed the intermediate bash script and calling Python scripts
from it - moving the logic to Breeze entirely
* added better diagnostics of what happens when packages are classified with
particular types of changes (added `special` style to show it)
* cleaned and clarified `prepare-provider-documentation` commmand line flags
* introduce explicit "non-interactive" mode that is used to run and test the
command in CI and to test it locally
* replace str with Path were files were used in the moved code
* add unit tests covering unit-testable parts of the moved code
* refactore the moved code to use utils available in Breeze
* split the code into packages and versions (reusable utils) and specific
code for preparing package documentation
* cached provider.yaml information retrieved from providers
* move provider documentation templates to Breeze
* better error handling - they are now regular exceptions in Python process
rather than interpreting the exit_codes passed from python sub-scripts returnig
the codes to Bash
* when release manager classifies package, only relevant sections are
generated (Features/Breaking changes) based on decision and changes are
automatically "guessed" only if release manager chose the section where they
would fall in
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
