potiuk commented on code in PR #33144: URL: https://github.com/apache/airflow/pull/33144#discussion_r1285856941
########## dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md: ########## @@ -73,6 +74,89 @@ git commit -m "Your commit message here" --no-verify git push ``` +# Manually updating already tagged constraint files + +Sometimes - very rarely - we need to fix historical constraint files when Airflow fails to install with the +constraints that were used in the past. This happened already several times and usually only happens when +there is a backwards-incompatible change in the build environment in Python installation toolchain +(pip, setuptools, wheel, Cython etc.). The Python build environment is not controllable by us - by default +pip uses `build isolation` which means that it will install the latest version of the build tools that +are matching the expectations of the package being build and it might mean that new version of such tools can +break installation. This happened for example in July 2023 when major (3.0.0) version of Cython has +been released and it broke `pymssql` installation - we had to update the constraint file to use `pymssql` 2.2.8 +instead of `2.2.7` because 2.2.7 did not limit but also did not work with the new version of Cython. + +Breeze has `update-constraints` command in `release-management` group that can be used to update the +constraints. + +The way how updating constraints work: + +1. You need to have "airflow" repository checked out separately from the repository you are working on. For + example in `/home/myuser/airflow-constratints` folder. +2. You need to checkout `constraints-main` branch in this repository. By default the command expects that + there is a remote named "apache" pointing to the official Apache repository. You can override this + by passing `--remote-name` option to the command. +3. You need to run `breeze release-management update-constraints` command. You can pass various options to + the command: + * path to the "constraints" repository + * remote name (optionally - default "apache") + * list of airflow versions to update constraints for + * list of constraints to update in the form of "package==version" (you can specify more than one) + * message to be used in commit message + + Make sure you use exactly the same case for the package to be updated as the one already in the + constraints. + + It's a good idea to add `--dry-run` option to the command to see what will be updated before you actually + run the command. However, even if you do not use `--dry-run` option, the command will ask you to + confirm the updates so you will have a chance to verify it before each version change. + + You can also add `--verbose` instead of `--dry-run` and you will see the git commands being executed by + the command while it is doing its job. + + Notes about "potentially breaking" constraints: the command is designed to work in a very safe way + and provide ways of coming back to the previous state manually if needed. + + Tags are moved with ``--force`` option - this needs to be done because we are moving already existing tag, + however branches are pushed without force so there is no risk of losing history in the repository - you + can always see the history and revert the changes and restore old tags manually. Usually the "final" tags + are the same as the latest "rc*" tags for the same version so it is easy to find where the tag was + pointing before - we also print hash of the commits before attempting to make modifications so you can + always see what commit the tag has been pointing to before the command is run. + +4. The command will do the following for every Airflow version specified: + * checkout "constraints-<version>" tag + * reset "constraints-<version>-fix" branch to the tag + * update constraints in-place + * commit the changes + * tag the commit with "constraints-<version>" tag + * push the "constraints-<version>-fix" branch with the commit to the remote selected + * push the tag to the remote selected + +You should verify manually if the change is as expected by inspecting the constraints at + +https://github.com/apache/airflow/tree/constraints-<airfow-version> + +Example of updating constraints for Airflow 2.5.0 - 2.6.3 and updating `pymssql` constraint to 2.2.8: + +```bash +breeze release-management update-constraints --constraints-repo /home/user/airflow-constraints \ + --airflow-versions 2.5.0,2.5.1,2.5.2,2.5.3,2.6.0,2.6.1,2.6.2,2.6.3 \ + --updated-constraint pymssql==2.2.8 \ + --message "Update pymssql constraint to 2.2.8" +``` + +Example of updating multiple constraints: + +```bash +breeze release-management update-constraints --constraints-repo /home/user/airflow-constraints \ + --airflow-versions 2.5.0,2.5.1,2.5.2,2.5.3,2.6.0,2.6.1,2.6.2,2.6.3 \ + --updated-constraint pymssql==2.2.8 \ + --updated-constraint Authlib==1.3.0 \ + --message "Update pymssql constraint to 2.2.8 and Authlib to 1.3.0" +``` + Review Comment: Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
