mik-laj commented on a change in pull request #8807: URL: https://github.com/apache/airflow/pull/8807#discussion_r422770469
########## File path: dev/BACKPORT_PACKAGES.md ########## @@ -0,0 +1,119 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Backport packages + +# Overview + +This document describes process of preparing backport packages for release and releasing them. +Backport packages are packages (per provider) that make possible to easily use Hooks, Operators, +Sensors, Protocols and Secrets from the 2.0 version of Airflow in the 1.10.* series. + +Once the packages are released you can simply install them with + +``` +pip install apache-airflow-backport-providers-<PROVIDER>[<EXTRA>] +``` + +Where <PROVIDER> is the provider id and <EXTRA> is optional extra packages to install. The +provider packages dependencies and extras are described in the README.md files in each provider +package (in `airflow/providers/<PROVIDER>` folder) as well as in the PyPI installation page. + +Backport providers are great way to migrate your DAGs to Airflow-2.0 compatible DAGs. You can +switch to the new Airflow-2.0 packages in your DAGs, long before you attempt to migrate +airflow to 2.0 line. + +## Deciding when to release + +Backport packages are released separately on ad-hoc basis, whenever we find that given provider needs +to be released - due to new features or due to bugfixes. Each backport package is released separately +- although we decided to release all backport packages together in one go 2020.05.10. + +We are using [CALVER](https://calver.org/) versioning scheme for the backport packages. We also have +automated way to prepare and build the packages, so it should be very easy to release the packages often and +separately. + +## Preparing release notes + +When you want to release the package you simply need to run: + +``` +./scripts/ci/ci_generate_backport_readme.sh YYYY.MM.DD <PACKAGE_ID> ... +``` + +* YYYY.MM.DD - is the CALVER version of the package to prepare. Note that this date cannot be earlier + than already release version (the script will fail if it will be) and it can be set in the future + anticipating future release date. + +* <PACKAGE_ID> is usually directory in the `airflow/providers` folder (for example `googl` but in several + cases it might be one level deeper separated with `.` for example `apache.hive` + +You can run the script with multiple package names if you want to prepare several packages at the same time. +You can also re-run the script with the same version (date) - this way - in case you have any bugfixes +merged in the master, they will be automatically taken into account. + +Whenever you are satisfied with the release notes generated you can commit generated changes/new files +to the repository. + +Before preparing the release you must also update the version to release in the +`backport_packages/setup_backport_packages.py` - this is needed because you should add rc1/rc2 etc. +before relasing final version and only when you are ready, the final version should be released and updated +in the `backport_packages/setup_backport_packages.py`. + +## Content of the release notes + +The script nicely generates all the necessary information: + +* summary of requirements for each backport package +* list of dependencies (including extras to install them) when package + depends on other providers packages +* table of new hooks/operators/sensors/protocols/secrets +* table of moved hooks/operators/sensors/protocols/secrets with + information where they were moved from +* changelog of all the changes to the provider package (this will be + automatically updated with incremental changelog whenever we decide to + release separate packages. + +Note that our CI system builds the release notes for backport packages automatically with every build and +current date - this way you might be sure the automated generation of the release notes continues to +work and you can preview the generated readme files (by downloading artifacts uploaded to file.io). +Readme files and changes are not modified if there is no change in repo for that provider. + + +## Building the packages + +Before building packages, make sure that you update and commit the version in the +`backport_packages/setup_backport_packages.py` + +Packages can bye build by running: Review comment: ```suggestion Packages can be built by running: ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
