GitHub user Sxnan edited a discussion: Flink Agents Release Process
## One-time setup instructions ### GPG Key You need to have a GPG key to sign the release artifacts. Please be aware of the ASF-wide [release signing guidelines](https://www.apache.org/dev/release-signing.html). If you don’t have a GPG key associated with your Apache account, please create one according to the guidelines. Determine your Apache GPG Key and Key ID, as follows: ```plaintext gpg --list-keys ``` This will list your GPG keys. One of these should reflect your Apache account, for example: ```plaintext -------------------------------------------------- pub 2048R/845E6689 2016-02-23 uid Nomen Nescio <[email protected]> sub 2048R/BA4D50BE 2016-02-23 ``` Here, the key ID is the 8-digit hex string in the `pub` line: `845E6689`. Now, add your Apache GPG key to the Flink’s `KEYS` file in the [`release`](https://dist.apache.org/repos/dist/release/flink/KEYS) repository at [dist.apache.org](http://dist.apache.org/). Follow the instructions listed at the top of these files. (Note: Only PMC members have write access to the release repository. If you end up getting 403 errors ask on the mailing list for assistance.) PMC member can refer following scripts to add your Apache GPG key to the KEYS in the release repository. ```plaintext svn co https://dist.apache.org/repos/dist/release/flink flink-dist-release-repo cd flink-dist-release-repo (gpg --list-sigs <YOUR_KEY_ID> && gpg --armor --export <YOUR_KEY_ID>) >> KEYS svn ci -m "[flink] Add <YOUR_NAME>'s public key" ``` Configure `git` to use this key when signing code by giving it your key ID, as follows: ```plaintext git config --global user.signingkey 845E6689 ``` You may drop the `--global` option if you’d prefer to use this key for the current repository only. You may wish to start `gpg-agent` to unlock your GPG key only once using your passphrase. Otherwise, you may need to enter this passphrase hundreds of times. The setup for `gpg-agent` varies based on operating system, but may be something like this: ```plaintext eval $(gpg-agent --daemon --no-grab --write-env-file $HOME/.gpg-agent-info) export GPG_TTY=$(tty) export GPG_AGENT_INFO ``` ### Access to Apache Nexus repository Configure access to the [Apache Nexus repository](https://repository.apache.org/), which enables final deployment of releases to the Maven Central Repository. 1. You log in with your Apache account. 2. Confirm you have appropriate access by finding `org.apache.flink` under `Staging Profiles`. 3. Navigate to your `Profile` (top right dropdown menu of the page). 4. Choose `User Token` from the dropdown, then click `Access User Token`. Copy a snippet of the Maven XML configuration block. 5. Insert this snippet twice into your global Maven `settings.xml` file, typically `${HOME}/.m2/settings.xml`. The end result should look like this, where `TOKEN_NAME` and `TOKEN_PASSWORD`are your secret tokens: ```plaintext <settings> <servers> <server> <id>apache.releases.https</id> <username>TOKEN_NAME</username> <password>TOKEN_PASSWORD</password> </server> <server> <id>apache.snapshots.https</id> <username>TOKEN_NAME</username> <password>TOKEN_PASSWORD</password> </server> </servers> </settings> ``` ### Website development setup Get ready for updating the Flink website by following the [website development instructions](https://flink.apache.org/contributing/improve-website.html). ### GNU Tar Setup for Mac Skip this step if you are not using a Mac. The default tar application on Mac does not support GNU archive format and defaults to Pax. This bloats the archive with unnecessary metadata that can result in additional files when decompressing (see [1.15.2-RC2 vote thread](https://lists.apache.org/thread/mzbgsb7y9vdp9bs00gsgscsjv2ygy58q)). Install gnu-tar and create a symbolic link to use in preference of the default tar program. ```plaintext brew install gnu-tar ln -s /opt/homebrew/bin/gtar /usr/local/bin/tar # Make sure which directory "gtar" is installed to, at the time of writing it is "/opt/homebrew/bin/gtar" which tar ``` The core of the release process is the build-vote-fix cycle. Each cycle produces one release candidate. The Release Manager repeats this cycle until the community approves one release candidate, which is then finalized. ## Build and stage Java and Python artifacts Set up a few environment variables to simplify Maven commands that follow. This identifies the release candidate being built. Start with `RC_NUM` equal to `1` and increment it for each candidate. ```plaintext RELEASE_VERSION="0.1.0" SHORT_RELEASE_VERSION="0.1" CURRENT_SNAPSHOT_VERSION="$SHORT_RELEASE_VERSION-SNAPSHOT" NEXT_SNAPSHOT_VERSION="0.2-SNAPSHOT" SHORT_NEXT_SNAPSHOT_VERSION="0.2" RC_NUM="1" TAG="release-${RELEASE_VERSION}-rc${RC_NUM}" ``` Now, create a release branch: ```plaintext cd tools OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$RELEASE_VERSION RELEASE_CANDIDATE=$RC_NUM releasing/create_release_branch.sh ``` Tag the release commit: ```plaintext git tag -s ${TAG} -m "${TAG}" ``` You can use -c "user.signingkey=" as an additional git parameter if you have multiple signing keys in your keychain. We now need to do several things: * Create the source release archive * Deploy jar artefacts to the Apache Nexus Repository, which is the staging area for deploying the jars to Maven Central * Build wheel packages You might want to create a directory on your local machine for collecting the various source and binary releases before uploading them. First, we build the source release: ```plaintext RELEASE_VERSION=$RELEASE_VERSION releasing/create_source_release.sh ``` Next, we stage the maven artifacts: ```plaintext releasing/deploy_staging_jars.sh ``` Review all staged artifacts in the staging repositories([https://repository.apache.org/#stagingRepositories](https://repository.apache.org/#stagingRepositories)). They should contain all relevant parts for each module, including `pom.xml`, jar, test jar, source, test source, javadoc, etc. Carefully review any new artifacts. Close the staging repository on Apache Nexus. When prompted for a description, enter “Apache Flink Agents, version X, release candidate Y“. Then, you need to build the flink-agents wheel packages. 1. Push the release candidate branch to your forked personal Flink repository, e.g. ```plaintext git push <remote> refs/heads/release-${RELEASE_VERSION}-rc${RC_NUM}:release-${RELEASE_VERSION}-rc${RC_NUM} ``` 2. Trigger the “Build Wheel“ GitHub workflow manually 1. Go to your GitHub fork website page 2. Under the “Actions“ tab, click “Build Wheel“ 3. Select your recently pushed release candidate branch 4. You should now see that a new run is triggered 3. Download the wheel packages from the triggered “Build Wheel“ run artifacts after finished. 1. Download the wheel packages 1. Open the workflow run result page 2. Scroll down to the “Artifacts“ section 3. Click `wheel-1` download the zip files 2. Unzip the zip file ```plaintext cd /path/to/downloaded_wheel_packages mkdir wheels unzip wheel-*.zip -d wheels ``` 3. Create directory `dist` under the directory of `python` ```plaintext cd <flink-agents-dir> mkdir python/dist ``` 4. Move the unzipped wheel packages to the directory of `python/dist` ```plaintext mv /path/to/wheels/* python/dist/ ``` Finally, we create the binary convenience release files: ```plaintext cd tools RELEASE_VERSION=$RELEASE_VERSION releasing/create_binary_release.sh ``` ### Stage source and binary releases on [**dist.apache.org**](http://dist.apache.org/) Copy the source release to the dev repository of dist.apache.org. 1. If you have not already, check out the Flink section of the `dev` repository on via Subversion. In a fresh directory: ```plaintext svn checkout https://dist.apache.org/repos/dist/dev/flink --depth=immediates ``` 2. Make a directory for the new release: ```plaintext mkdir flink/flink-agents-${RELEASE_VERSION}-rc${RC_NUM} ``` Copy Flink source/binary distributions, hashes, and GPG signature and the python subdirectory: ```plaintext mv <flink-agents-dir>/tools/releasing/release/* flink/flink-agents-${RELEASE_VERSION}-rc${RC_NUM} ``` 3. Add and commit all the files. ```plaintext cd flink svn add flink-agents-${RELEASE_VERSION}-rc${RC_NUM} svn commit -m "Add flink-agents-${RELEASE_VERSION}-rc${RC_NUM}" ``` 4. Verify that files are [present](https://dist.apache.org/repos/dist/dev/flink) **(Push the release tag)** If you haven't pushed the release tag yet, here's the command: ```plaintext git push <remote> refs/tags/release-${RELEASE_VERSION}-rc${RC_NUM} ``` ### Propose a pull request for website updates Add a blog post announcing the release, e.g., [Release 0.1.0 blog post](https://github.com/apache/flink-web/pull/809) ### Vote on the release candidate As a sub-project from the Apache Flink community, once you have built and individually reviewed the release candidate, please share it for the Apache Flink community-wide review, e.g. [VOTE](https://lists.apache.org/thread/slw2nhjpst2ngrpbzhpbwz5md6s6f59y). Here is an email template, please adjust as you see fit. ``` From: Release Manager To: [email protected] Subject: [VOTE] Release 1.2.3, release candidate #3 Hi everyone, Please review and vote on the release candidate #3 for the version 1.2.3, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * the official Apache source release and binary convenience releases to be deployed to dist.apache.org [1], which are signed with the key with fingerprint FFFFFFFF [2], * all artifacts to be deployed to the Maven Central Repository [3], * source code tag "release-1.2.3-rc3" [4], * website pull request listing the new release and adding announcement blog post [5]. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Thanks, Release Manager [1] link [2] https://dist.apache.org/repos/dist/release/flink/KEYS [3] link [4] link [5] link ``` The vote should be open for at least 72 hours, and accept majority approval, with at least 3 PMC affirmative votes. See section _Vote on the release candidate_ in [Creating a Flink Release](https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release) for more details. ## Finalize the release Once the release candidate has been reviewed and approved by the community, the release should be finalized. This involves the final deployment of the release candidate to the release repositories, merging of the website changes, etc. ### Deploy Python artifacts to PyPI Release manager should create a PyPI account and ask the PMC to add this account to pyflink collaborator list with Maintainer role to deploy the Python artifacts to PyPI. The artifacts could be uploaded using twine([https://pypi.org/project/twine/](https://pypi.org/project/twine/)). To install twine, just run: ```plaintext pip install --upgrade twine==1.12.0 ``` Note: Please ensure that the version of `urllib3` is less than `2.0` in your python env, otherwise you may encounter some problem like `unexpected keyword argument method_whitelist`. Download the python artifacts from [dist.apache.org](http://dist.apache.org/) and upload it to [pypi.org](http://pypi.org/): ```plaintext svn checkout https://dist.apache.org/repos/dist/dev/flink/flink-agents-${RELEASE_VERSION}-rc${RC_NUM} cd flink-agents-${RELEASE_VERSION}-rc${RC_NUM} cd python #uploads wheels for f in *.whl; do twine upload -r pypi $f; done ``` PyPI has removed GPG signatures support, so it is recommended to no longer upload signatures to PyPI. If upload failed or incorrect for some reason(e.g. network transmission problem), you need to delete the uploaded release package of the same version(if exists) and rename the artifact to flink-agents-${RELEASE\_VERSION}.post0.tar.gz, then re-upload. Note: re-uploading to pypi.org must be avoided as much as possible because it will cause some irreparable problems. If that happens, users cannot install the flink-agents package by explicitly specifying the package version, i.e. the following command "pip install flink-agents==${RELEASE\_VERSION}.post0" to install the package. ### Deploy artifacts to Maven Central Repository #### Deploy artifacts to Maven Central Repository Use the [Apache Nexus](https://repository.apache.org/) repository to release the staged binary artifacts to the Maven Central repository. In the `Staging Repositories` section, find the relevant release candidate `orgapacheflink-XXX` entry and click `Release`. Drop all other release candidates that are not being released. #### Deploy source and binary releases to [**dist.apache.org**](http://dist.apache.org/) Copy the source and binary releases from the `dev` repository to the `release` repository at using Subversion. ```plaintext svn move -m "Release Flink Agents ${RELEASE_VERSION}" https://dist.apache.org/repos/dist/dev/flink/flink-agents-${RELEASE_VERSION}-rc${RC_NUM} https://dist.apache.org/repos/dist/release/flink/flink-agents-${RELEASE_VERSION} ``` (Note: Only PMC members have access to the release repository. If you do not have access, ask on the mailing list for assistance.) #### Remove old release candidates from [**dist.apache.org**](http://dist.apache.org/) Remove the old release candidates from [https://dist.apache.org/repos/dist/dev/flink](https://dist.apache.org/repos/dist/dev/flink) using Subversion. ```plaintext svn checkout https://dist.apache.org/repos/dist/dev/flink --depth=immediates cd flink svn remove flink-agents-${RELEASE_VERSION}-rc* svn commit -m "Remove old release candidates for Apache Flink Agents ${RELEASE_VERSION}" ``` ### Git tag Create and push a new Git tag for the released version by copying the tag for the final release candidate, as follows: ```plaintext git tag -s "release-${RELEASE_VERSION}" refs/tags/${TAG}^{} -m "Release Flink Agents ${RELEASE_VERSION}" git push <remote> refs/tags/release-${RELEASE_VERSION} ``` ## Promote the release Once the release has been finalized, the last step of the process is to promote the release within the project and beyond. Please wait for 24h after finalizing the release in accordance with the [ASF release policy](http://www.apache.org/legal/release-policy.html#release-announcements). ### Merge website pull request Merge the website pull request to [list the release](http://flink.apache.org/downloads.html). Make sure to regenerate the website as well, as it isn't build automatically. ### Apache mailing lists Announce on the dev@ mailing list that the release has been finished. Announce on the release on the user@ mailing list, listing major improvements and contributions. Announce the release on the [email protected] mailing list. ```plaintext From: Release Manager To: [email protected], [email protected], [email protected], [email protected] Subject: [ANNOUNCE] Apache Flink Agents 0.1.0 released The Apache Flink community is very happy to announce the release of Apache Flink Agents 0.1.0. Apache Flink Agents® is a sub-project from the Apache Flink community, providing an open-source framework for building event-driven streaming agents. The release is available for download at: https://flink.apache.org/downloads.html Please check out the release blog post for an overview of the improvements for this bugfix release: <blob post link> We would like to thank all contributors of the Apache Flink community who made this release possible! Feel free to reach out to the release managers (or respond to this thread) with feedback on the release process. Our goal is to constantly improve the release process. Feedback on what could be improved or things that didn't go so well are appreciated. Regards, <Release Manager> ``` ### Recordkeeping Use [reporter.apache.org](https://reporter.apache.org/addrelease.html?flink) to seed the information about the release into future project reports. (Note: Only PMC members have access report releases. If you do not have access, ask on the mailing list for assistance.) ## Improve the process It is important that we improve the release processes over time. Once you’ve finished the release, please take a step back and look what areas of this process and be improved. Perhaps some part of the process can be simplified. Perhaps parts of this guide can be clarified. GitHub link: https://github.com/apache/flink-agents/discussions/294 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
