This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit 33288e94a72db1dee23e861becf6921a16bb2351 Author: Boyuan Zhang <[email protected]> AuthorDate: Wed Aug 8 17:35:59 2018 -0700 Add instructions of using automation scripts --- src/contribute/release-guide.md | 373 +++++++++++++++++++++++++++------------- 1 file changed, 255 insertions(+), 118 deletions(-) diff --git a/src/contribute/release-guide.md b/src/contribute/release-guide.md index a120f0f..73a5840 100644 --- a/src/contribute/release-guide.md +++ b/src/contribute/release-guide.md @@ -77,41 +77,59 @@ To prepare for each release, you should audit the project status in the JIRA iss You need to have a GPG key to sign the release artifacts. Please be aware of the ASF-wide [release signing guidelines](https://www.apache.org/dev/release-signing.html). If you don’t have a GPG key associated with your Apache account, please create one according to the guidelines. -Get more entropy for creating a GPG key +There are 2 ways to configure your GPG key for release, either using release automation script(which is recommended), +or running all commands manually. - sudo apt-get install rng-tools - sudo rngd -r /dev/urandom +##### Use [preparation_before_release.sh](https://github.com/apache/beam/blob/master/release/src/main/scripts/preparation_before_release.sh) to setup GPG +* Usage + ``` + ./beam/release/src/main/scripts/preparation_before_release.sh + ``` +* Tasks included + 1. Help you create a new GPG key if you want. + 1. Configure ```git user.signingkey``` with chosen pubkey. + 1. Add chosen pubkey into [dev KEYS](https://dist.apache.org/repos/dist/dev/beam/KEYS) and [release KEYS](https://dist.apache.org/repos/dist/release/beam/KEYS) + + **NOTES**: Only PMC can write into [release repo](https://dist.apache.org/repos/dist/release/beam/). + 1. Start GPG agents. -Create a GPG key - - gpg --full-generate-key +##### Run all commands manually -Determine your Apache GPG Key and Key ID, as follows: +* Get more entropy for creating a GPG key - gpg --list-keys + sudo apt-get install rng-tools + sudo rngd -r /dev/urandom -This will list your GPG keys. One of these should reflect your Apache account, for example: +* Create a GPG key + + gpg --full-generate-key - -------------------------------------------------- - pub 2048R/845E6689 2016-02-23 - uid Nomen Nescio <[email protected]> - sub 2048R/BA4D50BE 2016-02-23 +* Determine your Apache GPG Key and Key ID, as follows: + + gpg --list-keys + + This will list your GPG keys. One of these should reflect your Apache account, for example: + + -------------------------------------------------- + pub 2048R/845E6689 2016-02-23 + uid Nomen Nescio <[email protected]> + sub 2048R/BA4D50BE 2016-02-23 -Here, the key ID is the 8-digit hex string in the `pub` line: `845E6689`. + Here, the key ID is the 8-digit hex string in the `pub` line: `845E6689`. -Now, add your Apache GPG key to the Beam’s `KEYS` file both in [`dev`](https://dist.apache.org/repos/dist/dev/beam/KEYS) and [`release`](https://dist.apache.org/repos/dist/release/beam/KEYS) repositories at `dist.apache.org`. Follow the instructions listed at the top of these files. (Note: Only PMC members have write access to the release repository. If you end up getting 403 errors ask on the mailing list for assistance.) + Now, add your Apache GPG key to the Beam’s `KEYS` file both in [`dev`](https://dist.apache.org/repos/dist/dev/beam/KEYS) and [`release`](https://dist.apache.org/repos/dist/release/beam/KEYS) repositories at `dist.apache.org`. Follow the instructions listed at the top of these files. (Note: Only PMC members have write access to the release repository. If you end up getting 403 errors ask on the mailing list for assistance.) -Configure `git` to use this key when signing code by giving it your key ID, as follows: +* Configure `git` to use this key when signing code by giving it your key ID, as follows: - git config --global user.signingkey 845E6689 + git config --global user.signingkey 845E6689 -You may drop the `--global` option if you’d prefer to use this key for the current repository only. + You may drop the `--global` option if you’d prefer to use this key for the current repository only. -You may wish to start `gpg-agent` to unlock your GPG key only once using your passphrase. Otherwise, you may need to enter this passphrase hundreds of times. The setup for `gpg-agent` varies based on operating system, but may be something like this: +* Start GPG agent in order to unlock your GPG key - eval $(gpg-agent --daemon --no-grab --write-env-file $HOME/.gpg-agent-info) - export GPG_TTY=$(tty) - export GPG_AGENT_INFO + eval $(gpg-agent --daemon --no-grab --write-env-file $HOME/.gpg-agent-info) + export GPG_TTY=$(tty) + export GPG_AGENT_INFO #### Access to Apache Nexus repository @@ -181,135 +199,201 @@ You should verify that the issues listed automatically by JIRA are appropriate t Adjust any of the above properties to the improve clarity and presentation of the Release Notes. -### Verify that a Release Build Works +### Create a release branch in apache/beam repository -Pre-installation for python build -* Install pip - - ``` - curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py - python get-pip.py - ``` -* Install virtualenv +Attention: Only committer has permission to create release branch in apache/beam. - ``` - pip install --upgrade virtualenv - ``` -* Cython +Release candidates are built from a release branch. As a final step in preparation for the release, you should create the release branch, push it to the Apache code repository, and update version information on the original branch. + +There are 2 ways to cut a release branch: either running automation script(recommended), or running all commands manually. +#### Use [cut_release_branch.sh](https://github.com/apache/beam/blob/master/release/src/main/scripts/cut_release_branch.sh) to cut a release branch +* Usage ``` - sudo pip install cython - sudo apt-get install gcc - sudo apt-get install python-dev + # Cut a release branch + ./beam/release/src/main/scripts/cut_release_branch.sh + --release= ${RELEASE_VERSION} + --next_release=${NEXT_VERSION} + + # Show help page + ./beam/release/src/main/scripts/cut_release_branch.sh -h ``` -Make sure your ```time``` alias to ```/usr/bin/time```, if not: +* Tasks included + 1. Create release-${RELEASE_VERSION} branch locally. + 1. Change and commit dev versoin number in master branch: + + [BeamModulePlugin.groovy](https://github.com/apache/beam/blob/e8abafe360e126818fe80ae0f6075e71f0fc227d/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L209), + [gradle.properties](https://github.com/apache/beam/blob/e8abafe360e126818fe80ae0f6075e71f0fc227d/gradle.properties#L25), + [version.py](https://github.com/apache/beam/blob/e8abafe360e126818fe80ae0f6075e71f0fc227d/sdks/python/apache_beam/version.py#L21) + 1. Change and commit version number in release branch: + + [version.py](https://github.com/apache/beam/blob/release-2.6.0/sdks/python/apache_beam/version.py#L21), + [build.gradle](https://github.com/apache/beam/blob/release-2.6.0/runners/google-cloud-dataflow-java/build.gradle#L39) + +#### Run all steps manually +* Checkout working branch + + Check out the version of the codebase from which you start the release. For a new minor or major release, this may be `HEAD` of the `master` branch. To build a hotfix/incremental release, instead of the `master` branch, use the release tag of the release being patched. (Please make sure your cloned repository is up-to-date before starting.) -``` -sudo apt-get install time -alias time='/usr/bin/time' -``` + git checkout <master branch OR release tag> -Run gradle release build + **NOTE**: If you are doing an incremental/hotfix release (e.g. 2.5.1), please check out the previous release tag, rather than the master branch. -* Clean current workspace +* Set up environment variables - ``` - git clean -fdx - ./gradlew clean - ``` + Set up a few environment variables to simplify Maven commands that follow. (We use `bash` Unix syntax in this guide.) -* Unlock the secret key - ``` - gpg --output ~/doc.sig --sign ~/.bashrc - ``` + RELEASE=2.5.0 + NEXT_VERSION_IN_BASE_BRANCH=2.6.0 + BRANCH=release-${RELEASE} -* Run build command - ``` - ./gradlew build -PisRelease --no-parallel --scan --stacktrace - ``` - -### Update and Verify Javadoc + Version represents the release currently underway, while next version specifies the anticipated next version to be released from that branch. Normally, 1.2.0 is followed by 1.3.0, while 1.2.3 is followed by 1.2.4. -The build with `-PisRelease` creates the combined Javadoc for the release in `sdks/java/javadoc`. + **NOTE**: Only if you are doing an incremental/hotfix release (e.g. 2.5.1), please check out the previous release tag, before running the following instructions: -The file `sdks/java/javadoc/ant.xml` file contains a list of modules to include -in and exclude, plus a list of offline URLs that populate links from Beam's -Javadoc to the Javadoc for other modules that Beam depends on. + BASE_RELEASE=2.5.0 + RELEASE=2.5.1 + NEXT_VERSION_IN_BASE_BRANCH=2.6.0 + git checkout tags/${BASE_RELEASE} -* Confirm that new modules added since the last release have been added to the - inclusion list as appropriate. +* Create release branch locally + + git branch ${BRANCH} -* Confirm that the excluded package list is up to date. +* Update version files in the master branch. + + # Now change the version in existing gradle files, and Python files + sed -i -e "s/'${RELEASE}'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g" build_rules.gradle + sed -i -e "s/${RELEASE}/${NEXT_VERSION_IN_BASE_BRANCH}/g" gradle.properties + sed -i -e "s/${RELEASE}/${NEXT_VERSION_IN_BASE_BRANCH}/g" sdks/python/apache_beam/version.py + + # Save changes in master branch + git add gradle.properties build_rules.gradle sdks/python/apache_beam/version.py + git commit -m "Moving to ${NEXT_VERSION_IN_BASE_BRANCH}-SNAPSHOT on master branch." -* Verify the version numbers for offline links match the versions used by Beam. If - the version number has changed, download a new version of the corresponding - `<module>-docs/package-list` file. +* Check out the release branch. -### Create a release branch in apache/beam repository + git checkout ${BRANCH} + -Attention: Only committer has permission to create release branch in apache/beam. +* Update version files in release branch + + DEV=${RELEASE}.dev + sed -i -e "s/${DEV}/${RELEASE}/g" sdks/python/apache_beam/version.py + sed -i -e "s/'beam-master-.*'/'beam-${RELEASE}'/g" runners/google-cloud-dataflow-java/build.gradle -Release candidates are built from a release branch. As a final step in preparation for the release, you should create the release branch, push it to the Apache code repository, and update version information on the original branch. -Check out the version of the codebase from which you start the release. For a new minor or major release, this may be `HEAD` of the `master` branch. To build a hotfix/incremental release, instead of the `master` branch, use the release tag of the release being patched. (Please make sure your cloned repository is up-to-date before starting.) +### Start a snapshot build - git checkout <master branch OR release tag> +Start a build of [the nightly snapshot](https://builds.apache.org/view/A-D/view/Beam/job/beam_Release_NightlySnapshot/) against master branch. +Some processes, including our archetype tests, rely on having a live SNAPSHOT of the current version +from the `master` branch. Once the release branch is cut, these SNAPSHOT versions are no longer found, +so builds will be broken until a new snapshot is available. -**NOTE**: If you are doing an incremental/hotfix release (e.g. 2.5.1), please check out the previous release tag, rather than the master branch. +There are 2 ways to trigger a nightly build, either using automation script(recommended), or perform all operations manually. -Set up a few environment variables to simplify Maven commands that follow. (We use `bash` Unix syntax in this guide.) +#### Run [start_snapshot_build.sh](https://github.com/apache/beam/blob/master/release/src/main/scripts/start_snapshot_build.sh) to trigger build +* Usage + + ./beam/release/src/main/scripts/start_snapshot_build.sh - RELEASE=2.5.0 - NEXT_VERSION_IN_BASE_BRANCH=2.6.0 - BRANCH=release-${RELEASE} +* Tasks included + 1. Install [hub](https://github.com/github/hub) within your agreement. + 1. Touch an empty txt file and commit changes into ```${your remote beam repo}/snapshot_build``` + 1. Use hub to create a PR against apache:master, which triggers a Jenkins job to build snapshot. + +* Tasks you need to do manually + 1. Check whether the Jenkins job gets triggered. If not, please comment ```Run Gradle Publish``` into the generated PR. + 1. After verifying build succeeded, you need to close PR manually. + +#### Do all operations manually -Version represents the release currently underway, while next version specifies the anticipated next version to be released from that branch. Normally, 1.2.0 is followed by 1.3.0, while 1.2.3 is followed by 1.2.4. +* Find one PR against apache:master in beam. +* Comment ```Run Gradle Publish``` in this pull request to trigger build. +* Verify that build successes. -**NOTE**: Only if you are doing an incremental/hotfix release (e.g. 2.5.1), please check out the previous release tag, before running the following instructions: - BASE_RELEASE=2.5.0 - RELEASE=2.5.1 - NEXT_VERSION_IN_BASE_BRANCH=2.6.0 - git checkout tags/${BASE_RELEASE} +### Verify that a Release Build Works -Create a new branch, and update version files in the master branch. +There are 2 ways to perform this verification, either running automation script(recommended), or running all commands manually. - git branch ${BRANCH} +#### Run [verify_release_build.sh](https://github.com/apache/beam/blob/master/release/src/main/scripts/verify_release_build.sh) to verity a release build +* Usage + + ./beam/release/src/main/scripts/verify_release_build.sh - # Now change the version in existing gradle files, and Python files - sed -i -e "s/'${RELEASE}'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g" build_rules.gradle - sed -i -e "s/${RELEASE}/${NEXT_VERSION_IN_BASE_BRANCH}/g" gradle.properties - sed -i -e "s/${RELEASE}/${NEXT_VERSION_IN_BASE_BRANCH}/g" sdks/python/apache_beam/version.py +* Tasks included + 1. Install ```pip```, ```virtualenv```, ```cython``` and ```/usr/bin/time``` within your agreements. + 1. Run ```gradle release build``` against release branch. + +* Tasks you need to do manually + 1. Check the build result. + 1. If build failed, scan log will contain all failures. + 1. You should stabilize the release branch until release build succeeded. + +#### Run all commands manually +* Pre-installation for python build + 1. Install pip + + ``` + curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py + python get-pip.py + ``` + 1. Install virtualenv - # Save changes in master branch - git add gradle.properties build_rules.gradle sdks/python/apache_beam/version.py - git commit -m "Moving to ${NEXT_VERSION_IN_BASE_BRANCH}-SNAPSHOT on master branch." + ``` + pip install --upgrade virtualenv + ``` + 1. Cython -Check out the release branch. + ``` + sudo pip install cython + sudo apt-get install gcc + sudo apt-get install python-dev + ``` + 1. Make sure your ```time``` alias to ```/usr/bin/time```, if not: - git checkout ${BRANCH} - -The rest of this guide assumes that commands are run in the root of a repository on `${BRANCH}` with the above environment variables set. + ``` + sudo apt-get install time + alias time='/usr/bin/time' + ``` -### Update the Python SDK version +* Run gradle release build -Update [sdks/python/apache_beam/version.py](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/version.py) in both master branch and release branch. + 1. Clean current workspace -* In the release branch, update the Python SDK version to the release version(e.g. `2.5.0-dev` to `2.5.0`). + ``` + git clean -fdx + ./gradlew clean + ``` -### Update release specific configurations + 1. Unlock the secret key + ``` + gpg --output ~/doc.sig --sign ~/.bashrc + ``` -Update Java runner specific configurations in release branch: -* [beam/runners/google-cloud-dataflow-java/build.gradle](https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/build.gradle): change value of 'dataflow.container_version' to 'beam-release_version_number'(e.g, 'beam-master-20180601' to 'beam-2.5.0') + 1. Run build command + ``` + ./gradlew build -PisRelease --no-parallel --scan --stacktrace --continue + ``` + +### Update and Verify Javadoc -### Start a snapshot build +The build with `-PisRelease` creates the combined Javadoc for the release in `sdks/java/javadoc`. -Start a build of [the nightly snapshot](https://builds.apache.org/view/A-D/view/Beam/job/beam_Release_NightlySnapshot/) against master branch. -Some processes, including our archetype tests, rely on having a live SNAPSHOT of the current version -from the `master` branch. Once the release branch is cut, these SNAPSHOT versions are no longer found, -so builds will be broken until a new snapshot is available. +The file `sdks/java/javadoc/ant.xml` file contains a list of modules to include +in and exclude, plus a list of offline URLs that populate links from Beam's +Javadoc to the Javadoc for other modules that Beam depends on. + +* Confirm that new modules added since the last release have been added to the + inclusion list as appropriate. + +* Confirm that the excluded package list is up to date. + +* Verify the version numbers for offline links match the versions used by Beam. If + the version number has changed, download a new version of the corresponding + `<module>-docs/package-list` file. -Comment ```Run Gradle Publish``` in one pull request to trigger build. ### Checklist to proceed to the next step @@ -331,7 +415,31 @@ Comment ```Run Gradle Publish``` in one pull request to trigger build. The core of the release process is the build-vote-fix cycle. Each cycle produces one release candidate. The Release Manager repeats this cycle until the community approves one release candidate, which is then finalized. -### Build and stage Java artifacts with Gradle +For this step, we recommend you using automation script to create a RC, but you still can perform all steps manually if you want. + +### Run [build_release_candidate.sh](https://github.com/apache/beam/blob/master/release/src/main/scripts/build_release_candidate.sh) to create RC +* Usage + + ./beam/release/src/main/scripts/build_release_candidate.sh + +* Tasks included + 1. Run gradle release to create rc tag and push source release into github repo. + 1. Run gradle publish to push java artifacts into Maven staging repo. + 1. Stage source release into dist.apache.org dev [repo](https://dist.apache.org/repos/dist/dev/beam/). + 1. Stage,sign and hash python binaries into dist.apache.ord dev repo python dir + 1. Create a PR to update beam-site, changes includes: + * Copy python doc into beam-site + * Copy java doc into beam-site + * Update release version into [_config.yml](https://github.com/apache/beam-site/blob/asf-site/_config.yml). + +* Tasks you need to do manually + 1. Add new release into src/get-started/downloads.md + 1. Update last release download links in src/get-started/downloads.md + 1. Update the Pydoc link on this page to point to the new version (in src/documentation/sdks/pydoc/current.md. + +### Run all steps manually + +#### Build and stage Java artifacts with Gradle Set up a few environment variables to simplify the commands that follow. These identify the release candidate being built, and the branch where you will stage files. Start with `RC_NUM` equal to `1` and increment it for each candidate. @@ -356,7 +464,7 @@ Review all staged artifacts. They should contain all relevant parts for each mod Close the staging repository on Apache Nexus. When prompted for a description, enter “Apache Beam, version X, release candidate Y”. -### Stage source release on dist.apache.org +#### Stage source release on dist.apache.org Attention: Only committer has permissions to perform following steps. @@ -388,7 +496,7 @@ Copy the source release to the dev repository of `dist.apache.org`. 1. Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam). -### Stage python binaries on dist.apache.org +#### Stage python binaries on dist.apache.org Build python binaries in release branch in sdks/python dir. @@ -411,7 +519,7 @@ Staging binaries Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam). -### Build the Pydoc API reference +#### Build the Pydoc API reference Make sure you have ```tox``` installed: @@ -425,7 +533,7 @@ cd sdks/python && tox -e docs ``` By default the Pydoc is generated in `sdks/python/target/docs/_build`. Let `${PYDOC_ROOT}` be the absolute path to `_build`. -### Propose a pull request for website updates +#### Propose a pull request for website updates The final step of building the candidate is to propose a website pull request. @@ -444,7 +552,7 @@ Add the new Javadoc to [SDK API Reference page]({{ site.baseurl }}/documentation * Set up the necessary git commands to account for the new and deleted files from the javadoc. * Update the Javadoc link on this page to point to the new version (in `src/documentation/sdks/javadoc/current.md`). -#### Create Pydoc +##### Create Pydoc Add the new Pydoc to [SDK API Reference page]({{ site.baseurl }}/documentation/sdks/pydoc/) page, as follows: * Copy the generated Pydoc into the website repository: `cp -r ${PYDOC_ROOT} src/documentation/sdks/pydoc/${RELEASE}`. @@ -453,7 +561,7 @@ Add the new Pydoc to [SDK API Reference page]({{ site.baseurl }}/documentation/s Finally, propose a pull request with these changes. (Don’t merge before finalizing the release.) -### Checklist to proceed to the next step +#### Checklist to proceed to the next step 1. Maven artifacts deployed to the staging repository of [repository.apache.org](https://repository.apache.org/content/repositories/) 1. Source distribution deployed to the dev repository of [dist.apache.org](https://dist.apache.org/repos/dist/dev/beam/) @@ -528,6 +636,34 @@ If there are no issues, reply on the vote thread to close the voting. Then, tall ### Run validation tests All tests listed in this [spreadsheet](https://s.apache.org/beam-release-validation) +Since there are a bunch of tests, we recommend you running validatoins using automation script. In case of script failure, you can still run all of them manually. + +#### Run validations using [run_rc_validation.sh](https://github.com/apache/beam/blob/master/release/src/main/scripts/run_rc_validation.sh) +* Usage + + ./beam/release/src/main/scripts/run_rc_validation.sh + +* Tasks included + 1. Run Java quickstart with Direct Runner, Apex local runner, Flink local runner, Spark local runner and Dataflow runner. + 1. Run Java Mobile Games(UserScore, HourlyTeamScore, Leaderboard) with Dataflow runner. + 1. Create a PR against apache:master to trigger python validation job, including + * Python quickstart in batch and streaming mode with direct runner and Dataflow runner. + * Python Mobile Games(UserScore, HourlyTeamScore) with direct runner and Dataflow runner. + 1. Run Python Streaming MobileGames, includes + * Start a new terminal to run Java Pubsub injector. + * Start a new terminal to run python LeaderBoard with Direct Runner. + * Start a new terminal to run python LeaderBoard with Dataflow Runner. + * Start a new terminal to run python GameStats with Direct Runner. + * Start a new terminal to run python GameStats with Dataflow Runner. + +* Tasks you need to do manually + 1. Check whether validations succeed by following console output instructions. + 1. Terminate streaming jobs and java injector. + 1. Sign up [spreadsheet](https://s.apache.org/beam-release-validation). + 1. Vote in the release thread. + +#### Run validations manually + _Note_: -Prepourl and -Pver can be found in the RC vote email sent by Release Manager. * Java Quickstart Validation @@ -590,7 +726,7 @@ _Note_: -Prepourl and -Pver can be found in the RC vote email sent by Release Ma -Prepourl=https://repository.apache.org/content/repositories/orgapachebeam-${KEY} \ -Pver= ${RELEASE_VERSION}\ -PgcpProject=${YOUR_GCP_PROJECT} \ - -PgcsBucket=gs://dataflow-samples/game/gaming_data1.csv \ + -PgcsBucket=${YOUR_GCP_BUCKET} \ -PbqDataset=${YOUR_DATASET} -PpubsubTopic=${YOUR_PROJECT_PUBSUB_TOPIC} ``` * Python Quickstart(batch & streaming), MobileGame(UserScore, HourlyTeamScore) @@ -738,6 +874,7 @@ _Note_: -Prepourl and -Pver can be found in the RC vote email sent by Release Ma * Goto your BigQuery console and check whether your ${USER}_test has game_stats_teams and game_stats_sessions table. * bq head -n 10 ${USER}_test.game_stats_teams * bq head -n 10 ${USER}_test.game_stats_sessions + ### Checklist to proceed to the finalization step 1. Community votes to release the proposed candidate, with at least three approving PMC votes
