pingtimeout commented on code in PR #2156: URL: https://github.com/apache/polaris/pull/2156#discussion_r2254317979
########## releasey/06-build-and-stage-docker-images.sh: ########## @@ -0,0 +1,137 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +# +# Build and Stage Docker Images Script +# +# Builds and publishes multi-platform Docker images to DockerHub for release candidates. +# This script automates the "Build and staging Docker images" step from the release guide. +# + +set -euo pipefail + +releases_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +libs_dir="${releases_dir}/libs" + +source "${libs_dir}/_log.sh" +source "${libs_dir}/_constants.sh" +source "${libs_dir}/_exec.sh" +source "${libs_dir}/_files.sh" + +function usage() { + cat << EOF +$(basename "$0") [--help | -h] + + Builds and publishes multi-platform Docker images to DockerHub for release candidates. + The version is automatically determined from the current git tag. I.e. this script must + be run from a release tag. + + Prerequisites: + - Docker with buildx support must be installed and configured + - DockerHub credentials must be configured (docker login) + - Must be run from a release tag (apache-polaris-x.y.z-incubating-rcN) + + Options: + -h --help + Print usage information. + + Examples: + $(basename "$0") + +EOF +} + +ensure_cwd_is_project_root + +while [[ $# -gt 0 ]]; do + case $1 in + --help|-h) + usage + exit 0 + ;; + *) + print_error "Unknown option/argument $1" + usage >&2 + exit 1 + ;; + esac +done + +# Determine version from current git tag +print_info "Determining version from current git tag..." + +if ! git_tag=$(git describe --tags --exact-match HEAD 2>/dev/null); then + print_error "Current HEAD is not on a release tag. Please checkout a release tag first." + print_error "Use: git checkout apache-polaris-x.y.z-incubating-rcN" + exit 1 +fi +print_info "Found git tag: ${git_tag}" + +# Extract version components from git tag in one regex match +git_tag_regex="^apache-polaris-([0-9]+)\.([0-9]+)\.([0-9]+)-incubating-rc([0-9]+)$" +if [[ ! ${git_tag} =~ ${git_tag_regex} ]]; then + print_error "Invalid git tag format: ${git_tag}" + print_error "Expected format: apache-polaris-x.y.z-incubating-rcN" + exit 1 +fi + +# Extract version components from regex match +major="${BASH_REMATCH[1]}" +minor="${BASH_REMATCH[2]}" +patch="${BASH_REMATCH[3]}" +rc_number="${BASH_REMATCH[4]}" +version="${major}.${minor}.${patch}-incubating" Review Comment: My point, exactly. This would mean that everywhere in the scripts, instead of having: ``` version="${major}.${minor}.${patch}-incubating" ``` We would have ``` version="${major}.${minor}.${patch}-${INCUBATING_SUFFIX}" ``` The `INCUBATING_SUFFIX` variable would be present all over the place, even though it will serve no purpose at all after graduation. Hence, no readability benefit whatsoever. ########## releasey/README.md: ########## @@ -0,0 +1,131 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Polaris Release Automation Scripts + +This directory contains scripts to automate the Apache Polaris release process, following the [official release guide](https://github.com/apache/polaris/blob/main/docs/release-guide.md). + +## Release Scripts + +These scripts are used to perform actual releases: + +### 1. Prerequisites Verification +```bash +./01-prerequisites.sh +``` +Verifies that your environment is ready for releases: +- GPG setup and key configuration +- Maven/Gradle credentials +- Git remote setup for Apache repository +- GitHub token for test status verification + +### 2. Create Release Branch +```bash +./02-create-release-branch.sh --version 1.0.0-incubating [--commit HEAD] [--recreate] +``` +Creates a release branch and sets the target release version: +- Creates `release/x.y.z` branch from specified commit +- Updates `version.txt` with the release version +- Updates `CHANGELOG.md` using Gradle task +- Pushes changes to Apache remote + +### 3. Create Release Candidate Tag +```bash +./03-create-release-candidate-tag.sh --version 1.0.0-incubating-rc1 +``` +Creates a release candidate tag for a release candidate: +- Creates and pushes `apache-polaris-x.y.z-incubating-rcN` tag +- Validates RC sequence (RC2+ requires previous RC to exist) +- Checks out the created tag + +### 4. Build and Verify Tests +```bash +./04-build-and-test.sh +``` +Builds Polaris and ensures all integration and regression tests have been run: +- Performs clean build with Gradle +- Verifies that all GitHub checks have passed for current commit +- Ensures all CI workflows are successful before release + +### 5. Build and Stage Distributions +```bash +./05-build-and-stage-distributions.sh +``` +Builds and stages release artifacts: +- Must be run from a release candidate tag +- Builds source and binary distributions +- Stages artifacts to Apache dist dev repository +- Publishes Maven artifacts to Apache staging repository + +### 6. Build and Stage Docker Images +```bash +./06-build-and-stage-docker-images.sh +``` +Builds and publishes multi-platform Docker images: +- Must be run from a release candidate tag +- Builds polaris-server and polaris-admin Docker images +- Publishes images to DockerHub with RC tag +- Supports multi-platform builds (linux/amd64, linux/arm64) + +## Test Scripts + +These scripts are used to test the release automation without making actual changes: + +### Running Tests +```bash +# Test individual scripts +./test/test-02-create-release-branch.sh +./test/test-03-create-release-candidate-tag.sh +./test/test-04-build-and-test.sh +./test/test-05-build-and-stage-distributions.sh +./test/test-06-build-and-stage-docker-images.sh +``` + +All test scripts run in dry-run mode and verify the exact commands that would be executed. + +## Environment Variables +The releases script initialize environment variables if not already set. +This means that it is possible to override the default values by setting the environment variables before running the scripts. +Example: + +- `DRY_RUN=0` - Disable dry-run mode (default: enabled) +- `APACHE_REMOTE_NAME=my-apache-remote` - Name of git remote where release branches and tags are pushed (default: "apache") +- `KEYSERVER` - GPG keyserver URL (default: "hkps://keyserver.ubuntu.com") +- ... + +See the content of `libs/_constants.sh` for the list of all environment variables and their default values. + +## Prerequisites Review Comment: Fixing... ########## releasey/README.md: ########## @@ -0,0 +1,131 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Polaris Release Automation Scripts + +This directory contains scripts to automate the Apache Polaris release process, following the [official release guide](https://github.com/apache/polaris/blob/main/docs/release-guide.md). + +## Release Scripts + +These scripts are used to perform actual releases: + +### 1. Prerequisites Verification +```bash +./01-prerequisites.sh +``` +Verifies that your environment is ready for releases: +- GPG setup and key configuration +- Maven/Gradle credentials +- Git remote setup for Apache repository +- GitHub token for test status verification + +### 2. Create Release Branch +```bash +./02-create-release-branch.sh --version 1.0.0-incubating [--commit HEAD] [--recreate] Review Comment: Good point, fixing... ########## releasey/04-build-and-test.sh: ########## @@ -0,0 +1,90 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +# +# Build and Test Script +# +# Builds Polaris completely and ensures that all integration and regression tests +# have been run prior to releasing by verifying the status of all GitHub Actions +# workflows for the current commit. +# + +set -euo pipefail + +releasey_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +LIBS_DIR="${releasey_dir}/libs" + +source "${LIBS_DIR}/_log.sh" +source "${LIBS_DIR}/_constants.sh" +source "${LIBS_DIR}/_exec.sh" +source "${LIBS_DIR}/_version.sh" +source "${LIBS_DIR}/_github.sh" + +function usage() { + cat << EOF +$(basename "$0") [--help | -h] + + Builds Polaris completely and ensures that all integration and regression tests + have been run prior to releasing by verifying GitHub CI status. + + Options: + -h --help + Print usage information. + + Examples: + $(basename "$0") + +EOF +} + +while [[ $# -gt 0 ]]; do + case $1 in + --help|-h) + usage + exit 0 + ;; + *) + print_error "Unknown option/argument $1" + usage >&2 + exit 1 + ;; + esac +done + +print_info "Starting build and test process..." +echo + +# Clean and build Polaris +print_info "Building Polaris..." +exec_process cd "${releasey_dir}/.." +exec_process ./gradlew clean build Review Comment: I don't think this command runs the regression tests, though. Does it? The script also contains a verification step against Github, see L82 below. The verification ensures that there is no failed Github check, which gives us the guarantee that regression tests passed. Does this make sense? ########## releasey/05-build-and-stage-distributions.sh: ########## @@ -0,0 +1,155 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +# +# Build and Stage Distributions Script +# +# Builds source and binary distributions and stages them to the Apache dist dev repository. +# This script automates the "Build and stage the distributions" step from the release guide. +# + +set -euo pipefail + +releasey_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +LIBS_DIR="${releasey_dir}/libs" + +source "${LIBS_DIR}/_log.sh" +source "${LIBS_DIR}/_constants.sh" +source "${LIBS_DIR}/_exec.sh" +source "${LIBS_DIR}/_version.sh" +source "${LIBS_DIR}/_gpg.sh" + +function usage() { + cat << EOF +$(basename "$0") [--help | -h] + + Builds source and binary distributions and stages them to the Apache dist dev repository. + The version is automatically determined from the current git tag. I.e. this script must + be run from a release candidate tag. + + Options: + -h --help + Print usage information. + + Examples: + $(basename "$0") + +EOF +} + +while [[ $# -gt 0 ]]; do + case $1 in + --help|-h) + usage + exit 0 + ;; + *) + print_error "Unknown option/argument $1" + usage >&2 + exit 1 + ;; + esac +done + +# Determine version from current git tag +print_info "Determining version from current git tag..." + +if ! git_tag=$(git describe --tags --exact-match HEAD 2>/dev/null); then + print_error "Current HEAD is not on a release candidate tag. Please checkout a release candidate tag first." + print_error "Use: git checkout apache-polaris-x.y.z-incubating-rcN" + exit 1 +fi +print_info "Found git tag: ${git_tag}" + +# Validate git tag format and extract version components +if ! validate_and_extract_git_tag_version "${git_tag}"; then + print_error "Invalid git tag format: ${git_tag}" + print_error "Expected format: apache-polaris-x.y.z-incubating-rcN" + exit 1 +fi + +polaris_version="${major}.${minor}.${patch}-incubating" + +print_info "Starting build and stage distributions process..." +print_info "Version: ${polaris_version}" +print_info "RC number: ${rc_number}" +echo + +# Build distributions +print_info "Building source and binary distributions..." +exec_process cd "${releasey_dir}/.." +exec_process ./gradlew build sourceTarball -Prelease -PuseGpgAgent -x test -x intTest + +# Create Helm package +print_info "Creating Helm package..." +exec_process cd "${releasey_dir}/../helm" +exec_process helm package polaris +exec_process helm gpg sign polaris-${polaris_version}.tgz +calculate_sha512 polaris-${polaris_version}.tgz polaris-${polaris_version}.tgz.sha512 +exec_process gpg --armor --output polaris-${polaris_version}.tgz.asc --detach-sig polaris-${polaris_version}.tgz +calculate_sha512 polaris-${polaris_version}.tgz.prov polaris-${polaris_version}.tgz.prov.sha512 +exec_process gpg --armor --output polaris-${polaris_version}.tgz.prov.asc --detach-sig polaris-${polaris_version}.tgz.prov + +# Stage to Apache dist dev repository +print_info "Staging artifacts to Apache dist dev repository..." + +dist_dev_dir=${releasey_dir}/polaris-dist-dev +print_info "Checking out ${APACHE_DIST_URL}${APACHE_DIST_PATH} to ${dist_dev_dir}..." +exec_process svn co "${APACHE_DIST_URL}${APACHE_DIST_PATH}" "${dist_dev_dir}" + +print_info "Copying files to destination directory..." +version_dir="${dist_dev_dir}/${polaris_version}" +helm_chart_version_dir="${dist_dev_dir}/helm-chart/${polaris_version}" +exec_process mkdir -p "${version_dir}" +exec_process mkdir -p "${helm_chart_version_dir}" +exec_process cp "build/distribution/*" "${version_dir}/" +exec_process cp "runtime/distribution/build/distributions/*" "${version_dir}/" + +print_info "Copying Helm package files..." +exec_process cp helm/polaris-${polaris_version}.tgz* "${helm_chart_version_dir}/" + +print_info "Adding files to SVN..." +exec_process svn add "${version_dir}" +exec_process svn add "${helm_chart_version_dir}" + +print_info "Committing changes..." +exec_process svn commit -m "Stage Apache Polaris ${polaris_version} RC${rc_number}" + +print_info "Updating Helm index..." +exec_process cd "${dist_dev_dir}/helm-chart" +exec_process helm repo index . +exec_process cd "${releasey_dir}/.." Review Comment: Good point, let me try a few things locally and update the script. ########## releasey/03-create-release-candidate-tag.sh: ########## @@ -0,0 +1,142 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +# +# Create Release Candidate Tag Script +# +# Automates the "Create release candidate tag" section of the release guide. +# + +set -euo pipefail + +releasey_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +LIBS_DIR="${releasey_dir}/libs" + +source "${LIBS_DIR}/_log.sh" +source "${LIBS_DIR}/_constants.sh" +source "${LIBS_DIR}/_exec.sh" +source "${LIBS_DIR}/_version.sh" + +function usage() { + cat << EOF +$(basename "$0") --version VERSION [--help | -h] + + Creates a release candidate tag for a release candidate. + + Options: + --version VERSION + The release version in format x.y.z-incubating-rcN where N is the RC number + -h --help + Print usage information. + + Examples: + $(basename "$0") --version 1.0.0-incubating-rc1 + $(basename "$0") --version 0.1.0-incubating-rc2 + +EOF +} + +version="" + +while [[ $# -gt 0 ]]; do + case $1 in + --version) + if [[ $# -lt 2 ]]; then + print_error "Missing argument for --version" + usage >&2 + exit 1 + fi + version="$2" + shift 2 + ;; + --help|-h) + usage + exit 0 + ;; + *) + print_error "Unknown option/argument $1" + usage >&2 + exit 1 + ;; + esac +done + +# Ensure the version is provided +if [[ -z ${version} ]]; then + print_error "Missing version" + usage >&2 + exit 1 +fi + +# Validate version format: x.y.z-incubating-rcN +# TODO: Remove incubating when we are a TLP +if ! validate_and_extract_rc_version "${version}"; then + print_error "Invalid version format. Expected: x.y.z-incubating-rcN where N>0, got: ${version}" + usage >&2 + exit 1 +fi + +# Define polaris_version from extracted components +polaris_version="${major}.${minor}.${patch}-incubating" + +print_info "Starting release candidate tag creation..." +print_info "Version: ${version}" +print_info "Polaris version: ${polaris_version}" +print_info "RC number: ${rc_number}" +echo + +# If we're creating RC2 or later, verify previous RC tag exists +if [[ ${rc_number} -gt 1 ]]; then + previous_rc=$((rc_number - 1)) + previous_tag="apache-polaris-${version%-rc*}-rc${previous_rc}" + + print_info "Checking for previous RC tag: ${previous_tag}" + if ! git tag -l "${previous_tag}" | grep -q "^${previous_tag}$"; then + print_error "Previous RC tag ${previous_tag} does not exist. Cannot create RC${rc_number} without RC${previous_rc}." + exit 1 + fi + print_info "Previous RC tag ${previous_tag} found" +fi Review Comment: I don't think this should be a concern. My rationale is that no RC tag should ever be deleted. I don't think it is allowed per ASF principles, as it means a release candidate, for which a vote was called, could disappear. Wdyt? ########## releasey/02-create-release-branch.sh: ########## @@ -0,0 +1,201 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +# +# Create Release Branch Script +# +# Automates the "Create release branch" section of the release guide. +# Creates a new release branch and sets the target release version. +# + +set -euo pipefail + +releasey_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +LIBS_DIR="${releasey_dir}/libs" + +source "${LIBS_DIR}/_log.sh" +source "${LIBS_DIR}/_constants.sh" +source "${LIBS_DIR}/_exec.sh" +source "${LIBS_DIR}/_version.sh" + +function usage() { + cat << EOF +$(basename "$0") --version VERSION [--commit GIT_COMMIT] [--recreate] [--help | -h] + + Creates a release branch and sets the target release version. + + Behavior: + - RC1: Creates the release branch and sets up the version + - RC2+: Exits successfully if release branch exists, errors if it doesn't + + Options: + --version VERSION + The release version in format x.y.z-incubating-rcN where N is the RC number + --commit GIT_COMMIT + The Git commit SHA to create the release branch from. Defaults to current HEAD. + --recreate + Drop the release branch if it already exists and recreate it to the new SHA. + -h --help + Print usage information. + + Examples: + $(basename "$0") --version 1.0.0-incubating-rc1 --commit HEAD + $(basename "$0") --version 0.1.0-incubating-rc1 --commit abc123def456 + $(basename "$0") --version 1.0.0-incubating-rc1 --commit HEAD --recreate + $(basename "$0") --version 1.0.0-incubating-rc2 # Will exit successfully if branch exists + +EOF +} + +version="" +commit="HEAD" +recreate=false + +while [[ $# -gt 0 ]]; do + case $1 in + --version) + if [[ $# -lt 2 ]]; then + print_error "Missing argument for --version" + usage >&2 + exit 1 + fi + version="$2" + shift 2 + ;; + --commit) + if [[ $# -lt 2 ]]; then + print_error "Missing argument for --commit" + usage >&2 + exit 1 + fi + commit="$2" + shift 2 + ;; + --recreate) + recreate=true + shift + ;; + --help|-h) + usage + exit 0 + ;; + *) + print_error "Unknown option/argument $1" + usage >&2 + exit 1 + ;; + esac +done + +# Ensure the version is provided +if [[ -z ${version} ]]; then + print_error "Missing version" + usage >&2 + exit 1 +fi + +# Validate that the commit exists +if ! git rev-parse --verify "${commit}" >/dev/null 2>&1; then + print_error "Invalid Git commit: ${commit}" + usage >&2 + exit 1 +fi + +# Validate version format: x.y.z-incubating-rcN +# TODO: Remove incubating when we are a TLP +if ! validate_and_extract_rc_version "${version}"; then + print_error "Invalid version format. Expected: x.y.z-incubating-rcN where N>0, got: ${version}" + usage >&2 + exit 1 +fi + +# Define polaris_version from extracted components +polaris_version="${major}.${minor}.${patch}-incubating" +release_branch="release/${polaris_version}" + +# Handle RC > 1 scenarios +if [[ ${rc_number} -gt 1 ]]; then + # Check if release branch already exists + if git show-ref --verify --quiet "refs/heads/${release_branch}"; then + print_info "RC${rc_number} detected and release branch ${release_branch} already exists." + print_info "This script only creates release branches for RC1. Nothing to do." + exit 0 + else + print_error "RC${rc_number} detected but release branch ${release_branch} does not exist." + exit 1 + fi +fi + +print_info "Starting release branch creation..." +print_info "Version: ${version}" +print_info "Polaris version: ${polaris_version}" +print_info "From commit: ${commit}" +echo + +# Check if release branch already exists +if git show-ref --verify --quiet "refs/heads/${release_branch}"; then + if [[ "${recreate}" == "true" ]]; then + print_info "Release branch ${release_branch} already exists, deleting it..." + exec_process git branch -D "${release_branch}" + if git show-ref --verify --quiet "refs/remotes/${APACHE_REMOTE_NAME}/${release_branch}"; then + print_info "Deleting remote release branch from ${APACHE_REMOTE_NAME}..." + exec_process git push "${APACHE_REMOTE_NAME}" --delete "${release_branch}" + fi + else + print_error "Release branch ${release_branch} already exists. Use --recreate to delete and recreate it." + exit 1 + fi +fi + +print_info "Checking out commit: ${commit}" +exec_process git checkout "${commit}" + +print_info "Creating release branch: ${release_branch}" +exec_process git branch "${release_branch}" + +print_info "Pushing release branch to ${APACHE_REMOTE_NAME}" +exec_process git push "${APACHE_REMOTE_NAME}" "${release_branch}" --set-upstream + +print_info "Checking out release branch" +exec_process git checkout "${release_branch}" + +print_info "Setting version to ${polaris_version} in version.txt" +update_version "${polaris_version}" + +print_info "Committing and pushing version change" +exec_process git add "$VERSION_FILE" "$HELM_CHART_YAML_FILE" "$HELM_README_FILE" "$HELM_VALUES_FILE" +exec_process git commit -m "[chore] Bump version to ${polaris_version} for release" +exec_process git push + +print_info "Updating CHANGELOG.md" +exec_process cd "${releasey_dir}/.." +exec_process ./gradlew patchChangelog Review Comment: That's a good question. I am not 100% sure about the answer, to be honest. The [release guide](https://polaris.apache.org/community/release-guide/) says that the changelog should be manually updated: > Manually add an -rcN suffix to the previously generated versioned CHANGELOG section. > Rerun the patchChangelog command > Manually remove RC sections from the CHANGELOG > Submit a PR to propagate CHANGELOG updates from the release branch to main. And IIRC there are still some discussions on changelog updates, whether those updates should be merged to main, etc... This is the reason why the script exits for RC>1. My goal with this PR is to have a 1:1 automation of what the release guide says, so that we can start having some semi-automated (albeit imperfect) release process for the August release. But I suspect we will iterate on this as we learn more about our practices. ########## releasey/05-build-and-stage-distributions.sh: ########## @@ -0,0 +1,155 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +# +# Build and Stage Distributions Script +# +# Builds source and binary distributions and stages them to the Apache dist dev repository. +# This script automates the "Build and stage the distributions" step from the release guide. +# + +set -euo pipefail + +releasey_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +LIBS_DIR="${releasey_dir}/libs" + +source "${LIBS_DIR}/_log.sh" +source "${LIBS_DIR}/_constants.sh" +source "${LIBS_DIR}/_exec.sh" +source "${LIBS_DIR}/_version.sh" +source "${LIBS_DIR}/_gpg.sh" + +function usage() { + cat << EOF +$(basename "$0") [--help | -h] + + Builds source and binary distributions and stages them to the Apache dist dev repository. + The version is automatically determined from the current git tag. I.e. this script must + be run from a release candidate tag. + + Options: + -h --help + Print usage information. + + Examples: + $(basename "$0") + +EOF +} + +while [[ $# -gt 0 ]]; do + case $1 in + --help|-h) + usage + exit 0 + ;; + *) + print_error "Unknown option/argument $1" + usage >&2 + exit 1 + ;; + esac +done + +# Determine version from current git tag +print_info "Determining version from current git tag..." + +if ! git_tag=$(git describe --tags --exact-match HEAD 2>/dev/null); then + print_error "Current HEAD is not on a release candidate tag. Please checkout a release candidate tag first." + print_error "Use: git checkout apache-polaris-x.y.z-incubating-rcN" + exit 1 +fi +print_info "Found git tag: ${git_tag}" + +# Validate git tag format and extract version components +if ! validate_and_extract_git_tag_version "${git_tag}"; then + print_error "Invalid git tag format: ${git_tag}" + print_error "Expected format: apache-polaris-x.y.z-incubating-rcN" + exit 1 +fi + +polaris_version="${major}.${minor}.${patch}-incubating" + +print_info "Starting build and stage distributions process..." +print_info "Version: ${polaris_version}" +print_info "RC number: ${rc_number}" +echo + +# Build distributions +print_info "Building source and binary distributions..." +exec_process cd "${releasey_dir}/.." +exec_process ./gradlew build sourceTarball -Prelease -PuseGpgAgent -x test -x intTest + +# Create Helm package +print_info "Creating Helm package..." +exec_process cd "${releasey_dir}/../helm" +exec_process helm package polaris +exec_process helm gpg sign polaris-${polaris_version}.tgz +calculate_sha512 polaris-${polaris_version}.tgz polaris-${polaris_version}.tgz.sha512 +exec_process gpg --armor --output polaris-${polaris_version}.tgz.asc --detach-sig polaris-${polaris_version}.tgz +calculate_sha512 polaris-${polaris_version}.tgz.prov polaris-${polaris_version}.tgz.prov.sha512 +exec_process gpg --armor --output polaris-${polaris_version}.tgz.prov.asc --detach-sig polaris-${polaris_version}.tgz.prov + +# Stage to Apache dist dev repository +print_info "Staging artifacts to Apache dist dev repository..." + +dist_dev_dir=${releasey_dir}/polaris-dist-dev +print_info "Checking out ${APACHE_DIST_URL}${APACHE_DIST_PATH} to ${dist_dev_dir}..." +exec_process svn co "${APACHE_DIST_URL}${APACHE_DIST_PATH}" "${dist_dev_dir}" + +print_info "Copying files to destination directory..." +version_dir="${dist_dev_dir}/${polaris_version}" +helm_chart_version_dir="${dist_dev_dir}/helm-chart/${polaris_version}" +exec_process mkdir -p "${version_dir}" +exec_process mkdir -p "${helm_chart_version_dir}" +exec_process cp "build/distribution/*" "${version_dir}/" +exec_process cp "runtime/distribution/build/distributions/*" "${version_dir}/" + +print_info "Copying Helm package files..." +exec_process cp helm/polaris-${polaris_version}.tgz* "${helm_chart_version_dir}/" + +print_info "Adding files to SVN..." +exec_process svn add "${version_dir}" +exec_process svn add "${helm_chart_version_dir}" + +print_info "Committing changes..." +exec_process svn commit -m "Stage Apache Polaris ${polaris_version} RC${rc_number}" + +print_info "Updating Helm index..." +exec_process cd "${dist_dev_dir}/helm-chart" +exec_process helm repo index . +exec_process cd "${releasey_dir}/.." + +# Publish Maven artifacts +print_info "Publishing Maven artifacts to Apache staging repository..." +exec_process ./gradlew publishToApache -Prelease -PuseGpgAgent -Dorg.gradle.parallel=false + +echo +print_success "🎉 Distributions built and staged successfully!" +echo +print_info "Staged artifacts:" +print_info "- Source and binary distributions: ${version_dir}" +print_info "- Helm charts: ${helm_chart_version_dir}" +print_info "- Maven artifacts: Published to Apache staging repository" +echo +print_info "Next steps:" +print_info "1. Close the staging repository in Nexus" Review Comment: There should be a way to automate that, indeed. The grand vision is to have a fully automated release process that runs in Github Workflows. However there are some things that needs discussed before this can happen. Things like : * Should we trigger when a Git tag is created or when an empty Github Release with a specific tag pattern is created, ... * As you pointed out in another comment, should we (can we?) automatically update the Changelog? * Should we implement fixes on branches of the supported release(s) first and then merge those branches up to main (in which case the changelog updates are easy)? Or do we want to cherry-pick commits to add certain fixes to release branches (in which case it requires manual updates)? * Does the ASF supports automatically sending announcement/vote/cancel e-mails? In which case this could be embedded as part of the release automation * Etc... As you can see, we are not quite there yet. I suggest we start with the current scripts, which provide a semi-automated model. And in parallel, we can start discussion to achieve the big goal. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@polaris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
