This is an automated email from the ASF dual-hosted git repository. jiayu pushed a commit to branch branch-0.1.0 in repository https://gitbox.apache.org/repos/asf/sedona-spatialbench.git
commit 3eee2d78e9228225472eab37c3ff86f4f90d7a41 Author: Jia Yu <[email protected]> AuthorDate: Tue Nov 25 22:34:26 2025 -0800 Update release script --- dev/release/README.md | 133 +++++++++++++++------------- dev/release/check-rat-report.py | 1 + dev/release/rat_exclude_files.txt | 1 + dev/release/run-rat.sh | 17 ++-- dev/release/sign-assets.sh | 149 -------------------------------- dev/release/upload-candidate.sh | 84 ------------------ dev/release/upload-release.sh | 55 ------------ dev/release/verify-release-candidate.sh | 26 +++++- 8 files changed, 109 insertions(+), 357 deletions(-) diff --git a/dev/release/README.md b/dev/release/README.md index 745bc93..7090f9b 100644 --- a/dev/release/README.md +++ b/dev/release/README.md @@ -21,18 +21,45 @@ ## Verifying a release candidate -Release candidates are verified using the script `verify-release-candidate.sh <version> <rc_num>`. -For example, to verify SpatialBench 0.1.0 RC1, run: +### Testing locally (before creating a release candidate) + +Before creating a release candidate, you should test your local checkout: ```shell # git clone https://github.com/apache/sedona-spatialbench.git && cd sedona-spatialbench # or # cd existing/sedona-spatialbench && git fetch upstream && git switch main && git pull upstream main +dev/release/verify-release-candidate.sh +``` + +This will run all verification tests on your local checkout without requiring any release artifacts. + +### Testing a local tarball + +If you have a tarball you want to test before uploading: + +```shell +dev/release/verify-release-candidate.sh /path/to/apache-sedona-spatialbench-0.1.0.tar.gz +``` + +### Verifying an official release candidate + +Once a release candidate has been uploaded to Apache dist, verify it using: + +```shell dev/release/verify-release-candidate.sh 0.1.0 1 ``` -Release verification requires a recent Rust toolchain. This toolchain can be installed -by following instructions from <https://rustup.rs/>. +This will download the release candidate from `https://dist.apache.org/repos/dist/dev/sedona/` and verify it. + +Release verification requires: +- A recent Rust toolchain (can be installed from <https://rustup.rs/>) +- Java (for Apache RAT license checking) +- Python (for RAT report filtering) + +The verification script will: +1. Run Apache RAT to check all files have proper license headers +2. Build and test all Rust crates in the workspace When verifying via Docker or on a smaller machine it may be necessary to limit the number of parallel jobs to avoid running out of memory: @@ -53,72 +80,71 @@ git branch -b branch-0.1.0 git push upstream -u branch-0.1.0:branch-0.1.0 ``` -Before creating a tag, download the tarball from the latest packaging run and -check it locally: - -```shell -dev/release/verify-release-candidate.sh path/to/tarball.tar.gz -``` - When the state of the `branch-x.x.x` branch is clean and checks are complete, the release candidate tag can be created: ```shell -git tag -a sedona-spatialbench-0.1.0-rc1 -m "Tag Apache SpatialBench 0.1.0-rc1" +git tag -a sedona-spatialbench-0.1.0-rc1 -m "Tag Apache Sedona SpatialBench 0.1.0-rc1" git push upstream sedona-spatialbench-0.1.0-rc1 ``` -This will trigger another packaging CI run that, if successful, will create a -pre-release at <https://github.com/apache/sedona-spatialbench/releases> with the release -artifacts uploaded from the CI run. +### Signing Commands -After the release has been created with the appropriate artifacts, the assets -need to be signed with signatures uploaded as release assets. Please create -dev/release/.env from dev/release/.env.example and set the GPG_KEY_ID variable. -The GPG_KEY_ID in dev/release/.env must have its public component listed in the -[Apache Sedona KEYS file](https://dist.apache.org/repos/dist/dev/sedona/KEYS). - -```shell -# sign-assets.sh <version> <rc_number> -dev/release/sign-assets.sh 0.1.0 1 -``` +Now the assets need to be signed with signatures. -After the assets are signed, they can be committed and uploaded to the -dev/sedona directory of the Apache distribution SVN. A helper script -is provided: +**GPG Signing:** ```shell -# upload-candidate.sh <version> <rc_number> -APACHE_USERNAME=your_apache_username dev/release/upload-candidate.sh 0.1.0 1 +# Sign a file (creates .asc file automatically) +gpg -ab apache-sedona-spatialbench-${SEDONA_VERSION}-src.tar.gz + +# Verify a signature +gpg --verify apache-sedona-spatialbench-${SEDONA_VERSION}-src.tar.gz.asc apache-sedona-spatialbench-${SEDONA_VERSION}-src.tar.gz ``` -## Vote +**SHA512 Checksum:** -An email must now be sent to `[email protected]` calling on developers to follow -the release verification instructions and vote appropriately on the source release. +```shell +# Generate SHA512 checksum +shasum -a 512 apache-sedona-spatialbench-${SEDONA_VERSION}-src.tar.gz > apache-sedona-spatialbench-${SEDONA_VERSION}-src.tar.gz.sha512 -## Publish +# Verify a checksum +shasum -a 512 --check apache-sedona-spatialbench-${SEDONA_VERSION}-src.tar.gz.sha512 +``` -### Upload/tag source release +**Upload to Apache SVN:** -After a successful release vote, the tarball needs to be uploaded to the official -Apache release repository. A helper script is provided: +After the assets are signed, they can be committed and uploaded to the +dev/sedona directory of the Apache distribution SVN: ```shell -# upload-release.sh <version> <rc_number> -APACHE_USERNAME=your_apache_username dev/release/upload-release.sh 0.1.0 1 +# Set version and RC number variables +SEDONA_VERSION="0.1.0" +RC_NUMBER="1" + +# Create the directory in SVN +svn mkdir -m "Adding folder" https://dist.apache.org/repos/dist/dev/sedona/sedona-spatialbench-${SEDONA_VERSION}-rc${RC_NUMBER} + +# Checkout the directory +svn co https://dist.apache.org/repos/dist/dev/sedona/sedona-spatialbench-${SEDONA_VERSION}-rc${RC_NUMBER} tmp + +# Copy files to the checked out directory +cp apache-sedona-spatialbench-${SEDONA_VERSION}-src.tar.gz tmp/ +cp apache-sedona-spatialbench-${SEDONA_VERSION}-src.tar.gz.asc tmp/ +cp apache-sedona-spatialbench-${SEDONA_VERSION}-src.tar.gz.sha512 tmp/ + +# Add and commit the files +cd tmp +svn add apache-sedona-spatialbench-${SEDONA_VERSION}-src.tar.gz* +svn ci -m "Apache SpatialBench ${SEDONA_VERSION} RC${RC_NUMBER}" +cd .. +rm -rf tmp ``` -An official GitHub tag must also be created: - -```shell -git tag -a sedona-spatialbench-0.1.0 -m "SpatialBench 0.1.0" sedona-spatialbench-0.1.0-rc1 -git push upstream sedona-spatialbench-0.1.0 -``` +## Vote -The prerelease located at <https://github.com/apache/sedona-spatialbench/releases/tag/sedona-spatialbench-0.1.0-rc1> -can now be edited to point to the official release tag and the GitHub release published -from the UI. +An email must now be sent to `[email protected]` calling on developers to follow +the release verification instructions and vote appropriately on the source release. ## Bump versions @@ -129,14 +155,3 @@ are currently all derived from `Cargo.toml`, which can be updated to: [workspace.package] version = "0.2.0" ``` - -Development versions and the changelog are derived from the presence of a development -tag on the main branch signifying where development of that version "started". After -the version bump PR merges, that commit should be tagged with the appropriate -development tag: - -```shell -git tag -a sedona-spatialbench-0.2.0.dev -m "tag dev 0.2.0" -git push upstream sedona-spatialbench-0.2.0.dev -``` - diff --git a/dev/release/check-rat-report.py b/dev/release/check-rat-report.py index b9fd85a..6a6da09 100644 --- a/dev/release/check-rat-report.py +++ b/dev/release/check-rat-report.py @@ -57,3 +57,4 @@ if not all_ok: print("OK") sys.exit(0) + diff --git a/dev/release/rat_exclude_files.txt b/dev/release/rat_exclude_files.txt index c3a17d7..5a4c249 100644 --- a/dev/release/rat_exclude_files.txt +++ b/dev/release/rat_exclude_files.txt @@ -9,3 +9,4 @@ spatialbench/data/sf-v1/*.tbl.gz spatialbench/data/sf-v1/*.parquet dev/release/rat_exclude_files.txt + diff --git a/dev/release/run-rat.sh b/dev/release/run-rat.sh index 78eba22..d5230c7 100755 --- a/dev/release/run-rat.sh +++ b/dev/release/run-rat.sh @@ -18,19 +18,21 @@ # under the License. RAT_VERSION=0.13 +RELEASE_DIR=$(cd "$(dirname "$BASH_SOURCE")"; pwd) +RAT_JAR="${RELEASE_DIR}/apache-rat-${RAT_VERSION}.jar" # download apache rat -if [ ! -f apache-rat-${RAT_VERSION}.jar ]; then - curl -s https://repo1.maven.org/maven2/org/apache/rat/apache-rat/${RAT_VERSION}/apache-rat-${RAT_VERSION}.jar > apache-rat-${RAT_VERSION}.jar +if [ ! -f "${RAT_JAR}" ]; then + curl -s https://repo1.maven.org/maven2/org/apache/rat/apache-rat/${RAT_VERSION}/apache-rat-${RAT_VERSION}.jar > "${RAT_JAR}" fi -RAT="java -jar apache-rat-${RAT_VERSION}.jar -x " - -RELEASE_DIR=$(cd "$(dirname "$BASH_SOURCE")"; pwd) +RAT="java -jar ${RAT_JAR} -x " # generate the rat report -$RAT $1 > rat.txt -python $RELEASE_DIR/check-rat-report.py $RELEASE_DIR/rat_exclude_files.txt rat.txt > filtered_rat.txt +# Run RAT from inside the target directory so it produces relative paths +cd "$1" +$RAT . > rat.txt +python3 $RELEASE_DIR/check-rat-report.py $RELEASE_DIR/rat_exclude_files.txt rat.txt > filtered_rat.txt cat filtered_rat.txt UNAPPROVED=`cat filtered_rat.txt | grep "NOT APPROVED" | wc -l` @@ -41,3 +43,4 @@ else exit 1 fi + diff --git a/dev/release/sign-assets.sh b/dev/release/sign-assets.sh deleted file mode 100755 index 18f56c8..0000000 --- a/dev/release/sign-assets.sh +++ /dev/null @@ -1,149 +0,0 @@ -#!/usr/bin/env bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -set -eu - -main() { - local -r source_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" - local -r source_top_dir="$( cd "${source_dir}/../../" && pwd )" - pushd "${source_top_dir}" - - if [ "$#" -ne 2 ]; then - echo "Usage: $0 <version> <rc-num>" - exit 1 - fi - - local -r version="$1" - local -r rc_number="$2" - local -r tag="sedona-spatialbench-${version}-rc${rc_number}" - local -r tarball="apache-sedona-spatialbench-${version}" - - : ${REPOSITORY:="apache/sedona-spatialbench"} - - if [[ ! -f "${source_dir}/.env" ]]; then - echo "You must create ${source_dir}/.env" - echo "You can use ${source_dir}/.env.example as a template" - fi - - source "${source_dir}/.env" - - header "Looking for GitHub Actions workflow on ${REPOSITORY}:${tag}" - local run_id="" - while [[ -z "${run_id}" ]] - do - echo "Waiting for run to start..." - run_id=$(gh run list \ - --repo "${REPOSITORY}" \ - --workflow=packaging \ - --json 'databaseId,event,headBranch,status' \ - --jq ".[] | select(.event == \"push\" and .headBranch == \"${tag}\") | .databaseId" | \ - head -n 1) - sleep 1 - done - - header "Found GitHub Actions workflow with ID: ${run_id}" - gh run watch --repo "${REPOSITORY}" --exit-status ${run_id} - gh run view --repo "${REPOSITORY}" "${run_id}" - - header "Downloading assets from release" - local -r download_dir="packages/${tag}" - mkdir -p "${download_dir}" - gh release download \ - "${tag}" \ - --repo "${REPOSITORY}" \ - --dir "${download_dir}" \ - --skip-existing - - header "Upload signatures for source" - upload_asset_signatures "${tag}" $(find "${download_dir}" -type f \( -name 'apache-sedona-spatialbench-*.tar.gz' \)) - - header "Upload signatures for docs" - upload_asset_signatures "${tag}" "${download_dir}/docs.tgz" - - popd -} - -header() { - echo "============================================================" - echo "${1}" - echo "============================================================" -} - -sign_asset() { - local -r asset="$1" - local -r sigfile="${asset}.asc" - - if [[ -f "${sigfile}" ]]; then - if env LANG=C gpg --verify "${sigfile}" "${asset}" >/dev/null 2>&1; then - echo "Valid signature at $(basename "${sigfile}"), skipping" - return - fi - rm "${sigfile}" - fi - - gpg \ - --armor \ - --detach-sign \ - --local-user "${GPG_KEY_ID}" \ - --output "${sigfile}" \ - "${asset}" - echo "Generated $(basename "${sigfile}")" -} - -sum_asset() { - local -r asset="$1" - local -r sumfile="${asset}.sha512" - - local -r digest=$(cd $(dirname "${asset}"); shasum --algorithm 512 $(basename "${asset}")) - if [[ -f "${sumfile}" ]]; then - if [[ "${digest}" = $(cat "${sumfile}") ]]; then - echo "Valid digest at $(basename "${sumfile}"), skipping" - return - fi - fi - - echo "${digest}" > "${sumfile}" - echo "Generated $(basename "${sumfile}")" -} - -upload_asset_signatures() { - local -r tag="${1}" - shift 1 - - local -r assets=("$@") - - for asset in "${assets[@]}"; do - sign_asset "${asset}" - sum_asset "${asset}" - done - - gh release upload \ - --repo "${REPOSITORY}" \ - "${tag}" \ - "${assets[@]/%/.asc}" \ - "${assets[@]/%/.sha512}" - - # Clean up - for asset in "${assets[@]}"; do - rm -f "${asset}" "${asset}.asc" "${asset}.sha512" - done -} - -main "$@" - diff --git a/dev/release/upload-candidate.sh b/dev/release/upload-candidate.sh deleted file mode 100755 index bffba60..0000000 --- a/dev/release/upload-candidate.sh +++ /dev/null @@ -1,84 +0,0 @@ -#!/usr/bin/env bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - - -set -eu - -main() { - local -r source_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" - local -r source_top_dir="$( cd "${source_dir}/../../" && pwd )" - - if [ "$#" -ne 2 ]; then - echo "Usage: $0 <version> <rc-num>" - exit 1 - fi - local -r version="$1" - local -r rc_number="$2" - local -r tag="sedona-spatialbench-${version}-rc${rc_number}" - local -r tarball="apache-sedona-spatialbench-${version}.tar.gz" - - : ${REPOSITORY:="apache/sedona-spatialbench"} - - if [[ ! -f "${source_dir}/.env" ]]; then - echo "You must create ${source_dir}/.env" - echo "You can use ${source_dir}/.env.example as a template" - fi - - source "${source_dir}/.env" - - header "Downloading assets from release" - local -r download_dir="packages/${tag}" - mkdir -p "${download_dir}" - gh release download \ - "${tag}" \ - --dir "${download_dir}" \ - --pattern "${tarball}*" \ - --repo "${REPOSITORY}" \ - --skip-existing - - echo "Uploading to dist.apache.org" - - # check out the sedona RC folder - svn co --depth=empty https://dist.apache.org/repos/dist/dev/sedona tmp - - # add the release candidate for the tag - mkdir -p "tmp/${tag}" - - # copy the rc tarball into the tmp dir - cp ${download_dir}/${tarball}* "tmp/${tag}" - - # commit to svn - svn add "tmp/${tag}" - svn ci --username=$APACHE_USERNAME -m "Apache SpatialBench ${version} RC${rc_number}" "tmp/${tag}" - - # clean up - rm -rf tmp - rm -rf "${download_dir}" - - echo "Uploaded at https://dist.apache.org/repos/dist/dev/sedona/${tag}"; -} - -header() { - echo "============================================================" - echo "${1}" - echo "============================================================" -} - -main "$@" - diff --git a/dev/release/upload-release.sh b/dev/release/upload-release.sh deleted file mode 100755 index 71bf530..0000000 --- a/dev/release/upload-release.sh +++ /dev/null @@ -1,55 +0,0 @@ -#!/usr/bin/env bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -set -eu - -main() { - local -r source_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" - - if [ "$#" -ne 2 ]; then - echo "Usage: $0 <version> <rc-num>" - exit 1 - fi - local -r version="$1" - local -r rc_number="$2" - local -r tag="sedona-spatialbench-${version}-rc${rc_number}" - - if [[ ! -f "${source_dir}/.env" ]]; then - echo "You must create ${source_dir}/.env" - echo "You can use ${source_dir}/.env.example as a template" - fi - - source "${source_dir}/.env" - - rc_id="sedona-spatialbench-${version}-rc${rc_number}" - release_id="apache-sedona-spatialbench-${version}" - echo "Moving dev/ to release/" - svn \ - mv \ - --username=$APACHE_USERNAME \ - -m "Apache SpatialBench ${version}" \ - https://dist.apache.org/repos/dist/dev/sedona/${rc_id} \ - https://dist.apache.org/repos/dist/release/sedona/${release_id} - - echo "Success! The release is available here:" - echo " https://dist.apache.org/repos/dist/release/sedona/${release_id}"; -} - -main "$@" - diff --git a/dev/release/verify-release-candidate.sh b/dev/release/verify-release-candidate.sh index 4a55d9b..3192014 100755 --- a/dev/release/verify-release-candidate.sh +++ b/dev/release/verify-release-candidate.sh @@ -28,7 +28,7 @@ fi check_dependencies() { local missing_deps=0 - local required_deps=("curl" "git" "gpg" "cargo") + local required_deps=("curl" "git" "gpg" "cargo" "java" "python") for dep in "${required_deps[@]}"; do if ! command -v $dep &> /dev/null; then echo "Error: $dep is not installed or not in PATH" @@ -102,12 +102,19 @@ show_info() { SPATIALBENCH_DIST_URL='https://dist.apache.org/repos/dist/dev/sedona' download_dist_file() { - curl \ + local url="${SPATIALBENCH_DIST_URL}/$1" + if ! curl \ --silent \ --show-error \ --fail \ --location \ - --remote-name $SPATIALBENCH_DIST_URL/$1 + --remote-name "$url"; then + echo "Error: Failed to download $url" + echo "This usually means the release candidate has not been uploaded yet." + echo "To test locally, run: $0" + echo "Or to test a local tarball: $0 /path/to/tarball.tar.gz" + exit 1 + fi } download_rc_file() { @@ -196,6 +203,14 @@ test_rust() { popd } +test_rat() { + show_header "Running Apache RAT license check" + + # Use SOURCE_DIR which was computed at script startup (before any directory changes) + # Run RAT check using the run-rat.sh script from the repo + bash "${SOURCE_DIR}/run-rat.sh" "${SPATIALBENCH_SOURCE_DIR}" +} + ensure_source_directory() { show_header "Ensuring source directory" @@ -234,6 +249,10 @@ ensure_source_directory() { test_source_distribution() { pushd $SPATIALBENCH_SOURCE_DIR + if [ ${TEST_RAT} -gt 0 ]; then + test_rat + fi + if [ ${TEST_RUST} -gt 0 ]; then test_rust fi @@ -247,6 +266,7 @@ test_source_distribution() { : ${TEST_DEFAULT:=1} : ${TEST_SOURCE:=${TEST_DEFAULT}} +: ${TEST_RAT:=${TEST_SOURCE}} : ${TEST_RUST:=${TEST_SOURCE}} TEST_SUCCESS=no
