kou commented on code in PR #174: URL: https://github.com/apache/arrow-adbc/pull/174#discussion_r1037545137
########## dev/release/02-source.sh: ########## @@ -0,0 +1,110 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -eu + +: ${SOURCE_UPLOAD:="0"} + +main() { + local -r source_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" + local -r source_top_dir="$( cd "${source_dir}/../../" && pwd )" + + if type shasum >/dev/null 2>&1; then + local -r sha256_generate="shasum -a 256" + local -r sha512_generate="shasum -a 512" + else + local -r sha256_generate="sha256sum" + local -r sha512_generate="sha512sum" + fi + + if [ "$#" -ne 2 ]; then + echo "Usage: $0 <version> <rc-num>" + exit 1 + fi + local -r version="$1" + local -r rc_number="$2" + + local -r tag="adbc-${version}" + local -r rc_branch="release-${version}-rc${rc_number}" + + echo "Preparing source for tag ${tag}" + local -r release_hash=$(cd "${source_top_dir}" && git rev-list --max-count=1 ${tag} --) + + if [[ -z "${release_hash}" ]]; then + echo "Cannot continue: unknown Git tag: ${tag}" + exit 1 + fi + + echo "Using commit ${release_hash}" + + local -r tarball="${tag}.tar.gz" + + pushd "${source_top_dir}" + + rm -rf "${tag}/" + git archive "${release_hash}" --prefix "${tag}/" | tar xf - + + # Resolve all hard and symbolic links + rm -rf "${tag}.tmp/" + mv "${tag}/" "${tag}.tmp/" + cp -R -L "${tag}.tmp" "${tag}" + rm -rf "${tag}.tmp/" + + # Create new tarball + tar czf "${tarball}" "${tag}/" + rm -rf "${tag}/" + + # check licenses + "${source_dir}/run-rat.sh" "${tarball}" + + # Sign the archive + gpg --armor --output "${tarball}.asc" --detach-sig "${tarball}" + ${sha256_generate} "${tarball}" | tee "${tarball}.sha256" + ${sha512_generate} "${tarball}" | tee "${tarball}.sha512" + + # Upload + if [[ "${SOURCE_UPLOAD}" -gt 0 ]]; then + echo "Uploading to dist.apache.org" + + local -r tagrc="${tag}-rc${rc_number}" Review Comment: Could you use `apache-arrow-adbc-` prefix like others? https://dist.apache.org/repos/dist/dev/arrow/ ########## dev/release/01-prepare.sh: ########## @@ -0,0 +1,81 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +set -ue + +SOURCE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" + +if [ "$#" -ne 3 ]; then + echo "Usage: $0 <version> <next_version> <rc-num>" + exit 1 +fi + +. $SOURCE_DIR/utils-prepare.sh + +version=$1 +next_version=$2 +next_version_snapshot="${next_version}-SNAPSHOT" +rc_number=$3 + +release_tag="adbc-${version}" +release_branch="release-${version}" +release_candidate_branch="release-${version}-rc${rc_number}" + +if [ $(git tag -l "${release_tag}") ]; then + echo "Delete existing git tag $release_tag" + git tag -d "${release_tag}" +fi + +if [[ $(git branch -l "${release_candidate_branch}") ]]; then + next_rc_number=$(($rc_number+1)) + echo "Branch ${release_candidate_branch} already exists, so create a new release candidate:" + echo "1. Checkout the default branch for major releases and maint-<version> for patch releases." + echo "2. Execute the script again with bumped RC number." + echo "Commands:" + echo " git checkout main" + echo " dev/release/01-prepare.sh ${version} ${next_version} ${next_rc_number}" + exit 1 +fi + +############################## Pre-Tag Commits ############################## + +echo "Updating changelog for $version" +# Update changelog +cz ch --incremental --unreleased-version "${release_tag}" +git add ${SOURCE_DIR}/../../CHANGELOG.md +git commit -m "chore: update CHANGELOG.md for $version" + +echo "Prepare release ${version} on tag ${release_tag}" + +update_versions "${version}" "${next_version}" "release" +git commit -m "chore: update versions for ${version}" + +############################## Tag the Release ############################## + +git tag -a "${release_tag}" -m "ADBC Libraries ${version}" Review Comment: How about using `adbc-${version}-rc${rc_number}` for RC tag? apache/arrow uses `apache-arrow-${version}` for RC tag but it's a historical reason. So ADBC doesn't need to follow the same style. ADBC can tag `adbc-${version}` to `adbc-${version}-rc${rc_number}` after the RC is accepted. FYI: Java's build system uses `apache-arrow-${version}`. But recent apache/arrow's release system doesn't use Java's build system for release. So I think that we can use `apache-arrow-${version}-rc${rc_number}` in apache/arrow. ########## dev/release/post-03-python.sh: ########## @@ -0,0 +1,71 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -e +set -u +set -o pipefail + +main() { + local -r source_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" + local -r source_top_dir="$( cd "${source_dir}/../../" && pwd )" + + if [ "$#" -ne 2 ]; then + echo "Usage: $0 <version> <rc-num>" + exit 1 + fi + + local -r version="$1" + local -r rc_number="$2" + local -r tag="adbc-${version}" + + : ${REPOSITORY:="apache/arrow-adbc"} + + local -r tmp=$(mktemp -d -t "arrow-post-python.XXXXX") + + header "Downloading Python packages for ${version}" + + gh release download \ + --repo "${REPOSITORY}" \ + "${tag}" \ + --dir "${tmp}" \ + --pattern "*.whl" \ + --pattern "adbc_*.tar.gz" `# sdist` Review Comment: ```suggestion --pattern "adbc_*.tar.gz" # sdist ``` ########## dev/release/post-05-remove-old-artifacts.sh: ########## @@ -0,0 +1,58 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -e +set -u +set -o pipefail + +echo "Remove all RCs" +dev_base_url=https://dist.apache.org/repos/dist/dev/arrow +old_rcs=$( + svn ls ${dev_base_url}/ | \ + grep -E '^adbc-[0-9]' | \ + sort --version-sort +) +for old_rc in $old_rcs; do + echo "Remove RC: ${old_rc}" + svn \ + delete \ + -m "Remove old Apache Arrow ADBC RC: ${old_rc}" \ + ${dev_base_url}/${old_rc} +done + +echo "Keep only the latest release" +release_base_url=https://dist.apache.org/repos/dist/release/arrow +old_releases=$( + svn ls ${release_base_url}/ | \ + grep -E '^adbc-[0-9]' | \ Review Comment: Could you use `apache-arrow-adbc-`? ########## dev/release/utils-binary.sh: ########## @@ -0,0 +1,87 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review Comment: We can remove this file. ########## dev/release/03-binary-submit.sh: ########## @@ -0,0 +1,64 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -ex + +main() { + local -r source_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" + local -r source_top_dir="$( cd "${source_dir}/../../" && pwd )" + pushd "${source_top_dir}" + + if [ "$#" -ne 2 ]; then + echo "Usage: $0 <version> <rc-num>" + exit 1 + fi + + local -r version="$1" + local -r rc_number="$2" + local -r tag="adbc-${version}" + local -r rc_branch="release-${version}-rc${rc_number}" + + : ${REPOSITORY:="apache/arrow"} + + echo "Starting GitHub Actions workflow on ${REPOSITORY}:${rc_branch}" + + gh workflow run \ + --repo "${REPOSITORY}" \ + --ref "${rc_branch}" \ + packaging-wheels.yml \ + --raw-field upload_artifacts=false Review Comment: How about adding the following to `.github/workflows/packaging-wheels.yml` instead of creating and running this file manually? ```diff diff --git a/.github/workflows/packaging-wheels.yml b/.github/workflows/packaging-wheels.yml index 19f88e5..6774260 100644 --- a/.github/workflows/packaging-wheels.yml +++ b/.github/workflows/packaging-wheels.yml @@ -18,6 +18,11 @@ name: Packaging - Nightly Artifacts on: + push: + branches-ignore: + - '**' + tags: + - '*' schedule: - cron: "0 0 * * *" workflow_dispatch: ``` ########## dev/release/.env.example: ########## @@ -0,0 +1,32 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# The GPG key ID to sign artifacts. The GPG key ID must be registered +# to both of the followings: +# +# * https://dist.apache.org/repos/dist/dev/arrow/KEYS +# * https://dist.apache.org/repos/dist/release/arrow/KEYS +# +# See these files how to import your GPG key ID to these files. +# +# You must set this. +#GPG_KEY_ID=08D3564B7C6A9CAFBFF6A66791D18FCF079F8007 + +# The Artifactory API key to upload artifacts to Artifactory. +# +# You must set this. +#ARTIFACTORY_API_KEY=secret Review Comment: We can remove this. ########## dev/release/06-java-upload.sh: ########## @@ -0,0 +1,133 @@ +#!/usr/bin/env bash Review Comment: How about making https://github.com/apache/arrow/blob/master/dev/release/06-java-upload.sh reusable from other repositories like https://github.com/apache/arrow/blob/master/dev/release/05-binary-upload.sh instead of coping this file? I can work on it. ########## dev/release/post-01-upload.sh: ########## @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -eu + +main() { + if [ "$#" -ne 2 ]; then + echo "Usage: $0 <version> <rc-num>" + exit 1 + fi + local -r version="$1" + local -r rc_number="$2" + + rc_id="adbc-${version}-rc${rc}" Review Comment: Could you use `apache-arrow-adbc-` prefix like others? https://dist.apache.org/repos/dist/dev/arrow/ ########## dev/release/post-05-remove-old-artifacts.sh: ########## @@ -0,0 +1,58 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -e +set -u +set -o pipefail + +echo "Remove all RCs" +dev_base_url=https://dist.apache.org/repos/dist/dev/arrow +old_rcs=$( + svn ls ${dev_base_url}/ | \ + grep -E '^adbc-[0-9]' | \ Review Comment: Could you use `apache-arrow-adbc-`? ########## docs/source/development/releasing.rst: ########## @@ -0,0 +1,255 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +======================== +Release Management Guide +======================== + +This page provides detailed information on the steps followed to perform +a release. It can be used both as a guide to learn the ADBC release +process and as a comprehensive checklist for the Release Manager when +performing a release. + +.. seealso:: + + `Apache Arrow Release Management Guide + <https://arrow.apache.org/docs/dev/developers/release.html>`_ + +Principles +========== + +The Apache Arrow Release follows the guidelines defined at the +`Apache Software Foundation Release Policy <https://www.apache.org/legal/release-policy.html>`_. + +Preparing for the release +========================= + +Some steps of the release require being a committer or a PMC member. + +- A GPG key in the Apache Web of Trust to sign artifacts. This will have to be cross signed by other Apache committers/PMC members. If you have multiple GPG keys, you must set the correct GPG key ID in ``~/.gnupg/gpg.conf`` by adding: + + .. code-block:: + + default-key ${YOUR_GPG_KEY_ID} + +- The GPG key needs to be added to this `SVN repo <https://dist.apache.org/repos/dist/dev/arrow/>`_ and `this one <https://dist.apache.org/repos/dist/release/arrow/>`_. +- Configure Maven to `publish artifacts to Apache repositories <http://www.apache.org/dev/publishing-maven-artifacts.html>`_. You will need to `setup a master password <https://maven.apache.org/guides/mini/guide-encryption.html>`_ at ``~/.m2/settings-security.xml`` and ``settings.xml`` as specified on the `Apache guide <http://www.apache.org/dev/publishing-maven-artifacts.html#dev-env>`_. It can be tested with the following command: + + .. code-block:: + + # You might need to export GPG_TTY=$(tty) to properly prompt for a passphrase + mvn clean install -Papache-release + +- Install ``en_US.UTF-8`` locale. You can confirm available locales by ``locale -a``. +- Install Conda with conda-forge, and create and activate the environment. + + .. code-block:: + + mamba create -n adbc -c conda-forge --file ci/conda_env_dev.txt + + This will install two tools used in the release process: ``commitizen`` (generates changelog from commit messages) and ``gh`` (submit jobs/download artifacts). + +- Have Docker and docker-compose installed. Review Comment: We don't need them now. ########## docs/source/development/releasing.rst: ########## @@ -0,0 +1,255 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +======================== +Release Management Guide +======================== + +This page provides detailed information on the steps followed to perform +a release. It can be used both as a guide to learn the ADBC release +process and as a comprehensive checklist for the Release Manager when +performing a release. + +.. seealso:: + + `Apache Arrow Release Management Guide + <https://arrow.apache.org/docs/dev/developers/release.html>`_ + +Principles +========== + +The Apache Arrow Release follows the guidelines defined at the +`Apache Software Foundation Release Policy <https://www.apache.org/legal/release-policy.html>`_. + +Preparing for the release +========================= + +Some steps of the release require being a committer or a PMC member. + +- A GPG key in the Apache Web of Trust to sign artifacts. This will have to be cross signed by other Apache committers/PMC members. If you have multiple GPG keys, you must set the correct GPG key ID in ``~/.gnupg/gpg.conf`` by adding: + + .. code-block:: + + default-key ${YOUR_GPG_KEY_ID} Review Comment: We specify this in `dev/release/.env` not `~/.gnupg/gpg.conf`. ########## dev/release/04-binary-download.sh: ########## @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -ex + +main() { + local -r source_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" + local -r source_top_dir="$( cd "${source_dir}/../../" && pwd )" + pushd "${source_top_dir}" + + if [ "$#" -ne 2 ]; then + echo "Usage: $0 <version> <rc-num>" + exit 1 + fi + + local -r version="$1" + local -r rc_number="$2" + local -r tag="adbc-${version}" + local -r rc_branch="release-${version}-rc${rc_number}" + + : ${REPOSITORY:="apache/arrow"} + + echo "Waiting for GitHub Actions workflow on ${REPOSITORY}:${rc_branch}" + + local -r run_id=$(gh run list \ + --repo "${REPOSITORY}" \ + --workflow=packaging-wheels.yml \ + --json 'databaseId,event,headBranch' \ + --jq ".[] | select(.event == \"workflow_dispatch\" and .headBranch == \"${rc_branch}\") | .databaseId" \ + | head -n1) + + echo "Found GitHub Actions workflow with ID: ${run_id}" + gh run watch --repo "${REPOSITORY}" --exit-status "${run_id}" + gh run view --repo "${REPOSITORY}" "${run_id}" + + local -r download_dir="packages/release-${version}-rc${rc_number}" + rm -rf "${download_dir}" + mkdir -p "${download_dir}" + gh run download --repo "${REPOSITORY}" --dir "${download_dir}" "${run_id}" Review Comment: How about uploading artifacts to GitHub Releases in `.github/workflows/packaging-wheels.yml` instead of creating this file? We can use https://github.com/softprops/action-gh-release for it. We can upload signs and checksums for artifacts in GitHub Releases separately by adding `dev/release/04-binary-sign.sh`. It lists artifacts in GitHub Releases, signs them, computes checksums of them and uploads these signs and checksums. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
