This is an automated email from the ASF dual-hosted git repository.
stoty pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/phoenix.git
The following commit(s) were added to refs/heads/master by this push:
new 6d84d0f PHOENIX-6178 Consider adopting the create-release scripts and
process from HBase
6d84d0f is described below
commit 6d84d0fcf8b906219af1232f49013938f38f5711
Author: Istvan Toth <st...@apache.org>
AuthorDate: Tue Oct 6 10:01:33 2020 +0200
PHOENIX-6178 Consider adopting the create-release scripts and process from
HBase
* copy and adapt dev/create-release from HBase
* remove source assembly
* add binary lincenses to binary assembly
* use assembly as binary distribution
---
dev/create-release/README.txt | 131 ++++
dev/create-release/do-release-docker.sh | 345 ++++++++++
dev/create-release/do-release.sh | 153 +++++
dev/create-release/mac-sshd-gpg-agent/Dockerfile | 101 +++
dev/create-release/phoenix-rm/Dockerfile | 66 ++
dev/create-release/release-build.sh | 302 +++++++++
dev/create-release/release-util.sh | 707 +++++++++++++++++++++
dev/create-release/vote.tmpl | 31 +
phoenix-assembly/pom.xml | 19 +-
.../src/build/components/all-common-files.xml | 26 +-
phoenix-assembly/src/build/src.xml | 127 ----
pom.xml | 3 +
12 files changed, 1854 insertions(+), 157 deletions(-)
diff --git a/dev/create-release/README.txt b/dev/create-release/README.txt
new file mode 100644
index 0000000..2179ca1
--- /dev/null
+++ b/dev/create-release/README.txt
@@ -0,0 +1,131 @@
+Entrance script is _do-release-docker.sh_. Requires a local docker;
+for example, on macOS, Docker for Desktop installed and running.
+
+For usage, pass '-h':
+
+ $ ./do-release-docker.sh -h
+
+./do-release-docker.sh accepts the following options
+
+ -d [path] required. working directory. output will be written to "output"
in here.
+ -f "force" -- actually publish this release. Unless you specify
'-f', it will
+ default to dry run mode, which checks and does local builds,
but does not upload anything.
+ -t [tag] tag for the phoenix-rm docker image to use for building
(default: "latest").
+ -j [path] path to local JDK installation to use building. By default the
script will
+ use openjdk8 installed in the docker image.
+ -p [project] project to build, such as 'phoenix' or 'phoenix-connectors';
defaults to phoenix env var
+ -r [repo] git repo to use for remote git operations. defaults to ASF
gitbox for project.
+ -s [step] runs a single step of the process; valid steps are:
tag|publish-dist|publish-release.
+ If none specified, runs tag, then publish-dist, and then
publish-release.
+ 'publish-snapshot' is also an allowed, less used, option.
+ -h display usage information
+ -x debug. do less clean up. (env file, gpg forwarding on mac)
+
+For example, use the following command do a full dry run build of
phoenix-thirdparty:
+
+./do-release-docker.sh -d /tmp/thirdparty-build -p phoenix-thirdparty
+
+To run a build w/o invoking docker (not recommended!), use _do_release.sh_.
+
+Both scripts will query interactively for needed parameters and passphrases.
+For explanation of the parameters, execute:
+ $ release-build.sh --help
+
+Before starting the RC build, run a reconciliation of what is in
+JIRA with what is in the commit log. Make sure they align and that
+anomalies are explained up in JIRA.
+
+See http://hbase.apache.org/book.html#maven.release
+(Even though the above documentation is for HBase, we use the same process for
Phoenix.)
+
+Regardless of where your release build will run (locally, locally in docker,
on a remote machine,
+etc) you will need a local gpg-agent with access to your secret keys. A quick
way to tell gpg
+to clear out state and start a gpg-agent is via the following command phrase:
+
+$ gpgconf --kill all && gpg-connect-agent /bye
+
+Before starting an RC build, make sure your local gpg-agent has configs
+to properly handle your credentials, especially if you want to avoid
+typing the passphrase to your secret key.
+
+e.g. if you are going to run and step away, best to increase the TTL
+on caching the unlocked secret via ~/.gnupg/gpg-agent.conf
+ # in seconds, e.g. a day
+ default-cache-ttl 86400
+ max-cache-ttl 86400
+
+Running a build on GCE is easy enough. Here are some notes if of use.
+Create an instance. 4CPU/15G/10G disk seems to work well enough.
+Once up, run the below to make your machine fit for RC building:
+
+Note that according to
https://www.apache.org/legal/release-policy.html#owned-controlled-hardware
+Building an Apache release must be done on hardware owned and controlled by
the committer.
+
+# Presuming debian-compatible OS, do these steps on the VM
+# your VM username should be your ASF id, because it will show up in build
artifacts.
+# Follow the docker install guide:
https://docs.docker.com/engine/install/debian/
+$ sudo apt-get install -y \
+ apt-transport-https \
+ ca-certificates \
+ curl \
+ gnupg2 \
+ software-properties-common
+$ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
+$ sudo add-apt-repository -y \
+ "deb [arch=amd64] https://download.docker.com/linux/debian \
+ $(lsb_release -cs) \
+ stable"
+$ sudo apt-get update
+$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
+# Follow the post installation steps:
https://docs.docker.com/engine/install/linux-postinstall/
+$ sudo usermod -aG docker $USER
+# LOGOUT and then LOGIN again so $USERID shows as part of docker group
+# Test here by running docker's hello world as your build user
+$ docker run hello-world
+
+# Follow the GPG guide for forwarding your gpg-agent from your local machine
to the VM
+# https://wiki.gnupg.org/AgentForwarding
+# On the VM find out the location of the gpg agent socket and extra socket
+$ gpgconf --list-dir agent-socket
+/run/user/1000/gnupg/S.gpg-agent
+$ gpgconf --list-dir agent-extra-socket
+/run/user/1000/gnupg/S.gpg-agent.extra
+# On the VM configure sshd to remove stale sockets
+$ sudo bash -c 'echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config'
+$ sudo systemctl restart ssh
+# logout of the VM
+
+# Do these steps on your local machine.
+# make sure gpg-agent is running
+$ gpg-connect-agent /bye
+# Export your public key and copy it to the VM.
+# Assuming 'example.gce.host' maps to your VM's external IP (or use the IP)
+$ gpg --export exam...@apache.org > ~/gpg.example.apache.pub
+$ scp ~/gpg.example.apache.pub example.gce.host:
+# ssh into the VM while forwarding the remote gpg socket locations found above
to your local
+# gpg-agent's extra socket (this will restrict what commands the remote node
is allowed to have
+# your agent handle. Note that the gpg guide above can help you set this up
in your ssh config
+# rather than typing it in ssh like this every time.
+$ ssh -i ~/.ssh/my_id \
+ -R "/run/user/1000/gnupg/S.gpg-agent:$(gpgconf --list-dir
agent-extra-socket)" \
+ -R "/run/user/1000/gnupg/S.gpg-agent.extra:$(gpgconf --list-dir
agent-extra-socket)" \
+ example.gce.host
+
+# now in an SSH session on the VM with the socket forwarding
+# import your public key and test signing with the forwarding to your local
agent.
+$ gpg --no-autostart --import gpg.example.apache.pub
+$ echo "foo" > foo.txt
+$ gpg --no-autostart --detach --armor --sign foo.txt
+$ gpg --no-autostart --verify foo.txt.asc
+
+# install git and clone the main project on the build machine
+$ sudo apt-get install -y git
+$ git clone https://github.com/apache/phoenix.git
+# finally set up an output folder and launch a dry run.
+$ mkdir ~/build
+$ cd phoenix
+$ ./dev/create-release/do-release-docker.sh -d ~/build
+
+# for building the main repo specifically you can save an extra download by
pointing the build
+# to the local clone you just made
+$ ./dev/create-release/do-release-docker.sh -d ~/build -r .git
diff --git a/dev/create-release/do-release-docker.sh
b/dev/create-release/do-release-docker.sh
new file mode 100755
index 0000000..4535611
--- /dev/null
+++ b/dev/create-release/do-release-docker.sh
@@ -0,0 +1,345 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+# Creates a Phoenix release candidate. The script will update versions, tag
the branch,
+# build Phoenix binary packages and documentation, and upload maven artifacts
to a staging
+# repository. There is also a dry run mode where only local builds are
performed, and
+# nothing is uploaded to the ASF repos.
+#
+# Run with "-h" for options. For example, running below will do all
+# steps above using the 'rm' dir under Downloads as workspace:
+#
+# $ ./do-release-docker.sh -d ~/Downloads/rm
+#
+# The scripts in this directory came originally from spark [1]. They were then
+# modified to suite the hbase context, which were further adopted to Phoenix.
+# These scripts supercedes the old
+# ../make_rc.sh script for making release candidates because what is here is
more
+# comprehensive doing more steps of the RM process as well as running in a
+# container so the RM build environment can be a constant.
+#
+# It:
+# * Tags release
+# * Sets version to the release version
+# * Sets version to next SNAPSHOT version.
+# * Builds, signs, and hashes all artifacts.
+# * Pushes release tgzs to the dev dir in a apache dist.
+# * Pushes to repository.apache.org staging.
+#
+# The entry point is here, in the do-release-docker.sh script.
+#
+# 1. https://github.com/apache/spark/tree/master/dev/create-release
+#
+set -e
+
+# Set this to build other phoenix repos: e.g. PROJECT=phoenix-connectors
+export PROJECT="${PROJECT:-phoenix}"
+
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=SCRIPTDIR/release-util.sh
+. "$SELF/release-util.sh"
+ORIG_PWD="$(pwd)"
+
+function usage {
+ local NAME
+ NAME="$(basename "${BASH_SOURCE[0]}")"
+ cat <<EOF
+Usage: $NAME [options]
+
+This script runs the release scripts inside a docker image.
+
+Options:
+
+ -d [path] required. working directory. output will be written to "output"
in here.
+ -f "force" -- actually publish this release. Unless you specify
'-f', it will
+ default to dry run mode, which checks and does local builds,
but does not upload anything.
+ -t [tag] tag for the phoenix-rm docker image to use for building
(default: "latest").
+ -j [path] path to local JDK installation to use building. By default the
script will
+ use openjdk8 installed in the docker image.
+ -p [project] project to build, such as 'phoenix' or 'phoenix-connectors';
defaults to $PROJECT env var
+ -r [repo] git repo to use for remote git operations. defaults to ASF
gitbox for project.
+ -s [step] runs a single step of the process; valid steps are:
tag|publish-dist|publish-release.
+ If none specified, runs tag, then publish-dist, and then
publish-release.
+ 'publish-snapshot' is also an allowed, less used, option.
+ -h display usage information
+ -x debug. do less clean up. (env file, gpg forwarding on mac)
+EOF
+ exit 1
+}
+
+WORKDIR=
+IMGTAG=latest
+JAVA=
+RELEASE_STEP=
+GIT_REPO=
+while getopts "d:fhj:p:r:s:t:x" opt; do
+ case $opt in
+ d) WORKDIR="$OPTARG" ;;
+ f) DRY_RUN=0 ;;
+ t) IMGTAG="$OPTARG" ;;
+ j) JAVA="$OPTARG" ;;
+ p) PROJECT="$OPTARG" ;;
+ r) GIT_REPO="$OPTARG" ;;
+ s) RELEASE_STEP="$OPTARG" ;;
+ x) DEBUG=1 ;;
+ h) usage ;;
+ ?) error "Invalid option. Run with -h for help." ;;
+ esac
+done
+shift $((OPTIND-1))
+if (( $# > 0 )); then
+ error "Arguments can only be provided with option flags, invalid args: $*"
+fi
+export DEBUG
+
+if [ -z "$WORKDIR" ] || [ ! -d "$WORKDIR" ]; then
+ error "Work directory (-d) must be defined and exist. Run with -h for help."
+fi
+
+if [ -d "$WORKDIR/output" ]; then
+ read -r -p "Output directory already exists. Overwrite and continue? [y/n] "
ANSWER
+ if [ "$ANSWER" != "y" ]; then
+ error "Exiting."
+ fi
+fi
+
+if [ -f "${WORKDIR}/gpg-proxy.ssh.pid" ] || \
+ [ -f "${WORKDIR}/gpg-proxy.cid" ] || \
+ [ -f "${WORKDIR}/release.cid" ]; then
+ read -r -p "container/pid files from prior run exists. Overwrite and
continue? [y/n] " ANSWER
+ if [ "$ANSWER" != "y" ]; then
+ error "Exiting."
+ fi
+fi
+
+cd "$WORKDIR"
+rm -rf "$WORKDIR/output"
+rm -rf "${WORKDIR}/gpg-proxy.ssh.pid" "${WORKDIR}/gpg-proxy.cid"
"${WORKDIR}/release.cid"
+mkdir "$WORKDIR/output"
+
+banner "Gathering release details."
+HOST_OS="$(get_host_os)"
+get_release_info
+
+banner "Setup"
+
+# Place all RM scripts and necessary data in a local directory that must be
defined in the command
+# line. This directory is mounted into the image. Its WORKDIR, the arg passed
with -d.
+for f in "$SELF"/*; do
+ if [ -f "$f" ]; then
+ cp "$f" "$WORKDIR"
+ fi
+done
+
+# We need to import that public key in the container in order to use the
private key via the agent.
+GPG_KEY_FILE="$WORKDIR/gpg.key.public"
+echo "Exporting public key for ${GPG_KEY}"
+fcreate_secure "$GPG_KEY_FILE"
+$GPG "${GPG_ARGS[@]}" --export "${GPG_KEY}" > "${GPG_KEY_FILE}"
+
+function cleanup {
+ local id
+ banner "Release Cleanup"
+ if is_debug; then
+ echo "skipping due to debug run"
+ return 0
+ fi
+ echo "details in cleanup.log"
+ if [ -f "${ENVFILE}" ]; then
+ rm -f "$ENVFILE"
+ fi
+ rm -f "$GPG_KEY_FILE"
+ if [ -f "${WORKDIR}/gpg-proxy.ssh.pid" ]; then
+ id=$(cat "${WORKDIR}/gpg-proxy.ssh.pid")
+ echo "Stopping ssh tunnel for gpg-agent at PID ${id}" | tee -a cleanup.log
+ kill -9 "${id}" >>cleanup.log 2>&1 || true
+ rm -f "${WORKDIR}/gpg-proxy.ssh.pid" >>cleanup.log 2>&1
+ fi
+ if [ -f "${WORKDIR}/gpg-proxy.cid" ]; then
+ id=$(cat "${WORKDIR}/gpg-proxy.cid")
+ echo "Stopping gpg-proxy container with ID ${id}" | tee -a cleanup.log
+ docker kill "${id}" >>cleanup.log 2>&1 || true
+ rm -f "${WORKDIR}/gpg-proxy.cid" >>cleanup.log 2>&1
+ # TODO we should remove the gpgagent volume?
+ fi
+ if [ -f "${WORKDIR}/release.cid" ]; then
+ id=$(cat "${WORKDIR}/release.cid")
+ echo "Stopping release container with ID ${id}" | tee -a cleanup.log
+ docker kill "${id}" >>cleanup.log 2>&1 || true
+ rm -f "${WORKDIR}/release.cid" >>cleanup.log 2>&1
+ fi
+}
+
+trap cleanup EXIT
+
+echo "Host OS: ${HOST_OS}"
+if [ "${HOST_OS}" == "DARWIN" ]; then
+ run_silent "Building gpg-agent-proxy image with tag ${IMGTAG}..."
"docker-proxy-build.log" \
+ docker build --build-arg "UID=${UID}" --build-arg "RM_USER=${USER}" \
+ --tag "org.apache.phoenix/gpg-agent-proxy:${IMGTAG}"
"${SELF}/mac-sshd-gpg-agent"
+fi
+
+run_silent "Building phoenix-rm image with tag $IMGTAG..." "docker-build.log" \
+ docker build --tag "org.apache.phoenix/phoenix-rm:$IMGTAG" --build-arg
"UID=$UID" \
+ --build-arg "RM_USER=${USER}" "$SELF/phoenix-rm"
+
+banner "Final prep for container launch."
+echo "Writing out environment for container."
+# Write the release information to a file with environment variables to be
used when running the
+# image.
+ENVFILE="$WORKDIR/env.list"
+fcreate_secure "$ENVFILE"
+
+cat > "$ENVFILE" <<EOF
+PROJECT=$PROJECT
+DRY_RUN=$DRY_RUN
+SKIP_TAG=$SKIP_TAG
+RUNNING_IN_DOCKER=1
+GIT_BRANCH=$GIT_BRANCH
+NEXT_VERSION=$NEXT_VERSION
+RELEASE_VERSION=$RELEASE_VERSION
+RELEASE_TAG=$RELEASE_TAG
+GIT_REF=$GIT_REF
+ASF_USERNAME=$ASF_USERNAME
+GIT_NAME=$GIT_NAME
+GIT_EMAIL=$GIT_EMAIL
+GPG_KEY=$GPG_KEY
+ASF_PASSWORD=$ASF_PASSWORD
+RELEASE_STEP=$RELEASE_STEP
+API_DIFF_TAG=$API_DIFF_TAG
+HOST_OS=$HOST_OS
+EOF
+
+JAVA_MOUNT=()
+if [ -n "$JAVA" ]; then
+ echo "JAVA_HOME=/opt/phoenix-java" >> "$ENVFILE"
+ JAVA_MOUNT=(--mount "type=bind,src=${JAVA},dst=/opt/phoenix-java,readonly")
+fi
+
+#TODO some debug output would be good here
+GIT_REPO_MOUNT=()
+if [ -n "${GIT_REPO}" ]; then
+ case "${GIT_REPO}" in
+ # skip the easy to identify remote protocols
+ ssh://*|git://*|http://*|https://*|ftp://*|ftps://*) ;;
+ # for sure local
+ /*)
+ GIT_REPO_MOUNT=(--mount
"type=bind,src=${GIT_REPO},dst=/opt/phoenix-repo,consistency=delegated")
+ echo "HOST_GIT_REPO=${GIT_REPO}" >> "${ENVFILE}"
+ GIT_REPO="/opt/phoenix-repo"
+ ;;
+ # on the host but normally git wouldn't use the local optimization
+ file://*)
+ echo "[INFO] converted file:// git repo to a local path, which changes
git to assume --local."
+ GIT_REPO_MOUNT=(--mount
"type=bind,src=${GIT_REPO#file://},dst=/opt/phoenix-repo,consistency=delegated")
+ echo "HOST_GIT_REPO=${GIT_REPO}" >> "${ENVFILE}"
+ GIT_REPO="/opt/phoenix-repo"
+ ;;
+ # have to decide if it's a local path or the "scp-ish" remote
+ *)
+ declare colon_remove_prefix;
+ declare slash_remove_prefix;
+ declare local_path;
+ colon_remove_prefix="${GIT_REPO#*:}"
+ slash_remove_prefix="${GIT_REPO#*/}"
+ if [ "${GIT_REPO}" = "${colon_remove_prefix}" ]; then
+ # if there was no colon at all, we assume this must be a local path
+ local_path="no colon at all"
+ elif [ "${GIT_REPO}" != "${slash_remove_prefix}" ]; then
+ # if there was a colon and there is no slash, then we assume it must
be scp-style host
+ # and a relative path
+
+ if [ "${#colon_remove_prefix}" -lt "${#slash_remove_prefix}" ]; then
+ # Given the substrings made by removing everything up to the first
colon and slash
+ # we can determine which comes first based on the longer substring
length.
+ # if the slash is first, then we assume the colon is part of a path
name and if the colon
+ # is first then it is the seperator between a scp-style host name
and the path.
+ local_path="slash happened before a colon"
+ fi
+ fi
+ if [ -n "${local_path}" ]; then
+ # convert to an absolute path
+ GIT_REPO="$(cd "$(dirname "${ORIG_PWD}/${GIT_REPO}")"; pwd)/$(basename
"${ORIG_PWD}/${GIT_REPO}")"
+ GIT_REPO_MOUNT=(--mount
"type=bind,src=${GIT_REPO},dst=/opt/phoenix-repo,consistency=delegated")
+ echo "HOST_GIT_REPO=${GIT_REPO}" >> "${ENVFILE}"
+ GIT_REPO="/opt/phoenix-repo"
+ fi
+ ;;
+ esac
+ echo "GIT_REPO=${GIT_REPO}" >> "${ENVFILE}"
+fi
+
+GPG_PROXY_MOUNT=()
+if [ "${HOST_OS}" == "DARWIN" ]; then
+ GPG_PROXY_MOUNT=(--mount
"type=volume,src=gpgagent,dst=/home/${USER}/.gnupg/")
+ echo "Setting up GPG agent proxy container needed on OS X."
+ echo " we should clean this up for you. If that fails the container ID is
below and in " \
+ "gpg-proxy.cid"
+ #TODO the key pair used should be configurable
+ docker run --rm -p 62222:22 \
+ --detach --cidfile "${WORKDIR}/gpg-proxy.cid" \
+ --mount \
+
"type=bind,src=${HOME}/.ssh/id_rsa.pub,dst=/home/${USER}/.ssh/authorized_keys,readonly"
\
+ "${GPG_PROXY_MOUNT[@]}" \
+ "org.apache.phoenix/gpg-agent-proxy:${IMGTAG}"
+ # gotta trust the container host
+ ssh-keyscan -p 62222 localhost 2>/dev/null | sort >
"${WORKDIR}/gpg-agent-proxy.ssh-keyscan"
+ sort "${HOME}/.ssh/known_hosts" | comm -1 -3 -
"${WORKDIR}/gpg-agent-proxy.ssh-keyscan" \
+ > "${WORKDIR}/gpg-agent-proxy.known_hosts"
+ if [ -s "${WORKDIR}/gpg-agent-proxy.known_hosts" ]; then
+ echo "Your ssh known_hosts does not include the entries for the gpg-agent
proxy container."
+ echo "The following entry(ies) arre missing:"
+ sed -e 's/^/ /' "${WORKDIR}/gpg-agent-proxy.known_hosts"
+ read -r -p "Okay to add these entries to ${HOME}/.ssh/known_hosts? [y/n] "
ANSWER
+ if [ "$ANSWER" != "y" ]; then
+ error "Exiting."
+ fi
+ cat "${WORKDIR}/gpg-agent-proxy.known_hosts" >> "${HOME}/.ssh/known_hosts"
+ fi
+ echo "Launching ssh reverse tunnel from the container to gpg agent."
+ echo " we should clean this up for you. If that fails the PID is in
gpg-proxy.ssh.pid"
+ ssh -p 62222 -R "/home/${USER}/.gnupg/S.gpg-agent:$(gpgconf --list-dir
agent-extra-socket)" \
+ -i "${HOME}/.ssh/id_rsa" -N -n localhost >gpg-proxy.ssh.log 2>&1 &
+ echo $! > "${WORKDIR}/gpg-proxy.ssh.pid"
+else
+ # Note that on linux we always directly mount the gpg agent's extra socket
to limit what the
+ # container can ask the gpg-agent to do.
+ # When working on a remote linux machine you should be sure to forward both
the remote machine's
+ # agent socket and agent extra socket to your local gpg-agent's extra
socket. See the README.txt
+ # for an example.
+ GPG_PROXY_MOUNT=(--mount \
+ "type=bind,src=$(gpgconf --list-dir
agent-extra-socket),dst=/home/${USER}/.gnupg/S.gpg-agent")
+fi
+
+banner "Building $RELEASE_TAG; output will be at $WORKDIR/output"
+echo "We should clean the container up when we are done. If that fails then
the container ID " \
+ "is in release.cid"
+echo
+# Where possible we specifcy "consistency=delegated" when we do not need host
access during the
+# build run. On Mac OS X specifically this gets us a big perf improvement.
+cmd=(docker run --rm -ti \
+ --env-file "$ENVFILE" \
+ --cidfile "${WORKDIR}/release.cid" \
+ --mount
"type=bind,src=${WORKDIR},dst=/home/${USER}/phoenix-rm,consistency=delegated" \
+ "${JAVA_MOUNT[@]}" \
+ "${GIT_REPO_MOUNT[@]}" \
+ "${GPG_PROXY_MOUNT[@]}" \
+ "org.apache.phoenix/phoenix-rm:$IMGTAG")
+echo "${cmd[*]}"
+"${cmd[@]}"
diff --git a/dev/create-release/do-release.sh b/dev/create-release/do-release.sh
new file mode 100755
index 0000000..fd613e2
--- /dev/null
+++ b/dev/create-release/do-release.sh
@@ -0,0 +1,153 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+# Use the adjacent do-release-docker.sh instead, if you can.
+# Otherwise, this runs core of the release creation.
+# Will ask you questions on what to build and for logins
+# and passwords to use building.
+export PROJECT="${PROJECT:-phoenix}"
+
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=SCRIPTDIR/release-util.sh
+. "$SELF/release-util.sh"
+
+while getopts "b:fs:" opt; do
+ case $opt in
+ b) export GIT_BRANCH=$OPTARG ;;
+ f) export DRY_RUN=0 ;; # "force", ie actually publish this release
(otherwise defaults to dry run)
+ s) RELEASE_STEP="$OPTARG" ;;
+ ?) error "Invalid option: $OPTARG" ;;
+ esac
+done
+shift $((OPTIND-1))
+if (( $# > 0 )); then
+ error "Arguments can only be provided with option flags, invalid args: $*"
+fi
+
+function gpg_agent_help {
+ cat <<EOF
+Trying to sign a test file using your GPG setup failed.
+
+Please make sure you have a local gpg-agent running with access to your secret
keys prior to
+starting a release build. If you are creating release artifacts on a remote
machine please check
+that you have set up ssh forwarding to the gpg-agent extra socket.
+
+For help on how to do this please see the README file in the create-release
directory.
+EOF
+ exit 1
+}
+
+# If running in docker, import and then cache keys.
+if [ "$RUNNING_IN_DOCKER" = "1" ]; then
+ # when Docker Desktop for mac is running under load there is a delay before
the mounted volume
+ # becomes available. if we do not pause then we may try to use the gpg-agent
socket before docker
+ # has got it ready and we will not think there is a gpg-agent.
+ if [ "${HOST_OS}" == "DARWIN" ]; then
+ sleep 5
+ fi
+ # in docker our working dir is set to where all of our scripts are held
+ # and we want default output to go into the "output" directory that should
be in there.
+ if [ -d "output" ]; then
+ cd output
+ fi
+ echo "GPG Version: $("${GPG}" "${GPG_ARGS[@]}" --version)"
+ # Inside docker, need to import the GPG key stored in the current directory.
+ if ! $GPG "${GPG_ARGS[@]}" --import "$SELF/gpg.key.public" ; then
+ gpg_agent_help
+ fi
+
+ # We may need to adjust the path since JAVA_HOME may be overridden by the
driver script.
+ if [ -n "$JAVA_HOME" ]; then
+ echo "Using JAVA_HOME from host."
+ export PATH="$JAVA_HOME/bin:$PATH"
+ else
+ # JAVA_HOME for the openjdk package.
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
+ fi
+else
+ # Outside docker, need to ask for information about the release.
+ get_release_info
+fi
+
+GPG_TTY="$(tty)"
+export GPG_TTY
+echo "Testing gpg signing."
+echo "foo" > gpg_test.txt
+if ! "${GPG}" "${GPG_ARGS[@]}" --detach --armor --sign gpg_test.txt ; then
+ gpg_agent_help
+fi
+# In --batch mode we have to be explicit about what we are verifying
+if ! "${GPG}" "${GPG_ARGS[@]}" --verify gpg_test.txt.asc gpg_test.txt ; then
+ gpg_agent_help
+fi
+
+if [[ -z "$RELEASE_STEP" ]]; then
+ # If doing all stages, leave out 'publish-snapshot'
+ RELEASE_STEP="tag_publish-dist_publish-release"
+ # and use shared maven local repo for efficiency
+ export REPO="${REPO:-$(pwd)/$(mktemp -d phoenix-repo-XXXXX)}"
+fi
+
+function should_build {
+ local WHAT=$1
+ if [[ -z "$RELEASE_STEP" ]]; then
+ return 0
+ elif [[ "$RELEASE_STEP" == *"$WHAT"* ]]; then
+ return 0
+ else
+ return 1
+ fi
+}
+
+if should_build "tag" && [ "$SKIP_TAG" = 0 ]; then
+ if [ -z "${YETUS_HOME}" ] && [ "${RUNNING_IN_DOCKER}" != "1" ]; then
+ declare local_yetus="/opt/apache-yetus/0.12.0/"
+ if [ "$(get_host_os)" = "DARWIN" ]; then
+ local_yetus="/usr/local/Cellar/yetus/0.12.0/"
+ fi
+ YETUS_HOME="$(read_config "YETUS_HOME not defined. Absolute path to local
install of Apache Yetus" "${local_yetus}")"
+ export YETUS_HOME
+ fi
+ run_silent "Creating release tag $RELEASE_TAG..." "tag.log" \
+ "$SELF/release-build.sh" tag
+ if is_dry_run; then
+ export TAG_SAME_DRY_RUN="true";
+ fi
+else
+ echo "Skipping tag creation for $RELEASE_TAG."
+fi
+
+if should_build "publish-dist"; then
+ run_silent "Publishing distribution packages (tarballs)" "publish-dist.log" \
+ "$SELF/release-build.sh" publish-dist
+else
+ echo "Skipping publish-dist step."
+fi
+
+if should_build "publish-snapshot"; then
+ run_silent "Publishing snapshot" "publish-snapshot.log" \
+ "$SELF/release-build.sh" publish-snapshot
+
+elif should_build "publish-release"; then
+ run_silent "Publishing release" "publish-release.log" \
+ "$SELF/release-build.sh" publish-release
+else
+ echo "Skipping publish-release step."
+fi
diff --git a/dev/create-release/mac-sshd-gpg-agent/Dockerfile
b/dev/create-release/mac-sshd-gpg-agent/Dockerfile
new file mode 100644
index 0000000..084688d
--- /dev/null
+++ b/dev/create-release/mac-sshd-gpg-agent/Dockerfile
@@ -0,0 +1,101 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Image for use on Mac boxes to get a gpg agent socket available
+# within transient release building ocntainers.
+#
+# Copied directly from HBase
+#
+# build like:
+#
+# docker build --build-arg "UID=$UID" --build-arg "RM_USER=$USER" \
+# --tag org.apache.phoenix/gpg-agent-proxy mac-sshd-gpg-agent
+#
+# run like:
+#
+# docker run --rm -p 62222:22 \
+# --mount
"type=bind,src=${HOME}/.ssh/id_rsa.pub,dst=/home/${USER}/.ssh/authorized_keys,readonly"
\
+# --mount "type=volume,src=gpgagent,dst=/home/${USER}/.gnupg/" \
+# org.apache.phoenix/gpg-agent-proxy:latest
+#
+# test like:
+#
+# ssh -p 62222 -R "/home/${USER}/.gnupg/S.gpg-agent:$(gpgconf --list-dir
agent-extra-socket)" \
+# -i "${HOME}/.ssh/id_rsa" -N -n localhost
+#
+# launch a docker container to do work that shares the mount for the gpg agent
+# expressly does not need to be this same image, but needs to have defined the
same user
+#
+# docker run --rm -it \
+# --mount "type=volume,src=gpgagent,dst=/home/${USER}/.gnupg/" \
+# --mount
"type=bind,src=${HOME}/projects/phoenix-releases/KEYS,dst=/home/${USER}/KEYS,readonly"
\
+# --entrypoint /bin/bash --user "${USER}" --workdir "/home/${USER}/" \
+# org.apache.phoenix/gpg-agent-proxy:latest
+#
+#
+# Make sure to import the public keys
+#
+# gpg --no-autostart --import < ${HOME}/KEYS
+# Optional?
+# gpg --no-autostart --edit-key ${YOUR_KEY}
+# trust
+# 5
+# y
+# quit
+#ubu
+# gpg --no-autostart --armor --detach --sign foo
+# gpg --no-autostart --verify foo.asc
+#
+# For more info see
+# * gpg forwarding over ssh: https://wiki.gnupg.org/AgentForwarding
+# * example docker for sshd: https://github.com/hotblac/nginx-ssh
+# * why we have to bother with this:
https://github.com/docker/for-mac/issues/483
+#
+# If the docker image changes then the host key used by sshd will change and
you will get a
+# nastygram when launching ssh about host identification changing. This is
expected. you should
+# remove the previous host key.
+#
+# Tested with
+# * Docker Desktop 2.2.0.5
+# * gpg 2.2.20
+# * pinentry-mac 0.9.4
+# * yubikey 5
+#
+FROM ubuntu:18.04
+
+# This is all in a single "RUN" command so that if anything changes, "apt
update" is run to fetch
+# the most current package versions (instead of potentially using old versions
cached by docker).
+#
+# We only need gnupg2 here if we want the ability to test out the gpg-agent
forwarding by sshing
+# into the container rather than launching a new docker container.
+RUN DEBIAN_FRONTEND=noninteractive apt-get -qq -y update \
+ && DEBIAN_FRONTEND=noninteractive apt-get -qq -y install
--no-install-recommends \
+ openssh-server=1:7.6* gnupg2=2.2.4* && mkdir /run/sshd \
+ && echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+EXPOSE 22
+# Set up our ssh user
+ARG UID
+ARG RM_USER
+RUN groupadd sshgroup && \
+ useradd --create-home --shell /bin/bash --groups sshgroup --uid $UID
$RM_USER && \
+ mkdir /home/$RM_USER/.ssh /home/$RM_USER/.gnupg && \
+ chown -R $RM_USER:sshgroup /home/$RM_USER/ && \
+ chmod -R 700 /home/$RM_USER/
+# When we run we run sshd
+ENTRYPOINT ["/usr/sbin/sshd", "-D"]
diff --git a/dev/create-release/phoenix-rm/Dockerfile
b/dev/create-release/phoenix-rm/Dockerfile
new file mode 100644
index 0000000..3522760
--- /dev/null
+++ b/dev/create-release/phoenix-rm/Dockerfile
@@ -0,0 +1,66 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Image for building Phoenix releases. Based on Ubuntu 18.04.
+#
+# Copied directly from HBase
+#
+# Includes:
+# * Java 8
+FROM ubuntu:18.04
+
+# Install extra needed repos and refresh.
+#
+# This is all in a single "RUN" command so that if anything changes, "apt
update" is run to fetch
+# the most current package versions (instead of potentially using old versions
cached by docker).
+RUN DEBIAN_FRONTEND=noninteractive apt-get -qq -y update \
+ && DEBIAN_FRONTEND=noninteractive apt-get -qq -y install
--no-install-recommends \
+ curl='7.58.0-*' \
+ git='1:2.17.1-*' \
+ gnupg='2.2.4-*' \
+ libcurl4-openssl-dev='7.58.0-*' \
+ libxml2-dev='2.9.4+dfsg1-*' \
+ lsof='4.89+dfsg-*' \
+ maven='3.6.0-*' \
+ openjdk-8-jdk='8*' \
+ python-pip='9.0.1-*' \
+ subversion='1.9.7-*' \
+ wget='1.19.4-*' \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/* \
+ && update-alternatives --set java
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java \
+ && pip install \
+ python-dateutil==2.8.1
+# Install Apache Yetus
+ENV YETUS_VERSION 0.12.0
+SHELL ["/bin/bash", "-o", "pipefail", "-c"]
+RUN wget -qO-
"https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=/yetus/${YETUS_VERSION}/apache-yetus-${YETUS_VERSION}-bin.tar.gz";
| \
+ tar xvz -C /opt
+ENV YETUS_HOME /opt/apache-yetus-${YETUS_VERSION}
+
+ARG UID
+ARG RM_USER
+RUN groupadd phoenix-rm && \
+ useradd --create-home --shell /bin/bash -p phoenix-rm -u $UID $RM_USER && \
+ mkdir /home/$RM_USER/.gnupg && \
+ chown -R $RM_USER:phoenix-rm /home/$RM_USER && \
+ chmod -R 700 /home/$RM_USER
+
+USER $RM_USER:phoenix-rm
+WORKDIR /home/$RM_USER/phoenix-rm/
+
+ENTRYPOINT [ "./do-release.sh" ]
diff --git a/dev/create-release/release-build.sh
b/dev/create-release/release-build.sh
new file mode 100755
index 0000000..a1442e4
--- /dev/null
+++ b/dev/create-release/release-build.sh
@@ -0,0 +1,302 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+trap cleanup EXIT
+
+# Source in utils.
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=SCRIPTDIR/release-util.sh
+. "$SELF/release-util.sh"
+
+# Print usage and exit.
+function exit_with_usage {
+ cat <<'EOF'
+Usage: release-build.sh <tag|publish-dist|publish-snapshot|publish-release>
+Creates release deliverables from a tag or commit.
+Argument: one of 'tag', 'publish-dist', 'publish-snapshot', or
'publish-release'
+ tag Prepares for release on specified git branch: Set release
version,
+ update CHANGES and RELEASENOTES, create release tag,
+ increment version for ongoing dev, and publish to Apache
git repo.
+ publish-dist Build and publish distribution packages (tarballs) to
Apache dist repo
+ publish-snapshot Build and publish maven artifacts snapshot release to
Apache snapshots repo
+ publish-release Build and publish maven artifacts release to Apache
release repo, and
+ construct vote email from template
+
+All other inputs are environment variables. Please use do-release-docker.sh or
+do-release.sh to set up the needed environment variables. This script,
release-build.sh,
+is not intended to be called stand-alone, and such use is untested. The env
variables used are:
+
+Used for 'tag' and 'publish' stages:
+ PROJECT - The project to build. No default.
+ RELEASE_VERSION - Version used in pom files for release (e.g. 2.1.2)
+ Required for 'tag'; defaults for 'publish' to the version in pom at GIT_REF
+ RELEASE_TAG - Name of release tag (e.g. 2.1.2RC0), also used by
+ publish-dist as package version name in dist directory path
+ ASF_USERNAME - Username of ASF committer account
+ ASF_PASSWORD - Password of ASF committer account
+ DRY_RUN - 1:true (default), 0:false. If "1", does almost all the work, but
doesn't actually
+ publish anything to upstream source or object repositories. It defaults to
"1", so if you want
+ to actually publish you have to set '-f' (force) flag in do-release.sh or
do-release-docker.sh.
+
+Used only for 'tag':
+ YETUS_HOME - installation location for Apache Yetus
+ GIT_NAME - Name to use with git
+ GIT_EMAIL - E-mail address to use with git
+ GIT_BRANCH - Git branch on which to make release. Tag is always placed at
HEAD of this branch.
+ NEXT_VERSION - Development version after release (e.g. 2.1.3-SNAPSHOT)
+
+Used only for 'publish':
+ GIT_REF - Release tag or commit to build from (defaults to $RELEASE_TAG;
only need to
+ separately define GIT_REF if RELEASE_TAG is not actually present as a tag
at publish time)
+ If both RELEASE_TAG and GIT_REF are undefined it will default to HEAD of
master.
+ GPG_KEY - GPG key id (usually email addr) used to sign release artifacts
+ REPO - Set to full path of a directory to use as maven local repo
(dependencies cache)
+ to avoid re-downloading dependencies for each stage. It is automatically
set if you
+ request full sequence of stages (tag, publish-dist, publish-release) in
do-release.sh.
+
+For example:
+ $ PROJECT="phoenix-connectors" ASF_USERNAME=NAME ASF_PASSWORD=PASSWORD
GPG_KEY=st...@apache.org ./release-build.sh publish-dist
+EOF
+ exit 1
+}
+
+set -e
+
+function cleanup {
+ # If REPO was set, then leave things be. Otherwise if we defined a repo
clean it out.
+ if [[ -z "${REPO}" ]] && [[ -n "${MAVEN_LOCAL_REPO}" ]]; then
+ echo "Cleaning up temp repo in '${MAVEN_LOCAL_REPO}'. Set REPO to reuse
downloads." >&2
+ rm -f "${MAVEN_SETTINGS_FILE}" &> /dev/null || true
+ rm -rf "${MAVEN_LOCAL_REPO}" &> /dev/null || true
+ fi
+}
+
+if [ $# -ne 1 ]; then
+ exit_with_usage
+fi
+
+if [[ "$*" == *"help"* ]]; then
+ exit_with_usage
+fi
+
+init_locale
+init_java
+init_mvn
+init_python
+# Print out subset of perl version (used in git hooks and
japi-compliance-checker)
+perl --version | grep 'This is'
+
+rm -rf "${PROJECT}"
+
+if is_debug; then
+ set -x # detailed logging during action
+fi
+
+if [[ "$1" == "tag" ]]; then
+ init_yetus
+ # for 'tag' stage
+ set -o pipefail
+ check_get_passwords ASF_PASSWORD
+ check_needed_vars PROJECT RELEASE_VERSION RELEASE_TAG NEXT_VERSION GIT_EMAIL
GIT_NAME GIT_BRANCH
+ if [ -z "${GIT_REPO}" ]; then
+ check_needed_vars ASF_USERNAME ASF_PASSWORD
+ fi
+ git_clone_overwrite
+
+ # 'update_releasenotes' searches the project's Jira for issues where 'Fix
Version' matches specified
+ # $jira_fix_version. For most projects this is same as ${RELEASE_VERSION}.
However, the original 'phoenix-*'
+ # projects share the same PHOENIX jira name. To make this work, by
convention, the PHOENIX jira "Fix Version"
+ # field values have the sub-project name without the phoenix- prefix
pre-pended, as in "connectors-6.0.0".
+ # So, here we prepend the project name to the version, but only for the
original phoenix sub-projects.
+ # phoenix-omid and phoenix-tephra have their own JIRA projects and
versioning.
+ jira_fix_version="${RELEASE_VERSION}"
+ shopt -s nocasematch
+ if [[ "${PROJECT}" == "phoenix-queryserver" || "${PROJECT}" ==
"phoenix-connectors" || "${PROJECT}" == "phoenix-thirdparty" ]]; then
+ jira_fix_version="${PROJECT#phoenix-}-${RELEASE_VERSION}"
+ fi
+
+ shopt -u nocasematch
+ update_releasenotes "$(pwd)/${PROJECT}" "${jira_fix_version}"
+
+ cd "${PROJECT}"
+
+ git config user.name "$GIT_NAME"
+ git config user.email "$GIT_EMAIL"
+
+ # Create release version
+ maven_set_version "$RELEASE_VERSION"
+ git add RELEASENOTES.md CHANGES.md
+
+ git commit -a -m "Preparing ${PROJECT} release $RELEASE_TAG; tagging and
updates to CHANGES.md and RELEASENOTES.md"
+ echo "Creating tag $RELEASE_TAG at the head of $GIT_BRANCH"
+ git tag "$RELEASE_TAG"
+
+ # Create next version
+ maven_set_version "$NEXT_VERSION"
+
+ git commit -a -m "Preparing development version $NEXT_VERSION"
+
+ if ! is_dry_run; then
+ # Push changes
+ git push origin "$RELEASE_TAG"
+ git push origin "HEAD:$GIT_BRANCH"
+ cd ..
+ rm -rf "${PROJECT}"
+ else
+ cd ..
+ mv "${PROJECT}" "${PROJECT}.tag"
+ echo "Dry run: Clone with version changes and tag available as
${PROJECT}.tag in the output directory."
+ fi
+ exit 0
+fi
+
+### Below is for 'publish-*' stages ###
+check_get_passwords ASF_PASSWORD
+check_needed_vars PROJECT ASF_USERNAME ASF_PASSWORD GPG_KEY
+
+# Commit ref to checkout when building
+BASE_DIR=$(pwd)
+GIT_REF=${GIT_REF:-master}
+if [[ "$PROJECT" =~ ^phoenix ]]; then
+ RELEASE_STAGING_LOCATION="https://dist.apache.org/repos/dist/dev/phoenix";
+else
+ RELEASE_STAGING_LOCATION="https://dist.apache.org/repos/dist/dev/${PROJECT}";
+fi
+
+# in case of dry run, enable publish steps to chain from tag step
+if is_dry_run && [[ "${TAG_SAME_DRY_RUN:-}" == "true" && -d "${PROJECT}.tag"
]]; then
+ ln -s "${PROJECT}.tag" "${PROJECT}"
+else
+ git_clone_overwrite
+fi
+cd "${PROJECT}"
+git checkout "$GIT_REF"
+git_hash="$(git rev-parse --short HEAD)"
+echo "Checked out ${PROJECT} at ${GIT_REF} commit $git_hash"
+
+if [ -z "${RELEASE_VERSION}" ]; then
+ RELEASE_VERSION="$(maven_get_version)"
+fi
+
+# This is a band-aid fix to avoid the failure of Maven nightly snapshot in
some Jenkins
+# machines by explicitly calling /usr/sbin/lsof. Please see SPARK-22377 and
the discussion
+# in its pull request.
+LSOF=lsof
+if ! hash $LSOF 2>/dev/null; then
+ LSOF=/usr/sbin/lsof
+fi
+
+package_version_name="$RELEASE_TAG"
+if [ -z "$package_version_name" ]; then
+ package_version_name="${RELEASE_VERSION}-$(date +%Y_%m_%d_%H_%M)-${git_hash}"
+fi
+
+git clean -d -f -x
+cd ..
+
+if [[ "$1" == "publish-dist" ]]; then
+ # Source and binary tarballs
+ echo "Packaging release source tarballs"
+ make_src_release "${PROJECT}" "${RELEASE_VERSION}"
+
+ # we do not have binary tarballs for phoenix-thirdparty
+ if [[ "${PROJECT}" != "phoenix-thirdparty" ]]; then
+ make_binary_release "${PROJECT}" "${RELEASE_VERSION}"
+ fi
+
+ DEST_DIR_NAME="${PROJECT}-${package_version_name}"
+ svn_target="svn-${PROJECT}"
+ svn co --depth=empty "$RELEASE_STAGING_LOCATION" "$svn_target"
+ rm -rf "${svn_target:?}/${DEST_DIR_NAME}"
+ mkdir -p "$svn_target/${DEST_DIR_NAME}"
+
+ echo "Copying release tarballs"
+ cp "${PROJECT}"-*.tar.* "$svn_target/${DEST_DIR_NAME}/"
+ cp "${PROJECT}/CHANGES.md" "$svn_target/${DEST_DIR_NAME}/"
+ cp "${PROJECT}/RELEASENOTES.md" "$svn_target/${DEST_DIR_NAME}/"
+ shopt -s nocasematch
+ # Generate api report only if project is hbase for now. - Not for Phoenix
+ if [ "${PROJECT}" == "hbase" ]; then
+ # This script usually reports an errcode along w/ the report.
+ generate_api_report "./${PROJECT}" "${API_DIFF_TAG}" "${GIT_REF}" || true
+ cp api*.html "$svn_target/${DEST_DIR_NAME}/"
+ fi
+ shopt -u nocasematch
+
+ svn add "$svn_target/${DEST_DIR_NAME}"
+
+ if ! is_dry_run; then
+ cd "$svn_target"
+ svn ci --username "$ASF_USERNAME" --password "$ASF_PASSWORD" -m"Apache
${PROJECT} $package_version_name" --no-auth-cache
+ cd ..
+ rm -rf "$svn_target"
+ else
+ mv "$svn_target/${DEST_DIR_NAME}" "${svn_target}_${DEST_DIR_NAME}.dist"
+ echo "Dry run: svn-managed 'dist' directory with release tarballs,
CHANGES.md and RELEASENOTES.md available as
$(pwd)/${svn_target}_${DEST_DIR_NAME}.dist"
+ rm -rf "$svn_target"
+ fi
+
+ exit 0
+fi
+
+if [[ "$1" == "publish-snapshot" ]]; then
+ (
+ cd "${PROJECT}"
+ mvn_log="${BASE_DIR}/mvn_deploy_snapshot.log"
+ echo "Publishing snapshot to nexus"
+ maven_deploy snapshot "$mvn_log"
+ if ! is_dry_run; then
+ echo "Snapshot artifacts successfully published to repo."
+ rm "$mvn_log"
+ else
+ echo "Dry run: Snapshot artifacts successfully built, but not published
due to dry run."
+ fi
+ )
+ exit $?
+fi
+
+if [[ "$1" == "publish-release" ]]; then
+ (
+ cd "${PROJECT}"
+ mvn_log="${BASE_DIR}/mvn_deploy_release.log"
+ echo "Staging release in nexus"
+ maven_deploy release "$mvn_log"
+ declare staged_repo_id="dryrun-no-repo"
+ if ! is_dry_run; then
+ staged_repo_id=$(grep -o "Closing staging repository with ID .*"
"$mvn_log" \
+ | sed -e 's/Closing staging repository with ID "\([^"]*\)"./\1/')
+ echo "Release artifacts successfully published to repo ${staged_repo_id}"
+ rm "$mvn_log"
+ else
+ echo "Dry run: Release artifacts successfully built, but not published due
to dry run."
+ fi
+ # Dump out email to send. Where we find vote.tmpl depends
+ # on where this script is run from
+ PROJECT_TEXT="${PROJECT//-/ }" #substitute like 's/-/ /g'
+ export PROJECT_TEXT
+ eval "echo \"$(< "${SELF}/vote.tmpl")\"" |tee "${BASE_DIR}/vote.txt"
+ )
+ exit $?
+fi
+
+set +x # done with detailed logging
+cd ..
+rm -rf "${PROJECT}"
+echo "ERROR: expects to be called with 'tag', 'publish-dist',
'publish-release', or 'publish-snapshot'" >&2
+exit_with_usage
diff --git a/dev/create-release/release-util.sh
b/dev/create-release/release-util.sh
new file mode 100755
index 0000000..442f375
--- /dev/null
+++ b/dev/create-release/release-util.sh
@@ -0,0 +1,707 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+DRY_RUN=${DRY_RUN:-1} #default to dry run
+DEBUG=${DEBUG:-0}
+GPG=${GPG:-gpg}
+GPG_ARGS=(--no-autostart --batch)
+if [ -n "${GPG_KEY}" ]; then
+ GPG_ARGS=("${GPG_ARGS[@]}" --local-user "${GPG_KEY}")
+fi
+# Maven Profiles for publishing snapshots and release to Maven Central and Dist
+PUBLISH_PROFILES=("-P" "apache-release,release")
+
+set -e
+
+function error {
+ echo "Error: $*" >&2
+ exit 1
+}
+
+function read_config {
+ local PROMPT="$1"
+ local DEFAULT="$2"
+ local REPLY=
+
+ read -r -p "$PROMPT [$DEFAULT]: " REPLY
+ local RETVAL="${REPLY:-$DEFAULT}"
+ if [ -z "$RETVAL" ]; then
+ error "$PROMPT must be provided."
+ fi
+ echo "$RETVAL"
+}
+
+function parse_version {
+ # Heuristics to skip the parent version
+ grep -e '<version>.*\..*</version>' | \
+ head -n 1 | tail -n 1 | cut -d'>' -f2 | cut -d '<' -f1
+}
+
+function banner {
+ local msg="$1"
+ echo "========================"
+ echo "=== ${msg}"
+ echo
+}
+
+# current number of seconds since epoch
+function get_ctime {
+ date +"%s"
+}
+
+function run_silent {
+ local BANNER="$1"
+ local LOG_FILE="$2"
+ shift 2
+ local -i start_time
+ local -i stop_time
+
+ banner "${BANNER}"
+ echo "Command: $*"
+ echo "Log file: $LOG_FILE"
+ start_time="$(get_ctime)"
+
+ if ! "$@" 1>"$LOG_FILE" 2>&1; then
+ echo "Command FAILED. Check full logs for details."
+ tail "$LOG_FILE"
+ exit 1
+ fi
+ stop_time="$(get_ctime)"
+ echo "=== SUCCESS ($((stop_time - start_time)) seconds)"
+}
+
+function fcreate_secure {
+ local FPATH="$1"
+ rm -f "$FPATH"
+ touch "$FPATH"
+ chmod 600 "$FPATH"
+}
+
+# API compare version.
+function get_api_diff_version {
+ local version="$1"
+ local rev
+ local api_diff_tag
+ rev=$(echo "$version" | cut -d . -f 3)
+ if [ "$rev" != 0 ]; then
+ local short_version
+ short_version="$(echo "$version" | cut -d . -f 1-2)"
+ api_diff_tag="rel/${short_version}.$((rev - 1))"
+ else
+ local major minor
+ major="$(echo "$version" | cut -d . -f 1)"
+ minor="$(echo "$version" | cut -d . -f 2)"
+ if [ "$minor" != 0 ]; then
+ api_diff_tag="rel/${major}.$((minor - 1)).0"
+ else
+ api_diff_tag="rel/$((major - 1)).0.0"
+ fi
+ fi
+ api_diff_tag="$(read_config "api_diff_tag" "$api_diff_tag")"
+ echo "$api_diff_tag"
+}
+
+# Get all branches that begin with 'branch-', the hbase convention for
+# release branches, sort them and then pop off the most recent.
+function get_release_info {
+ PROJECT="$(read_config "PROJECT" "$PROJECT")"
+ export PROJECT
+
+ if [[ -z "${ASF_REPO}" ]]; then
+ ASF_REPO="https://gitbox.apache.org/repos/asf/${PROJECT}.git";
+ fi
+ if [[ -z "${ASF_REPO_WEBUI}" ]]; then
+ ASF_REPO_WEBUI="https://gitbox.apache.org/repos/asf?p=${PROJECT}.git";
+ fi
+ if [[ -z "${ASF_GITHUB_REPO}" ]]; then
+ ASF_GITHUB_REPO="https://github.com/apache/${PROJECT}";
+ fi
+ if [ -z "$GIT_BRANCH" ]; then
+ # If no branch is specified, find out the latest branch from the repo.
+ GIT_BRANCH="$(git ls-remote --heads "$ASF_REPO" |
+ grep refs/heads/branch- |
+ awk '{print $2}' |
+ sort -r |
+ head -n 1 |
+ cut -d/ -f3)"
+ fi
+
+ GIT_BRANCH="$(read_config "GIT_BRANCH" "$GIT_BRANCH")"
+ export GIT_BRANCH
+
+ # Find the current version for the branch.
+ # FIXME this only works with gitbox
+ local version
+ version="$(curl -s
"$ASF_REPO_WEBUI;a=blob_plain;f=pom.xml;hb=refs/heads/$GIT_BRANCH" |
+ parse_version)"
+ echo "Current branch VERSION is $version."
+
+ NEXT_VERSION="$version"
+ RELEASE_VERSION=""
+ SHORT_VERSION="$(echo "$version" | cut -d . -f 1-2)"
+ if [[ ! "$version" =~ .*-SNAPSHOT ]]; then
+ RELEASE_VERSION="$version"
+ else
+ RELEASE_VERSION="${version/-SNAPSHOT/}"
+ fi
+
+ local REV
+ REV="$(echo "${RELEASE_VERSION}" | cut -d . -f 3)"
+
+ # Find out what RC is being prepared.
+ # - If the current version is "x.y.0", then this is RC0 of the "x.y.0"
release.
+ # - If not, need to check whether the previous version has been already
released or not.
+ # - If it has, then we're building RC0 of the current version.
+ # - If it has not, we're building the next RC of the previous version.
+ local RC_COUNT
+ if [ "$REV" != 0 ]; then
+ local PREV_REL_REV=$((REV - 1))
+ PREV_REL_TAG="rel/${SHORT_VERSION}.${PREV_REL_REV}"
+ if git ls-remote --tags "$ASF_REPO" "$PREV_REL_TAG" | grep -q
"refs/tags/${PREV_REL_TAG}$" ; then
+ RC_COUNT=0
+ REV=$((REV + 1))
+ NEXT_VERSION="${SHORT_VERSION}.${REV}-SNAPSHOT"
+ else
+ RELEASE_VERSION="${SHORT_VERSION}.${PREV_REL_REV}"
+ RC_COUNT="$(git ls-remote --tags "$ASF_REPO" "${RELEASE_VERSION}RC*" |
wc -l)"
+ # This makes a 'number' of it.
+ RC_COUNT=$((RC_COUNT))
+ fi
+ else
+ REV=$((REV + 1))
+ NEXT_VERSION="${SHORT_VERSION}.${REV}-SNAPSHOT"
+ RC_COUNT=0
+ fi
+
+ RELEASE_VERSION="$(read_config "RELEASE_VERSION" "$RELEASE_VERSION")"
+ NEXT_VERSION="$(read_config "NEXT_VERSION" "$NEXT_VERSION")"
+ export RELEASE_VERSION NEXT_VERSION
+
+ RC_COUNT="$(read_config "RC_COUNT" "$RC_COUNT")"
+ RELEASE_TAG="${RELEASE_VERSION}RC${RC_COUNT}"
+ RELEASE_TAG="$(read_config "RELEASE_TAG" "$RELEASE_TAG")"
+
+ # Check if the RC already exists, and if re-creating the RC, skip tag
creation.
+ SKIP_TAG=0
+ if git ls-remote --tags "$ASF_REPO" "$RELEASE_TAG" | grep -q
"refs/tags/${RELEASE_TAG}$" ; then
+ read -r -p "$RELEASE_TAG already exists. Continue anyway [y/n]? " ANSWER
+ if [ "$ANSWER" != "y" ]; then
+ echo "Exiting."
+ exit 1
+ fi
+ SKIP_TAG=1
+ fi
+
+ export RELEASE_TAG SKIP_TAG
+
+ GIT_REF="$RELEASE_TAG"
+ if is_dry_run; then
+ echo "This is a dry run. If tag does not actually exist, please confirm
the ref that will be built for testing."
+ GIT_REF="$(read_config "GIT_REF" "$GIT_REF")"
+ fi
+ export GIT_REF
+
+ if [ "${PROJECT}" == "hbase" ]; then
+ API_DIFF_TAG="$(get_api_diff_version "$RELEASE_VERSION")"
+ fi
+
+ # Gather some user information.
+ ASF_USERNAME="$(read_config "ASF_USERNAME" "$LOGNAME")"
+
+ GIT_NAME="$(git config user.name || echo "")"
+ GIT_NAME="$(read_config "GIT_NAME" "$GIT_NAME")"
+
+ GIT_EMAIL="$asf_usern...@apache.org"
+ GPG_KEY="$(read_config "GPG_KEY" "$GIT_EMAIL")"
+ if ! GPG_KEY_ID=$("${GPG}" "${GPG_ARGS[@]}" --keyid-format 0xshort
--list-public-key "${GPG_KEY}" | grep "\[S\]" | grep -o "0x[0-9A-F]*") ||
+ [ -z "${GPG_KEY_ID}" ] ; then
+ GPG_KEY_ID=$("${GPG}" "${GPG_ARGS[@]}" --keyid-format 0xshort
--list-public-key "${GPG_KEY}" | head -n 1 | grep -o "0x[0-9A-F]*" || true)
+ fi
+ read -r -p "We think the key '${GPG_KEY}' corresponds to the key id
'${GPG_KEY_ID}'. Is this correct [y/n]? " ANSWER
+ if [ "$ANSWER" = "y" ]; then
+ GPG_KEY="${GPG_KEY_ID}"
+ fi
+ export API_DIFF_TAG ASF_USERNAME GIT_NAME GIT_EMAIL GPG_KEY
+
+ cat <<EOF
+================
+Release details:
+GIT_BRANCH: $GIT_BRANCH
+RELEASE_VERSION: $RELEASE_VERSION
+NEXT_VERSION: $NEXT_VERSION
+RELEASE_TAG: $RELEASE_TAG $([[ "$GIT_REF" != "$RELEASE_TAG" ]] && printf
"\n%s\n" "GIT_REF: $GIT_REF")
+API_DIFF_TAG: $API_DIFF_TAG
+ASF_USERNAME: $ASF_USERNAME
+GPG_KEY: $GPG_KEY
+GIT_NAME: $GIT_NAME
+GIT_EMAIL: $GIT_EMAIL
+DRY_RUN: $(is_dry_run && echo "yes" || echo "NO, THIS BUILD WILL BE
PUBLISHED!")
+================
+EOF
+
+ read -r -p "Is this info correct [y/n]? " ANSWER
+ if [ "$ANSWER" != "y" ]; then
+ echo "Exiting."
+ exit 1
+ fi
+ GPG_ARGS=("${GPG_ARGS[@]}" --local-user "${GPG_KEY}")
+
+ if ! is_dry_run; then
+ if [ -z "$ASF_PASSWORD" ]; then
+ stty -echo && printf "ASF_PASSWORD: " && read -r ASF_PASSWORD && printf
'\n' && stty echo
+ fi
+ else
+ ASF_PASSWORD="***INVALID***"
+ fi
+
+ export ASF_PASSWORD
+}
+
+function is_dry_run {
+ [[ "$DRY_RUN" = 1 ]]
+}
+
+function is_debug {
+ [[ "${DEBUG}" = 1 ]]
+}
+
+function check_get_passwords {
+ for env in "$@"; do
+ if [ -z "${!env}" ]; then
+ echo "The environment variable $env is not set. Please enter the
password or passphrase."
+ echo
+ # shellcheck disable=SC2229
+ stty -echo && printf "%s : " "$env" && read -r "$env" && printf '\n' &&
stty echo
+ fi
+ # shellcheck disable=SC2163
+ export "$env"
+ done
+}
+
+function check_needed_vars {
+ local missing=0
+ for env in "$@"; do
+ if [ -z "${!env}" ]; then
+ echo "$env must be set to run this script"
+ (( missing++ ))
+ else
+ # shellcheck disable=SC2163
+ export "$env"
+ fi
+ done
+ (( missing > 0 )) && exit_with_usage
+ return 0
+}
+
+function init_locale {
+ local locale_value
+ OS="$(uname -s)"
+ case "${OS}" in
+ Darwin*) locale_value="en_US.UTF-8";;
+ Linux*) locale_value="C.UTF-8";;
+ *) error "unknown OS";;
+ esac
+ export LC_ALL="$locale_value"
+ export LANG="$locale_value"
+}
+
+# Initializes JAVA_VERSION to the version of the JVM in use.
+function init_java {
+ if [ -z "$JAVA_HOME" ]; then
+ error "JAVA_HOME is not set."
+ fi
+ JAVA_VERSION=$("${JAVA_HOME}"/bin/javac -version 2>&1 | cut -d " " -f 2)
+ echo "java version: $JAVA_VERSION"
+ export JAVA_VERSION
+}
+
+function init_python {
+ if ! [ -x "$(command -v python2)" ]; then
+ error 'python2 needed by yetus. Install or add link? E.g: sudo ln -sf
/usr/bin/python2.7 /usr/local/bin/python2'
+ fi
+ echo "python version: $(python2 --version)"
+}
+
+# Set MVN
+function init_mvn {
+ if [ -n "$MAVEN_HOME" ]; then
+ MVN=("${MAVEN_HOME}/bin/mvn")
+ elif [ "$(type -P mvn)" ]; then
+ MVN=(mvn)
+ else
+ error "MAVEN_HOME is not set nor is mvn on the current path."
+ fi
+ # Add batch mode.
+ MVN=("${MVN[@]}" -B)
+ export MVN
+ echo -n "mvn version: "
+ "${MVN[@]}" --version
+ configure_maven
+}
+
+function init_yetus {
+ declare YETUS_VERSION
+ if [ -z "${YETUS_HOME}" ]; then
+ error "Missing Apache Yetus."
+ fi
+ # Work around yetus bug by asking test-patch for the version instead of rdm.
+ YETUS_VERSION=$("${YETUS_HOME}/bin/test-patch" --version)
+ echo "Apache Yetus version ${YETUS_VERSION}"
+}
+
+function configure_maven {
+ # Add timestamps to mvn logs.
+ MAVEN_OPTS="-Dorg.slf4j.simpleLogger.showDateTime=true
-Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss ${MAVEN_OPTS}"
+ # Suppress gobs of "Download from central:" messages
+
MAVEN_OPTS="-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn
${MAVEN_OPTS}"
+ MAVEN_LOCAL_REPO="${REPO:-$(pwd)/$(mktemp -d hbase-repo-XXXXX)}"
+ [[ -d "$MAVEN_LOCAL_REPO" ]] || mkdir -p "$MAVEN_LOCAL_REPO"
+ MAVEN_SETTINGS_FILE="${MAVEN_LOCAL_REPO}/tmp-settings.xml"
+ MVN=("${MVN[@]}" --settings "${MAVEN_SETTINGS_FILE}")
+ export MVN MAVEN_OPTS MAVEN_SETTINGS_FILE MAVEN_LOCAL_REPO
+ export ASF_USERNAME ASF_PASSWORD
+ # reference passwords from env rather than storing in the settings.xml file.
+ cat <<'EOF' > "$MAVEN_SETTINGS_FILE"
+<?xml version="1.0" encoding="UTF-8"?>
+<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
http://maven.apache.org/xsd/settings-1.0.0.xsd";>
+ <localRepository>/${env.MAVEN_LOCAL_REPO}</localRepository>
+ <servers>
+
<server><id>apache.snapshots.https</id><username>${env.ASF_USERNAME}</username>
+ <password>${env.ASF_PASSWORD}</password></server>
+
<server><id>apache.releases.https</id><username>${env.ASF_USERNAME}</username>
+ <password>${env.ASF_PASSWORD}</password></server>
+ </servers>
+ <profiles>
+ <profile>
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <gpg.keyname>${env.GPG_KEY}</gpg.keyname>
+ </properties>
+ </profile>
+ </profiles>
+</settings>
+EOF
+}
+
+# clone of the repo, deleting anything that exists in the working directory
named after the project.
+# optionally with auth details for pushing.
+function git_clone_overwrite {
+ local asf_repo
+ if [ -z "${PROJECT}" ] || [ "${PROJECT}" != "${PROJECT#/}" ]; then
+ error "Project name must be defined and not start with a '/'.
PROJECT='${PROJECT}'"
+ fi
+ rm -rf "${PROJECT}"
+
+ if [[ -z "${GIT_REPO}" ]]; then
+ asf_repo="gitbox.apache.org/repos/asf/${PROJECT}.git"
+ echo "[INFO] clone will be of the gitbox repo for ${PROJECT}."
+ if [ -n "${ASF_USERNAME}" ] && [ -n "${ASF_PASSWORD}" ]; then
+ # Ugly!
+ encoded_username=$(python -c "import urllib; print
urllib.quote('''$ASF_USERNAME''', '')")
+ encoded_password=$(python -c "import urllib; print
urllib.quote('''$ASF_PASSWORD''', '')")
+ GIT_REPO="https://$encoded_username:$encoded_password@${asf_repo}";
+ else
+ GIT_REPO="https://${asf_repo}";
+ fi
+ else
+ echo "[INFO] clone will be of provided git repo."
+ fi
+ # N.B. we use the shared flag because the clone is short lived and if a
local repo repo was
+ # given this will let us refer to objects there directly instead of
hardlinks or copying.
+ # The option is silently ignored for non-local repositories. see the
note on git help clone
+ # for the --shared option for details.
+ git clone --shared -b "${GIT_BRANCH}" -- "${GIT_REPO}" "${PROJECT}"
+ # If this was a host local git repo then add in an alternates and remote
that will
+ # work back on the host if the RM needs to do any post-processing steps,
i.e. pushing the git tag
+ # for more info see 'git help remote' and 'git help repository-layout'.
+ if [ -n "$HOST_GIT_REPO" ]; then
+ echo "${HOST_GIT_REPO}/objects" >>
"${PROJECT}/.git/objects/info/alternates"
+ (cd "${PROJECT}"; git remote add host "${HOST_GIT_REPO}")
+ fi
+}
+
+function start_step {
+ local name=$1
+ if [ -z "${name}" ]; then
+ name="${FUNCNAME[1]}"
+ fi
+ echo "$(date -u +'%Y-%m-%dT%H:%M:%SZ') ${name} start" >&2
+ get_ctime
+}
+
+function stop_step {
+ local name=$2
+ local start_time=$1
+ local stop_time
+ if [ -z "${name}" ]; then
+ name="${FUNCNAME[1]}"
+ fi
+ stop_time="$(get_ctime)"
+ echo "$(date -u +'%Y-%m-%dT%H:%M:%SZ') ${name} stop ($((stop_time -
start_time)) seconds)"
+}
+
+# Writes report into cwd!
+# TODO should have option for maintenance release that include LimitedPrivate
in report
+function generate_api_report {
+ local project="$1"
+ local previous_tag="$2"
+ local release_tag="$3"
+ local previous_version
+ local timing_token
+ timing_token="$(start_step)"
+ # Generate api report.
+ "${project}"/dev-support/checkcompatibility.py --annotation \
+ org.apache.yetus.audience.InterfaceAudience.Public \
+ "$previous_tag" "$release_tag"
+ previous_version="$(echo "${previous_tag}" | sed -e 's/rel\///')"
+ cp "${project}/target/compat-check/report.html"
"./api_compare_${previous_version}_to_${release_tag}.html"
+ stop_step "${timing_token}"
+}
+
+# Look up the Jira name associated with project.
+# Currently all the 'hbase-*' projects share the same HBASE jira name. This
works because,
+# by convention, the HBASE jira "Fix Version" field values have the
sub-project name pre-pended,
+# as in "hbase-operator-tools-1.0.0".
+# TODO: For non-hbase-related projects, enhance this to use Jira API query
instead of text lookup.
+function get_jira_name {
+ local project="$1"
+ local jira_name
+ case "${project}" in
+ phoenix-tephra) jira_name="TEPHRA";;
+ phoenix-omid) jira_name="OMID";;
+ phoenix*) jira_name="PHOENIX";;
+ *) jira_name="";;
+ esac
+ if [[ -z "$jira_name" ]]; then
+ error "Sorry, can't determine the Jira name for project $project"
+ fi
+ echo "$jira_name"
+}
+
+# Update the CHANGES.md
+# DOES NOT DO COMMITS! Caller should do that.
+# requires yetus to have a defined home already.
+# yetus requires python2 to be on the path.
+function update_releasenotes {
+ local project_dir="$1"
+ local jira_fix_version="$2"
+ local jira_project
+ local timing_token
+ timing_token="$(start_step)"
+ jira_project="$(get_jira_name "$(basename "$project_dir")")"
+ "${YETUS_HOME}/bin/releasedocmaker" -p "${jira_project}" --fileversions -v
"${jira_fix_version}" \
+ -l --sortorder=newer --skip-credits
+ pwd
+ # First clear out the changes written by previous RCs.
+ if [ -f "${project_dir}/CHANGES.md" ]; then
+ sed -i -e \
+ "/^## Release ${jira_fix_version}/,/^## Release/ {//!d; /^## Release
${jira_fix_version}/d;}" \
+ "${project_dir}/CHANGES.md" || true
+ fi
+ if [ -f "${project_dir}/RELEASENOTES.md" ]; then
+ sed -i -e \
+ "/^# ${jira_project} ${jira_fix_version} Release Notes/,/^#
${jira_project}/{//!d; /^# ${jira_project} ${jira_fix_version} Release
Notes/d;}" \
+ "${project_dir}/RELEASENOTES.md" || true
+ fi
+
+ # The releasedocmaker call above generates RELEASENOTES.X.X.X.md and
CHANGELOG.X.X.X.md.
+ if [ -f "${project_dir}/CHANGES.md" ]; then
+ # To insert into project's CHANGES.md...need to cut the top off the
+ # CHANGELOG.X.X.X.md file removing license and first line and then
+ # insert it after the license comment closing where we have a
+ # DO NOT REMOVE marker text!
+ sed -i -e '/## Release/,$!d' "CHANGELOG.${jira_fix_version}.md"
+ sed -i -e "/DO NOT REMOVE/r CHANGELOG.${jira_fix_version}.md"
"${project_dir}/CHANGES.md"
+ else
+ mv "CHANGELOG.${jira_fix_version}.md" "${project_dir}/CHANGES.md"
+ fi
+ if [ -f "${project_dir}/RELEASENOTES.md" ]; then
+ # Similar for RELEASENOTES but slightly different.
+ sed -i -e '/Release Notes/,$!d' "RELEASENOTES.${jira_fix_version}.md"
+ sed -i -e "/DO NOT REMOVE/r RELEASENOTES.${jira_fix_version}.md" \
+ "${project_dir}/RELEASENOTES.md"
+ else
+ mv "RELEASENOTES.${jira_fix_version}.md" "${project_dir}/RELEASENOTES.md"
+ fi
+ stop_step "${timing_token}"
+}
+
+# Make src release.
+# Takes as arguments first the project name -- e.g. hbase or
hbase-operator-tools
+# -- and then the version string. Expects to find checkout adjacent to this
script
+# named for 'project', the first arg passed.
+# Expects the following three defines in the environment:
+# - GPG needs to be defined, with the path to GPG: defaults 'gpg'.
+# - GIT_REF which is the tag to create the tgz from: defaults to 'master'.
+# For example:
+# $ GIT_REF="master" make_src_release hbase-operator-tools 1.0.0
+make_src_release() {
+ # Tar up the src and sign and hash it.
+ local project="${1}"
+ local version="${2}"
+ local base_name="${project}-${version}"
+ local timing_token
+ timing_token="$(start_step)"
+ rm -rf "${base_name}"-src*
+ tgz="${base_name}-src.tar.gz"
+ cd "${project}" || exit
+ git clean -d -f -x
+ git archive --format=tar.gz --output="../${tgz}" --prefix="${base_name}/"
"${GIT_REF:-master}"
+ cd .. || exit
+ $GPG "${GPG_ARGS[@]}" --armor --output "${tgz}.asc" --detach-sig "${tgz}"
+ $GPG "${GPG_ARGS[@]}" --print-md SHA512 "${tgz}" > "${tgz}.sha512"
+ stop_step "${timing_token}"
+}
+
+# Make binary release.
+# Takes as arguments first the project name -- e.g. hbase or
hbase-operator-tools
+# -- and then the version string. Expects to find checkout adjacent to this
script
+# named for 'project', the first arg passed.
+# Expects the following three defines in the environment:
+# - GPG needs to be defined, with the path to GPG: defaults 'gpg'.
+# - GIT_REF which is the tag to create the tgz from: defaults to 'master'.
+# - MVN Default is "mvn -B --settings $MAVEN_SETTINGS_FILE".
+# For example:
+# $ GIT_REF="master" make_src_release hbase-operator-tools 1.0.0
+make_binary_release() {
+ local project="${1}"
+ local version="${2}"
+ local base_name="${project}-${version}"
+ local timing_token
+ timing_token="$(start_step)"
+ rm -rf "${base_name}"-bin*
+ cd "$project" || exit
+
+ git clean -d -f -x
+ # Three invocations of maven. This seems to work. One to
+ # populate the repo, another to build the site, and then
+ # a third to assemble the binary artifact. Trying to do
+ # all in the one invocation fails; a problem in our
+ # assembly spec to in maven. TODO. Meantime, three invocations.
+ #"${MVN[@]}" clean install -DskipTests
+ #"${MVN[@]}" site -DskipTests
+ kick_gpg_agent
+ "${MVN[@]}" clean package -DskipTests -Dcheckstyle.skip=true
"${PUBLISH_PROFILES[@]}"
+
+ # Check there is a bin gz output. The build may not produce one: e.g.
hbase-thirdparty.
+ local f_bin_prefix="./${PROJECT}-assembly/target/${base_name}"
+ if ls "${f_bin_prefix}"*-bin.tar.gz &>/dev/null; then
+ cp "${f_bin_prefix}"*-bin.tar.gz ..
+ cd .. || exit
+ for i in "${base_name}"*-bin.tar.gz; do
+ "${GPG}" "${GPG_ARGS[@]}" --armour --output "${i}.asc" --detach-sig
"${i}"
+ "${GPG}" "${GPG_ARGS[@]}" --print-md SHA512 "${i}" > "${i}.sha512"
+ done
+ else
+ cd .. || exit
+ echo "No ${f_bin_prefix}*-bin.tar.gz product; expected?"
+ fi
+
+ stop_step "${timing_token}"
+}
+
+# "Wake up" the gpg agent so it responds properly to maven-gpg-plugin, and
doesn't cause timeout.
+# Specifically this is done between invocation of 'mvn site' and 'mvn
assembly:single', because
+# the 'site' build takes long enough that the gpg-agent does become
unresponsive and the following
+# 'assembly' build (where gpg signing occurs) experiences timeout, without
this "kick".
+function kick_gpg_agent {
+ # All that's needed is to run gpg on a random file
+ # TODO could we just call gpg-connect-agent /bye
+ local i
+ i="$(mktemp)"
+ echo "This is a test file" > "$i"
+ "${GPG}" "${GPG_ARGS[@]}" --armour --output "${i}.asc" --detach-sig "${i}"
+ rm "$i" "$i.asc"
+}
+
+# Do maven command to set version into local pom
+function maven_set_version { #input: <version_to_set>
+ local this_version="$1"
+ echo "${MVN[@]}" versions:set -DnewVersion="$this_version"
+ "${MVN[@]}" versions:set -DnewVersion="$this_version" | grep -v "no value" #
silence logs
+ #Hacking around nonstandard maven submodules
+ for i in phoenix-hbase-compat-*; do
+ if [ -e "$i" ]; then
+ "${MVN[@]}" -pl $i versions:set -DnewVersion="$this_version" | grep -v
"no value" # silence logs
+ fi
+ done
+}
+
+# Do maven command to read version from local pom
+function maven_get_version {
+ # shellcheck disable=SC2016
+ "${MVN[@]}" -q -N -Dexec.executable="echo" -Dexec.args='${project.version}'
exec:exec
+}
+
+# Do maven deploy to snapshot or release artifact repository, with checks.
+function maven_deploy { #inputs: <snapshot|release> <log_file_path>
+ local timing_token
+ # Invoke with cwd=$PROJECT
+ local deploy_type="$1"
+ local mvn_log_file="$2" #secondary log file used later to extract
staged_repo_id
+ if [[ "$deploy_type" != "snapshot" && "$deploy_type" != "release" ]]; then
+ error "unrecognized deploy type, must be 'snapshot'|'release'"
+ fi
+ if [[ -z "$mvn_log_file" ]] || ! touch "$mvn_log_file"; then
+ error "must provide writable maven log output filepath"
+ fi
+ timing_token=$(start_step)
+ # shellcheck disable=SC2153
+ if [[ "$deploy_type" == "snapshot" ]] && ! [[ "$RELEASE_VERSION" =~
-SNAPSHOT$ ]]; then
+ error "Snapshots must have a version with suffix '-SNAPSHOT'; you gave
version '$RELEASE_VERSION'"
+ elif [[ "$deploy_type" == "release" ]] && [[ "$RELEASE_VERSION" =~ SNAPSHOT
]]; then
+ error "Non-snapshot release version must not include the word 'SNAPSHOT';
you gave version '$RELEASE_VERSION'"
+ fi
+ # Publish ${PROJECT} to Maven repo
+ # shellcheck disable=SC2154
+ echo "Publishing ${PROJECT} checkout at '$GIT_REF' ($git_hash)"
+ echo "Publish version is $RELEASE_VERSION"
+ # Coerce the requested version
+ maven_set_version "$RELEASE_VERSION"
+ # Prepare for signing
+ kick_gpg_agent
+ declare -a mvn_goals=(clean install)
+ if ! is_dry_run; then
+ mvn_goals=("${mvn_goals[@]}" deploy)
+ fi
+ echo "${MVN[@]}" -DskipTests -Dcheckstyle.skip=true "${PUBLISH_PROFILES[@]}"
\
+ "${mvn_goals[@]}"
+ echo "Logging to ${mvn_log_file}. This will take a while..."
+ rm -f "$mvn_log_file"
+ # The tortuous redirect in the next command allows mvn's stdout and stderr
to go to mvn_log_file,
+ # while also sending stderr back to the caller.
+ # shellcheck disable=SC2094
+ if ! "${MVN[@]}" -DskipTests -Dcheckstyle.skip=true "${PUBLISH_PROFILES[@]}"
\
+ "${mvn_goals[@]}" 1>> "$mvn_log_file" 2> >( tee -a "$mvn_log_file" >&2
); then
+ error "Deploy build failed, for details see log at '$mvn_log_file'."
+ fi
+ echo "BUILD SUCCESS."
+ stop_step "${timing_token}"
+ return 0
+}
+
+# guess the host os
+# * DARWIN
+# * LINUX
+function get_host_os() {
+ uname -s | tr '[:lower:]' '[:upper:]'
+}
diff --git a/dev/create-release/vote.tmpl b/dev/create-release/vote.tmpl
new file mode 100644
index 0000000..8f0638b
--- /dev/null
+++ b/dev/create-release/vote.tmpl
@@ -0,0 +1,31 @@
+Please vote on this Apache ${PROJECT_TEXT} release candidate,
+${PROJECT}-${RELEASE_TAG}
+
+The VOTE will remain open for at least 72 hours.
+
+[ ] +1 Release this package as Apache ${PROJECT_TEXT} ${RELEASE_VERSION}
+[ ] -1 Do not release this package because ...
+
+The tag to be voted on is ${RELEASE_TAG}:
+
+ https://github.com/apache/${PROJECT}/tree/${RELEASE_TAG}
+
+The release files, including signatures, digests, as well as CHANGES.md
+and RELEASENOTES.md included in this RC can be found at:
+
+ https://dist.apache.org/repos/dist/dev/phoenix/${RELEASE_TAG}/
+
+Maven artifacts are available in a staging repository at:
+
+ https://repository.apache.org/content/repositories/${staged_repo_id}/
+
+Artifacts were signed with the ${GPG_KEY} key which can be found in:
+
+ https://dist.apache.org/repos/dist/release/phoenix/KEYS
+
+To learn more about Apache ${PROJECT_TEXT}, please see
+
+ http://phoenix.apache.org/
+
+Thanks,
+Your Phoenix Release Manager
diff --git a/phoenix-assembly/pom.xml b/phoenix-assembly/pom.xml
index 0a88f97..2e7b6a1 100644
--- a/phoenix-assembly/pom.xml
+++ b/phoenix-assembly/pom.xml
@@ -107,7 +107,7 @@
<goal>single</goal>
</goals>
<configuration>
- <finalName>phoenix-${project.version}</finalName>
+ <finalName>phoenix-${project.version}-bin</finalName>
<attach>false</attach>
<tarLongFileMode>gnu</tarLongFileMode>
<appendAssemblyId>false</appendAssemblyId>
@@ -117,23 +117,6 @@
<tarLongFileMode>posix</tarLongFileMode>
</configuration>
</execution>
- <execution>
- <id>package-to-source-tar</id>
- <phase>package</phase>
- <goals>
- <goal>single</goal>
- </goals>
- <configuration>
- <finalName>phoenix-${project.version}-source</finalName>
- <attach>false</attach>
- <tarLongFileMode>gnu</tarLongFileMode>
- <appendAssemblyId>false</appendAssemblyId>
- <descriptors>
- <descriptor>src/build/src.xml</descriptor>
- </descriptors>
- <tarLongFileMode>posix</tarLongFileMode>
- </configuration>
- </execution>
</executions>
</plugin>
</plugins>
diff --git a/phoenix-assembly/src/build/components/all-common-files.xml
b/phoenix-assembly/src/build/components/all-common-files.xml
index a17a138..9348397 100644
--- a/phoenix-assembly/src/build/components/all-common-files.xml
+++ b/phoenix-assembly/src/build/components/all-common-files.xml
@@ -30,6 +30,13 @@
<include>*.txt</include>
</includes>
</fileSet>
+ <!-- Copy the binary license files -->
+ <fileSet>
+ <directory>${project.basedir}/../dev/release_files</directory>
+ <fileMode>0644</fileMode>
+ <directoryMode>0755</directoryMode>
+ <outputDirectory>/</outputDirectory>
+ </fileSet>
<!-- Copy the executable files from the bin directory -->
<fileSet>
<directory>${project.basedir}/../bin</directory>
@@ -77,19 +84,14 @@
<fileMode>0644</fileMode>
<directoryMode>0755</directoryMode>
</fileSet>
- <fileSet>
- <directory>${project.basedir}/../phoenix-pherf/target</directory>
- <outputDirectory>phoenix-pherf</outputDirectory>
- <includes>
- <include>*.zip</include>
- </includes>
- <fileMode>0644</fileMode>
- <directoryMode>0755</directoryMode>
- </fileSet>
<fileSet>
- <directory>${project.basedir}/../python</directory>
- <fileMode>0644</fileMode>
- <directoryMode>0755</directoryMode>
+ <directory>${project.basedir}/../phoenix-pherf/target</directory>
+ <outputDirectory>phoenix-pherf</outputDirectory>
+ <includes>
+ <include>*.zip</include>
+ </includes>
+ <fileMode>0644</fileMode>
+ <directoryMode>0755</directoryMode>
</fileSet>
</fileSets>
</component>
diff --git a/phoenix-assembly/src/build/src.xml
b/phoenix-assembly/src/build/src.xml
deleted file mode 100644
index cfa73d6..0000000
--- a/phoenix-assembly/src/build/src.xml
+++ /dev/null
@@ -1,127 +0,0 @@
-<!--
-
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
--->
-<assembly
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0";
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
-
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0
http://maven.apache.org/xsd/assembly-1.1.0.xsd";>
- <!--This 'all' id is not appended to the produced bundle because we do this:
http://maven.apache.org/plugins/maven-assembly-plugin/faq.html#required-classifiers
-->
- <id>src</id>
- <formats>
- <format>tar.gz</format>
- </formats>
- <includeBaseDirectory>true</includeBaseDirectory>
-
- <moduleSets>
- <moduleSet>
- <!-- Enable access to all projects in the current multimodule build.
Eclipse
- says this is an error, but builds from the command line just fine. -->
- <useAllReactorProjects>true</useAllReactorProjects>
- <!-- Include all the sources in the top directory -->
- <sources>
- <fileSets>
- <fileSet>
- <!-- Make sure this excludes is same as the phoenix-hadoop2-compat
- excludes below -->
- <excludes>
- <exclude>target/</exclude>
- <exclude>test/</exclude>
- <exclude>.classpath</exclude>
- <exclude>.project</exclude>
- <exclude>.settings/</exclude>
- </excludes>
- </fileSet>
- </fileSets>
- </sources>
- </moduleSet>
- </moduleSets>
-
- <fileSets>
- <!--This one is weird. When we assemble src, it'll be default profile
which
- at the moment is hadoop1. But we should include the hadoop2 compat
module
- too so can build hadoop2 from src -->
- <fileSet>
- <directory>${project.basedir}/..</directory>
- <fileMode>0644</fileMode>
- <directoryMode>0755</directoryMode>
- <includes>
- <include>phoenix-*</include>
- </includes>
- <excludes>
- <exclude>target/</exclude>
- <exclude>test/</exclude>
- <exclude>.classpath</exclude>
- <exclude>.project</exclude>
- <exclude>.settings/</exclude>
- </excludes>
- </fileSet>
- <fileSet>
- <!--Get misc project files -->
- <directory>${project.basedir}/..</directory>
- <outputDirectory>/</outputDirectory>
- <includes>
- <include>*.txt</include>
- <include>*.md</include>
- <include>pom.xml</include>
- </includes>
- </fileSet>
- <!-- Top level directories -->
- <fileSet>
- <directory>${project.basedir}/../bin</directory>
- <outputDirectory>bin</outputDirectory>
- <fileMode>0755</fileMode>
- <directoryMode>0755</directoryMode>
- <includes>
- <include>*.py</include>
- <include>*.sh</include>
- </includes>
- </fileSet>
- <fileSet>
- <directory>${project.basedir}/../bin</directory>
- <outputDirectory>bin</outputDirectory>
- <fileMode>0644</fileMode>
- <directoryMode>0755</directoryMode>
- <excludes>
- <exclude>*.py*</exclude>
- <exclude>*.sh*</exclude>
- </excludes>
- </fileSet>
- <fileSet>
- <directory>${project.basedir}/../dev</directory>
- <fileMode>0644</fileMode>
- <directoryMode>0755</directoryMode>
- </fileSet>
- <fileSet>
- <directory>${project.basedir}/../docs</directory>
- <fileMode>0644</fileMode>
- <directoryMode>0755</directoryMode>
- </fileSet>
- <fileSet>
- <directory>${project.basedir}/../examples</directory>
- <fileMode>0644</fileMode>
- <directoryMode>0755</directoryMode>
- </fileSet>
- <fileSet>
- <directory>${project.basedir}/../python</directory>
- <fileMode>0644</fileMode>
- <directoryMode>0755</directoryMode>
- </fileSet>
- </fileSets>
-
-</assembly>
diff --git a/pom.xml b/pom.xml
index 658c98d..aa6e874 100644
--- a/pom.xml
+++ b/pom.xml
@@ -578,6 +578,9 @@
<excludes>
<!-- Header on changelog isn't normal -->
<exclude>CHANGES</exclude>
+ <!-- create-release scripts -->
+ <exclude>dev/create-release/README.txt</exclude>
+ <exclude>dev/create-release/vote.tmpl</exclude>
<!-- IDE configuration -->
<exclude>dev/phoenix.importorder</exclude>
<!-- Release L&N -->
