Repository: cassandra-builds Updated Branches: refs/heads/master 81ab99ead -> f3fdb2bd9
Unify version handling for docker based package building Build either snapshot or release packages based on specified git path. Releases will only be build from tags. Provided changes will simplify manual invocation of docker for developers, as well as making it possible to automatically build packages in Jenkins at some point. Closes #3 Project: http://git-wip-us.apache.org/repos/asf/cassandra-builds/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra-builds/commit/f3fdb2bd Tree: http://git-wip-us.apache.org/repos/asf/cassandra-builds/tree/f3fdb2bd Diff: http://git-wip-us.apache.org/repos/asf/cassandra-builds/diff/f3fdb2bd Branch: refs/heads/master Commit: f3fdb2bd94b0128a90fabec6809d7397fa9df8b5 Parents: 81ab99e Author: Stefan Podkowinski <[email protected]> Authored: Tue May 16 20:20:11 2017 +0200 Committer: Stefan Podkowinski <[email protected]> Committed: Tue May 16 22:57:55 2017 +0200 ---------------------------------------------------------------------- README.md | 11 +++- build-scripts/cassandra-deb-packaging.sh | 16 ++++++ build-scripts/cassandra-rpm-packaging.sh | 17 ++++++ docker/build-debs.sh | 75 ++++++++++++++++++++++++--- docker/build-rpms.sh | 60 ++++++++++++++++++--- docker/centos7-image.docker | 3 +- docker/jessie-image.docker | 3 +- 7 files changed, 169 insertions(+), 16 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/f3fdb2bd/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index f483b17..3b77fdd 100644 --- a/README.md +++ b/README.md @@ -15,14 +15,21 @@ ```docker build -f docker/jessie-image.docker docker/``` * RPM: ```docker build -f docker/centos7-image.docker docker/``` + The image will contain a clone of the Apache git repository by default. Using a different repository is possible by adding the `--build-arg CASSANDRA_GIT_URL=https://github.com/myuser/cassandra.git` parameter. All successive builds will be executed based on the repository cloned during docker image creation. 2. Run build script through docker (specify branch, e.g. cassandra-3.0 and version, e.g. 3.0.11): * Debian: - ```docker run -v `pwd`/dist:/dist `docker images -f label=org.cassandra.buildenv=jessie -q` /home/build/build-debs.sh <branch>``` + ```docker run --rm -v `pwd`/dist:/dist `docker images -f label=org.cassandra.buildenv=jessie -q` /home/build/build-debs.sh <branch/tag>``` * RPM: - ```docker run -v `pwd`/dist:/dist `docker images -f label=org.cassandra.buildenv=centos -q` /home/build/build-rpms.sh <branch> <version>``` + ```docker run --rm -v `pwd`/dist:/dist `docker images -f label=org.cassandra.buildenv=centos -q` /home/build/build-rpms.sh <branch/tag>``` You should find newly created Debian and RPM packages in the `dist` directory. +### Note about versioning + +Packages for official releases can only be build from tags. In this case, the tag must match the known versioning scheme. A number of sanity checks will be run to make sure the version matches any version defined in `build.xml` and `debian/changes`. But you'll have to manually keep these values in sync for every release. + +Builds based on any branch will use the version defined in either `build.xml` (RPM) or `debian/changes` (deb). Afterwards a snapshot indicator will be appended. + ## Publishing packages TODO http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/f3fdb2bd/build-scripts/cassandra-deb-packaging.sh ---------------------------------------------------------------------- diff --git a/build-scripts/cassandra-deb-packaging.sh b/build-scripts/cassandra-deb-packaging.sh new file mode 100755 index 0000000..4466ced --- /dev/null +++ b/build-scripts/cassandra-deb-packaging.sh @@ -0,0 +1,16 @@ +#!/bin/bash -xe + +CASSANDRA_BUILDS_DIR="${WORKSPACE}/cassandra-builds" +CASSANDRA_GIT_URL=$1 +CASSANDRA_BRANCH=$2 + +# Create build images containing the build tool-chain, Java and an Apache Cassandra git working directory +docker build --build-arg CASSANDRA_GIT_URL=$CASSANDRA_GIT_URL -f ${CASSANDRA_BUILDS_DIR}/docker/jessie-image.docker ${CASSANDRA_BUILDS_DIR}/docker/ + +# Create target directory for packages generated in docker run below +mkdir -p ${CASSANDRA_BUILDS_DIR}/dist +chmod 777 ${CASSANDRA_BUILDS_DIR}/dist + +# Run build script through docker (specify branch, e.g. cassandra-3.0 and version, e.g. 3.0.11): +docker run --rm -v ${CASSANDRA_BUILDS_DIR}/dist:/dist `docker images -f label=org.cassandra.buildenv=jessie -q | awk 'NR==1'` /home/build/build-debs.sh $CASSANDRA_BRANCH + http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/f3fdb2bd/build-scripts/cassandra-rpm-packaging.sh ---------------------------------------------------------------------- diff --git a/build-scripts/cassandra-rpm-packaging.sh b/build-scripts/cassandra-rpm-packaging.sh new file mode 100755 index 0000000..747d781 --- /dev/null +++ b/build-scripts/cassandra-rpm-packaging.sh @@ -0,0 +1,17 @@ +#!/bin/bash -xe + +CASSANDRA_BUILDS_DIR="${WORKSPACE}/cassandra-builds" +CASSANDRA_GIT_URL=$1 +CASSANDRA_BRANCH=$2 +CASSANDRA_VERSION=$3 + +# Create build images containing the build tool-chain, Java and an Apache Cassandra git working directory +docker build --build-arg CASSANDRA_GIT_URL=$CASSANDRA_GIT_URL -f ${CASSANDRA_BUILDS_DIR}/docker/centos7-image.docker ${CASSANDRA_BUILDS_DIR}/docker/ + +# Create target directory for packages generated in docker run below +mkdir -p ${CASSANDRA_BUILDS_DIR}/dist +chmod 777 ${CASSANDRA_BUILDS_DIR}/dist + +# Run build script through docker (specify branch, e.g. cassandra-3.0 and version, e.g. 3.0.11): +docker run --rm -v ${CASSANDRA_BUILDS_DIR}/dist:/dist `docker images -f label=org.cassandra.buildenv=centos -q | awk 'NR==1'` /home/build/build-rpms.sh $CASSANDRA_BRANCH $CASSANDRA_VERSION + http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/f3fdb2bd/docker/build-debs.sh ---------------------------------------------------------------------- diff --git a/docker/build-debs.sh b/docker/build-debs.sh index 9d55deb..39bd688 100755 --- a/docker/build-debs.sh +++ b/docker/build-debs.sh @@ -1,8 +1,8 @@ #!/bin/bash -x -set -eu +set -e if [ "$#" -ne 1 ]; then - echo "build-debs.sh branch" + echo "build-debs.sh <branch>" exit 1 fi @@ -10,10 +10,73 @@ CASSANDRA_BRANCH=$1 cd $CASSANDRA_DIR git fetch -git checkout $CASSANDRA_BRANCH -# javadoc target is broken in docker without this mkdir -mkdir -p ./build/javadoc -ant artifacts -Drelease=true +git checkout $CASSANDRA_BRANCH || exit 1 + +# Used version for build will always depend on the git referenced used for checkout above +# Branches will always be created as snapshots, while tags are releases +tag=`git describe --tags --exact-match` 2> /dev/null +branch=`git symbolic-ref -q --short HEAD` 2> /dev/null + +is_tag=false +is_branch=false +git_version='' + +if [ "$tag" ]; then + # Official release + is_tag=true + regx_tag="cassandra-([0-9.]+)(-tentative)?$" + if [[ $tag =~ $regx_tag ]]; then + git_version=${BASH_REMATCH[1]} + else + echo "Error: could not recognize version from tag $tag">&2 + exit 2 + fi +elif [ "$branch" ]; then + # Dev branch + is_branch=true + regx_branch="cassandra-([0-9.]+)$" + if [[ $branch =~ $regx_branch ]]; then + git_version=${BASH_REMATCH[1]} + else + # This could be either trunk or any dev branch, so we won't be able to get the version + # from the branch name. In this case, fall back to debian change log version. + git_version=$(dpkg-parsechangelog | sed -ne 's/^Version: \([^-|~|+]*\).*/\1/p') + if [ -z $git_version ]; then + echo "Error: could not recognize version from branch $branch">&2 + exit 2 + else + echo "Warning: could not recognize version from branch, using dpkg version: $git_version" + fi + fi +else + echo "Error: invalid git reference; must either be branch or tag">&2 + exit 1 +fi + +# The version used for the deb build process will the current version in the debian/changelog file. +# See debian/rules for how the value is read. The only thing left for us to do here is to check if +# the changes file contains the correct version for the checked out git revision. The version value +# has to be updated manually by a committer and we only warn and abort on mismatches here. +changelog_version=$(dpkg-parsechangelog | sed -ne 's/^Version: \([^~|+]*\).*/\1/p') +chl_expected="$git_version" +if $is_branch ; then + chl_expected="${git_version}[0-9.]*-SNAPSHOT" +fi +if [[ ! $changelog_version =~ $chl_expected ]]; then + echo "Error: changelog version seems to be missing -SNAPSHOT suffix for branch">&2 + exit 3 +fi + +# Version (base.version) in build.xml must be set manually as well. Let's validate the set value. +buildxml_version=`grep 'property\s*name="base.version"' build.xml |sed -ne 's/.*value="\([^"]*\)".*/\1/p'` +if [ $buildxml_version != $git_version ]; then + echo "Error: build.xml version ($buildxml_version) not matching git tag derived version ($git_version)">&2 + exit 4 +fi + +# Install build dependencies and build package echo "y" | sudo mk-build-deps --install dpkg-buildpackage -uc -us + +# Copy created artifacts to dist dir mapped to docker host directory (must have proper permissions) cp ../cassandra[-_]* $DEB_DIST_DIR http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/f3fdb2bd/docker/build-rpms.sh ---------------------------------------------------------------------- diff --git a/docker/build-rpms.sh b/docker/build-rpms.sh index abd875c..3bfd69e 100755 --- a/docker/build-rpms.sh +++ b/docker/build-rpms.sh @@ -1,20 +1,68 @@ #!/bin/bash -x -set -eu +#set -eu -if [ "$#" -ne 2 ]; then - echo "build-rpms.sh branch version" +if [ "$#" -ne 1 ]; then + echo "build-rpms.sh branch" exit 1 fi CASSANDRA_BRANCH=$1 -CASSANDRA_VERSION=$2 cd $CASSANDRA_DIR git fetch -git checkout $CASSANDRA_BRANCH +git checkout $CASSANDRA_BRANCH || exit 1 + +# Used version for build will always depend on the git referenced used for checkout above +# Branches will always be created as snapshots, while tags are releases +tag=`git describe --tags --exact-match` 2> /dev/null +branch=`git symbolic-ref -q --short HEAD` 2> /dev/null + +is_tag=false +is_branch=false +git_version='' + +# Parse version from build.xml so we can verify version against release tags and use the build.xml version +# for any branches. Truncate from snapshot suffix if needed. +buildxml_version=`grep 'property\s*name="base.version"' build.xml |sed -ne 's/.*value="\([^"]*\)".*/\1/p'` +regx_snapshot="([0-9.]+)-SNAPSHOT$" +if [[ $buildxml_version =~ $regx_snapshot ]]; then + buildxml_version=${BASH_REMATCH[1]} +fi + +if [ "$tag" ]; then + # Official release + is_tag=true + regx_tag="cassandra-([0-9.]+)(-tentative)?$" + if [[ $tag =~ $regx_tag ]]; then + git_version=${BASH_REMATCH[1]} + else + echo "Error: could not recognize version from tag $tag">&2 + exit 2 + fi + if [ $buildxml_version != $git_version ]; then + echo "Error: build.xml version ($buildxml_version) not matching git tag derived version ($git_version)">&2 + exit 4 + fi + CASSANDRA_VERSION=$git_version + CASSANDRA_REVISION='1%{?dist}' +elif [ "$branch" ]; then + # Dev branch + is_branch=true + # This could be either trunk or any dev branch, so we won't be able to get the version + # from the branch name. In this case, fall back to version specified in build.xml. + CASSANDRA_VERSION="${buildxml_version}" + dt=`date +"%Y%m%d"` + ref=`git rev-parse --short HEAD` + CASSANDRA_REVISION="${dt}git${ref}%{?dist}" +else + echo "Error: invalid git reference; must either be branch or tag">&2 + exit 1 +fi + # javadoc target is broken in docker without this mkdir mkdir -p ./build/javadoc +# Artifact will only be used internally for build process and won't be found with snapshot suffix ant artifacts -Drelease=true cp ./build/apache-cassandra-*-src.tar.gz ${RPM_BUILD_DIR}/SOURCES/ -rpmbuild --define="version ${CASSANDRA_VERSION}" -ba ./redhat/cassandra.spec +rpmbuild --define="version ${CASSANDRA_VERSION}" --define="revision ${CASSANDRA_REVISION}" -ba ./redhat/cassandra.spec cp $RPM_BUILD_DIR/SRPMS/*.rpm $RPM_BUILD_DIR/RPMS/noarch/*.rpm $RPM_DIST_DIR http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/f3fdb2bd/docker/centos7-image.docker ---------------------------------------------------------------------- diff --git a/docker/centos7-image.docker b/docker/centos7-image.docker index fc82d4d..c082939 100644 --- a/docker/centos7-image.docker +++ b/docker/centos7-image.docker @@ -40,7 +40,8 @@ USER build RUN mkdir -p $RPM_BUILD_DIR/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS} # Clone Cassandra and cache maven artifacts -RUN git clone https://git.apache.org/cassandra.git ${CASSANDRA_DIR} +ARG CASSANDRA_GIT_URL=https://git.apache.org/cassandra.git +RUN git clone ${CASSANDRA_GIT_URL} ${CASSANDRA_DIR} WORKDIR $CASSANDRA_DIR RUN ant maven-ant-tasks-retrieve-build http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/f3fdb2bd/docker/jessie-image.docker ---------------------------------------------------------------------- diff --git a/docker/jessie-image.docker b/docker/jessie-image.docker index 80bce88..b21e671 100644 --- a/docker/jessie-image.docker +++ b/docker/jessie-image.docker @@ -35,7 +35,8 @@ RUN echo "build ALL=(root) NOPASSWD:ALL" > /etc/sudoers.d/build && \ USER build # clone Cassandra and cache maven artifacts -RUN git clone https://git.apache.org/cassandra.git ${CASSANDRA_DIR} +ARG CASSANDRA_GIT_URL=https://git.apache.org/cassandra.git +RUN git clone ${CASSANDRA_GIT_URL} ${CASSANDRA_DIR} WORKDIR ${CASSANDRA_DIR} RUN ant maven-ant-tasks-retrieve-build --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
