> The general concept is one of traceable and repeatable builds. In the context of jenkins.buildacloud.org I would disagree with this. Only two build jobs are ever kept. When a time arises when you need to figure out what commit your systemvm was based of I'd be willing to bet that job has been cleared out of jenkins ages ago.
> In RPM land, the jenkins $BUILD_NUMBER should go into the RPM Release field and not into the RPM Version field. But in this case we don’t have RPM, or spec files Isn't this what we use /etc/cloudstack-release for? You could include extra information in there if you wish, version, branch, git sha1, etc. which would give great traceability. The supplied download-template script is dependant on having a local nexus/artifactory that can do searching on published build artifacts. In the context of jenkins.buildacloud.org the system vms are not uploaded to nexus/artifactory and just kept on the jenkins box. The RPM one grabs a zip of all RPM is the last successful build. The zip filename doesn't contain any version information. Changing this causes 404s for people example: https://mail-archives.apache.org/mod_mbox/cloudstack-users/201409.mbox/browser On 24 September 2014 07:37, Leo Simons <lsim...@schubergphilis.com> wrote: > Hey Ian, > > On Sep 24, 2014, at 2:04 AM, Ian Duffy <i...@ianduffy.ie> wrote: > > I noticed the filename of the generated systemvms changed on > > http://jenkins.buildacloud.org/job/build-systemvm64-master/ > > > > They now include a [0-9]* before the hypervisor name representing the > build > > number. > > That’s one of the things coming out of the 1290e10 merge. > > > Why do we do this? Its annoying for external resources linking to the > last > > successful build. > > The general concept is one of traceable and repeatable builds. > > In RPM terms, given just a random binary RPM found on a random system > * it should be possible to unambiguously determine the source RPM from > which it was built > * it should be possible to determine who or what built it, and when > * it should be possible to trace the source RPM back to the exact version > control revision > * it should be possible to determine easily whether that binary is the > same as > another one on another machine / in another repository that has the same > identifiers > > In RPM land, the jenkins $BUILD_NUMBER should go into the RPM Release > field and not into the RPM Version field. But in this case we don’t have > RPM, or spec files, so there is only the filename to accomplish the same > (which is why the branch name is also in there), and it has to be in this > spot or cloudstack gets a bit confused. > > What we do in our integration environment is to apply just a bit of logic > to parse out the version number from the file name. I.e. to get an RPM from > jenkins see fetch-rpms.sh included below, or to fetch the most recent > systemvm from a directory listing see download-template.sh also included > below. Would something like that work for you? > > If not, the logic for creating the version is in build.sh: > > version_tag= > if [ ! -z "${version}" ]; then > if [ ! -z "${BUILD_NUMBER}" ]; then > version="${version}.${BUILD_NUMBER}" > fi > version_tag="-${version}" > elif [ ! -z "${BUILD_NUMBER}" ]; then > version="${BUILD_NUMBER}" > version_tag="-${BUILD_NUMBER}” > fi > > So we can change that, or we can unset BUILD_NUMBER in the jenkins build > prior to invocation. What do you think? > > It would be much better to exclusively use yum/maven repositories and > their logic to centralize this kind of logic server-side, or at least to be > creating and updating symlinks to ‘latest’. That’s a work in progress... > > > cheers, > > > Leo > > download-templates.sh > ===================== > #!/bin/bash > # > # Licensed to the Apache Software Foundation (ASF) under one > # or more contributor license agreements. See the NOTICE file > # distributed with this work for additional information > # regarding copyright ownership. The ASF licenses this file > # to you under the Apache License, Version 2.0 (the > # "License"); you may not use this file except in compliance > # with the License. You may obtain a copy of the License at > # > # http://www.apache.org/licenses/LICENSE-2.0 > # > # Unless required by applicable law or agreed to in writing, > # software distributed under the License is distributed on an > # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY > # KIND, either express or implied. See the License for the > # specific language governing permissions and limitations > # under the License. > > # Download systemvm templates onto secondary storage > > set -e > echo $@ | grep debug >/dev/null && DEBUG="1" > echo $@ | grep trace >/dev/null && TRACE="1" > > function source_common() { > local SOURCE="${BASH_SOURCE[0]}" > DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )" > source "${DIR}/common.sh" > } > source_common > > NFS_HOST="${1:-localhost}" > NFS_SECONDARY_EXPORT="${2:-/storage/secondary}" > MOUNT_PATH="${3:-/mnt/secondary}" > BASE_URL="${4:-https://artifacts.schubergphilis.com/artifacts/cloudstack}"; > TEMPLATE="${5:-systemvmtemplate}" > DB_HOST="${6:-localhost}" > DB_USER="${7:-root}" > DB_PASSWORD="${8:-}" > > function find_template() { > if ! `echo "${TEMPLATE}" | egrep '^[a-zA-Z0-9]+-[0-9]+(\.[0-9]+)*$'`; > then > # (apache) directory listing > # find only links > # filter by template name > # find only xen template > # strip suffix > # take the first (newest) one > newest_first="?C=M;O=D" > TEMPLATE=$(curl "${BASE_URL}/${newest_first}" | \ > egrep -o 'href="[^"]+"' | \ > egrep "${TEMPLATE}" | \ > egrep -o "[^/\"]+-xen\.vhd\.bz2" | \ > sed 's/-xen.vhd.bz2//' |\ > head -n 1) > if [[ "${TEMPLATE}" == "" ]]; then > error "No possible match found for ${TEMPLATE} at ${BASE_URL}" > fi > log INFO "Using template ${TEMPLATE}" > fi > } > > function install_xen_template() { > XEN_URL="${BASE_URL}/${TEMPLATE}-xen.vhd.bz2" > # KVM_URL="${BASE_URL}/${TEMPLATE}-kvm.vhd.bz2" > > [ -d "${MOUNT_PATH}" ] || mkdir "${MOUNT_PATH}" > > mount -t nfs "${NFS_HOST}:${NFS_SECONDARY_EXPORT}" "${MOUNT_PATH}" || > true > > > /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt > \ > -F \ > -m "${MOUNT_PATH}" \ > -h "xenserver" \ > -u "${XEN_URL}" \ > -o "${DB_HOST}" \ > -r "${DB_USER}" \ > -d "${DB_PASSWORD}" > } > > function main() { > find_template > install_xen_template > } > > # we only run main() if not source-d > return 2>/dev/null || main > > > > fetch-rpms.sh > ============= > #!/bin/bash > # > # Licensed to the Apache Software Foundation (ASF) under one > # or more contributor license agreements. See the NOTICE file > # distributed with this work for additional information > # regarding copyright ownership. The ASF licenses this file > # to you under the Apache License, Version 2.0 (the > # "License"); you may not use this file except in compliance > # with the License. You may obtain a copy of the License at > # > # http://www.apache.org/licenses/LICENSE-2.0 > # > # Unless required by applicable law or agreed to in writing, > # software distributed under the License is distributed on an > # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY > # KIND, either express or implied. See the License for the > # specific language governing permissions and limitations > # under the License. > > # script to install http://jenkins.buildacloud.org/ RPMs to > # the local repo > > function usage() { > cat <<END > Usage: > ./fetch-rpms.sh [archiveUrl] > > * Pass [archiveURL] to link to the build artifacts > (default is > http://jenkins.buildacloud.org/job/ > package-rhel63-master/ > lastSuccessfulBuild/artifact/*zip* > /archive.zip > ) > * Set \$DEBUG=1 to enable debug logging > * Set \$TRACE=1 to enable trace logging > END > exit 0 > } > > echo $@ | grep help >/dev/null && usage > echo $@ | grep '\-h' >/dev/null && usage > echo $@ | grep debug >/dev/null && DEBUG="1" > echo $@ | grep trace >/dev/null && TRACE="1" > > function source_common() { > # from http://stackoverflow.com/a/246128 > local SOURCE="${BASH_SOURCE[0]}" > local DIR="" > > while [ -h "$SOURCE" ]; do > # resolve $SOURCE until the file is no longer a symlink > DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )" > SOURCE="$(readlink "${SOURCE}")" > # if $SOURCE was a relative symlink, we need to resolve it relative > # to the path where the symlink file was located > [[ ${SOURCE} != /* ]] && SOURCE="${DIR}/${SOURCE}" > done > DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )" > > source "${DIR}/common.sh" > } > source_common > > ### > ### Configuration > ### > > if [[ "$1" != "" ]]; then > ARCHIVE_URL="${1}" > else > JENKINS="http://jenkins.buildacloud.org"; > JOB="package-rhel63-master" > > ARCHIVE_URL="${JENKINS}/job/${JOB}/lastSuccessfulBuild/artifact/*zip*/archive.zip" > fi > STORAGE_DIR="${BASEDIR}/../storage" > ARCHIVE_FILE="${STORAGE_DIR}/archive.zip" > > ### > ### Script logic > ### > > function prepare() { > cd "${TARGET_DIR}" > init_settings > } > > function run() { > local do_download=1 > if [[ -f "${ARCHIVE_FILE}" ]]; then > if [[ "${CI_RUN}" == "1" ]]; then > log WARN "${ARCHIVE_FILE} already exists, removing" > rm -f "${ARCHIVE_FILE}" > else > log INFO "${ARCHIVE_FILE} already exists, reusing" > do_download=0 > fi > fi > if [[ ${do_download} -eq 1 ]]; then > log INFO "Downloading ${ARCHIVE_URL}" > curl -o "${ARCHIVE_FILE}" "${ARCHIVE_URL}" > fi > > if [[ -d "${STORAGE_DIR}/archive" ]]; then > rm -rf "${STORAGE_DIR}/archive" > fi > mkdir "${STORAGE_DIR}/archive" > cd "${STORAGE_DIR}" > unzip -o "${ARCHIVE_FILE}" > mkdir -p "${STORAGE_DIR}/rpmrepo/cloudstack-${CLOUDSTACK_VERSION}" > local numfiles=$(find \ > ${STORAGE_DIR}/archive \ > -name "cloudstack*${CLOUDSTACK_VERSION}*.rpm" | \ > wc -l | awk '{print $1}') > if [[ ${numfiles} -lt 4 ]]; then > error "Expecting at least 4 rpms matching > cloudstack*${CLOUDSTACK_VERSION}*.rpm, but got only ${numfiles}" > fi > find ${STORAGE_DIR}/archive -name > "cloudstack*${CLOUDSTACK_VERSION}*.rpm" -exec \ > mv \{\} "${STORAGE_DIR}/rpmrepo/cloudstack-${CLOUDSTACK_VERSION}/" \; > rm -rf "${STORAGE_DIR}/archive" > } > > # we only run main() if not source-d > return 2>/dev/null || main > >
