Re: SystemVM file name changes
Did this. Next builds _should_ be back to their old urls. Great! Thanks Leo. :) On 25 September 2014 00:04, Leo Simons lsim...@schubergphilis.com wrote: On Sep 24, 2014, at 5:33 PM, Ian Duffy i...@ianduffy.ie wrote: Aren’t they published to S3 somewhere? It’d probably be better if users fetch systemvms from S3…… Both not a clue what the S3 urls are though. Based on http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201402.mbox/%3c3f6b7b36-32a2-4f11-aeff-36a400d67...@gmail.com%3E It ought to be https://s3.amazonaws.com/systemvm-templates.cloudstack.org/master/32/systemvmtemplate-master-xen.vhd.bz2 https://s3.amazonaws.com/systemvm-templates.cloudstack.org/master/64/systemvm64template-master-xen.vhd.bz2 and so forth but I’m getting permission denied. Hugo, any idea? Isn't this what we use /etc/cloudstack-release for? You could include extra information in there if you wish, version, branch, git sha1, etc. which would give great traceability. Sounds like a plan. I’ll see about submitting a patch tonight. The issue is that /etc/cloudstack-release is parsed in a location or two (I forget the details) so it kind-of has to match existing regexes. Grmbl is all I have to say. In the meantime... export BUILD_NUMBER= at the right place in the relevant build job should fix the immediate issue... Did this. Next builds _should_ be back to their old urls. g’night! Leo
Re: SystemVM file name changes
Hey Ian,
On Sep 24, 2014, at 2:04 AM, Ian Duffy i...@ianduffy.ie wrote:
I noticed the filename of the generated systemvms changed on
http://jenkins.buildacloud.org/job/build-systemvm64-master/
They now include a [0-9]* before the hypervisor name representing the build
number.
That’s one of the things coming out of the 1290e10 merge.
Why do we do this? Its annoying for external resources linking to the last
successful build.
The general concept is one of traceable and repeatable builds.
In RPM terms, given just a random binary RPM found on a random system
* it should be possible to unambiguously determine the source RPM from which it
was built
* it should be possible to determine who or what built it, and when
* it should be possible to trace the source RPM back to the exact version
control revision
* it should be possible to determine easily whether that binary is the same as
another one on another machine / in another repository that has the same
identifiers
In RPM land, the jenkins $BUILD_NUMBER should go into the RPM Release field and
not into the RPM Version field. But in this case we don’t have RPM, or spec
files, so there is only the filename to accomplish the same (which is why the
branch name is also in there), and it has to be in this spot or cloudstack gets
a bit confused.
What we do in our integration environment is to apply just a bit of logic to
parse out the version number from the file name. I.e. to get an RPM from
jenkins see fetch-rpms.sh included below, or to fetch the most recent systemvm
from a directory listing see download-template.sh also included below. Would
something like that work for you?
If not, the logic for creating the version is in build.sh:
version_tag=
if [ ! -z ${version} ]; then
if [ ! -z ${BUILD_NUMBER} ]; then
version=${version}.${BUILD_NUMBER}
fi
version_tag=-${version}
elif [ ! -z ${BUILD_NUMBER} ]; then
version=${BUILD_NUMBER}
version_tag=-${BUILD_NUMBER}”
fi
So we can change that, or we can unset BUILD_NUMBER in the jenkins build prior
to invocation. What do you think?
It would be much better to exclusively use yum/maven repositories and their
logic to centralize this kind of logic server-side, or at least to be creating
and updating symlinks to ‘latest’. That’s a work in progress...
cheers,
Leo
download-templates.sh
=
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# License); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# Download systemvm templates onto secondary storage
set -e
echo $@ | grep debug /dev/null DEBUG=1
echo $@ | grep trace /dev/null TRACE=1
function source_common() {
local SOURCE=${BASH_SOURCE[0]}
DIR=$( cd -P $( dirname ${SOURCE} ) pwd )
source ${DIR}/common.sh
}
source_common
NFS_HOST=${1:-localhost}
NFS_SECONDARY_EXPORT=${2:-/storage/secondary}
MOUNT_PATH=${3:-/mnt/secondary}
BASE_URL=${4:-https://artifacts.schubergphilis.com/artifacts/cloudstack};
TEMPLATE=${5:-systemvmtemplate}
DB_HOST=${6:-localhost}
DB_USER=${7:-root}
DB_PASSWORD=${8:-}
function find_template() {
if ! `echo ${TEMPLATE} | egrep '^[a-zA-Z0-9]+-[0-9]+(\.[0-9]+)*$'`; then
# (apache) directory listing
# find only links
# filter by template name
# find only xen template
# strip suffix
# take the first (newest) one
newest_first=?C=M;O=D
TEMPLATE=$(curl ${BASE_URL}/${newest_first} | \
egrep -o 'href=[^]+' | \
egrep ${TEMPLATE} | \
egrep -o [^/\]+-xen\.vhd\.bz2 | \
sed 's/-xen.vhd.bz2//' |\
head -n 1)
if [[ ${TEMPLATE} == ]]; then
error No possible match found for ${TEMPLATE} at ${BASE_URL}
fi
log INFO Using template ${TEMPLATE}
fi
}
function install_xen_template() {
XEN_URL=${BASE_URL}/${TEMPLATE}-xen.vhd.bz2
# KVM_URL=${BASE_URL}/${TEMPLATE}-kvm.vhd.bz2
[ -d ${MOUNT_PATH} ] || mkdir ${MOUNT_PATH}
mount -t nfs ${NFS_HOST}:${NFS_SECONDARY_EXPORT} ${MOUNT_PATH} || true
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt \
-F \
-m ${MOUNT_PATH} \
-h xenserver \
-u ${XEN_URL} \
-o ${DB_HOST} \
-r ${DB_USER} \
-d ${DB_PASSWORD}
}
function main() {
find_template
install_xen_template
}
# we only run main() if not source-d
return 2/dev/null || main
fetch-rpms.sh
Re: SystemVM file name changes
The general concept is one of traceable and repeatable builds.
In the context of jenkins.buildacloud.org I would disagree with this. Only
two build jobs are ever kept. When a time arises when you need to figure
out what commit your systemvm was based of I'd be willing to bet that job
has been cleared out of jenkins ages ago.
In RPM land, the jenkins $BUILD_NUMBER should go into the RPM Release
field and not into the RPM Version field. But in this case we don’t have
RPM, or spec files
Isn't this what we use /etc/cloudstack-release for?
You could include extra information in there if you wish, version, branch,
git sha1, etc. which would give great traceability.
The supplied download-template script is dependant on having a local
nexus/artifactory that can do searching on published build artifacts. In
the context of jenkins.buildacloud.org the system vms are not uploaded to
nexus/artifactory and just kept on the jenkins box.
The RPM one grabs a zip of all RPM is the last successful build. The zip
filename doesn't contain any version information.
Changing this causes 404s for people example:
https://mail-archives.apache.org/mod_mbox/cloudstack-users/201409.mbox/browser
On 24 September 2014 07:37, Leo Simons lsim...@schubergphilis.com wrote:
Hey Ian,
On Sep 24, 2014, at 2:04 AM, Ian Duffy i...@ianduffy.ie wrote:
I noticed the filename of the generated systemvms changed on
http://jenkins.buildacloud.org/job/build-systemvm64-master/
They now include a [0-9]* before the hypervisor name representing the
build
number.
That’s one of the things coming out of the 1290e10 merge.
Why do we do this? Its annoying for external resources linking to the
last
successful build.
The general concept is one of traceable and repeatable builds.
In RPM terms, given just a random binary RPM found on a random system
* it should be possible to unambiguously determine the source RPM from
which it was built
* it should be possible to determine who or what built it, and when
* it should be possible to trace the source RPM back to the exact version
control revision
* it should be possible to determine easily whether that binary is the
same as
another one on another machine / in another repository that has the same
identifiers
In RPM land, the jenkins $BUILD_NUMBER should go into the RPM Release
field and not into the RPM Version field. But in this case we don’t have
RPM, or spec files, so there is only the filename to accomplish the same
(which is why the branch name is also in there), and it has to be in this
spot or cloudstack gets a bit confused.
What we do in our integration environment is to apply just a bit of logic
to parse out the version number from the file name. I.e. to get an RPM from
jenkins see fetch-rpms.sh included below, or to fetch the most recent
systemvm from a directory listing see download-template.sh also included
below. Would something like that work for you?
If not, the logic for creating the version is in build.sh:
version_tag=
if [ ! -z ${version} ]; then
if [ ! -z ${BUILD_NUMBER} ]; then
version=${version}.${BUILD_NUMBER}
fi
version_tag=-${version}
elif [ ! -z ${BUILD_NUMBER} ]; then
version=${BUILD_NUMBER}
version_tag=-${BUILD_NUMBER}”
fi
So we can change that, or we can unset BUILD_NUMBER in the jenkins build
prior to invocation. What do you think?
It would be much better to exclusively use yum/maven repositories and
their logic to centralize this kind of logic server-side, or at least to be
creating and updating symlinks to ‘latest’. That’s a work in progress...
cheers,
Leo
download-templates.sh
=
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# License); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# Download systemvm templates onto secondary storage
set -e
echo $@ | grep debug /dev/null DEBUG=1
echo $@ | grep trace /dev/null TRACE=1
function source_common() {
local SOURCE=${BASH_SOURCE[0]}
DIR=$( cd -P $( dirname ${SOURCE} ) pwd )
source ${DIR}/common.sh
}
source_common
NFS_HOST=${1:-localhost}
NFS_SECONDARY_EXPORT=${2:-/storage/secondary}
MOUNT_PATH=${3:-/mnt/secondary}
BASE_URL=${4:-https://artifacts.schubergphilis.com/artifacts/cloudstack};
Re: SystemVM file name changes
On Sep 24, 2014, at 11:26 AM, Ian Duffy i...@ianduffy.ie wrote: The general concept is one of traceable and repeatable builds. In the context of jenkins.buildacloud.org I would disagree with this. Only two build jobs are ever kept. When a time arises when you need to figure out what commit your systemvm was based of I'd be willing to bet that job has been cleared out of jenkins ages ago. In RPM land, the jenkins $BUILD_NUMBER should go into the RPM Release field and not into the RPM Version field. But in this case we don’t have RPM, or spec files Isn't this what we use /etc/cloudstack-release for? You could include extra information in there if you wish, version, branch, git sha1, etc. which would give great traceability. Sounds like a plan. I’ll see about submitting a patch tonight. The issue is that /etc/cloudstack-release is parsed in a location or two (I forget the details) so it kind-of has to match existing regexes. The supplied download-template script is dependant on having a local nexus/artifactory that can do searching on published build artifacts. In the context of jenkins.buildacloud.org the system vms are not uploaded to nexus/artifactory and just kept on the jenkins box. Aren’t they published to S3 somewhere? It’d probably be better if users fetch systemvms from S3……though one wonders why users need to fetch systemvms built from master at all… Changing this causes 404s for people example: https://mail-archives.apache.org/mod_mbox/cloudstack-users/201409.mbox/browser Hmm. We can’t have too many people badgering jenkins like this I guess. So let’s improve on that! In the meantime... export BUILD_NUMBER= at the right place in the relevant build job should fix the immediate issue... cheers, Leo
Re: SystemVM file name changes
Aren’t they published to S3 somewhere? It’d probably be better if users fetch systemvms from S3…… Both not a clue what the S3 urls are though. though one wonders why users need to fetch systemvms built from master at all… Ease of access for running from the latest master branch I'd imagine. On 24 September 2014 15:21, Leo Simons lsim...@schubergphilis.com wrote: On Sep 24, 2014, at 11:26 AM, Ian Duffy i...@ianduffy.ie wrote: The general concept is one of traceable and repeatable builds. In the context of jenkins.buildacloud.org I would disagree with this. Only two build jobs are ever kept. When a time arises when you need to figure out what commit your systemvm was based of I'd be willing to bet that job has been cleared out of jenkins ages ago. In RPM land, the jenkins $BUILD_NUMBER should go into the RPM Release field and not into the RPM Version field. But in this case we don’t have RPM, or spec files Isn't this what we use /etc/cloudstack-release for? You could include extra information in there if you wish, version, branch, git sha1, etc. which would give great traceability. Sounds like a plan. I’ll see about submitting a patch tonight. The issue is that /etc/cloudstack-release is parsed in a location or two (I forget the details) so it kind-of has to match existing regexes. The supplied download-template script is dependant on having a local nexus/artifactory that can do searching on published build artifacts. In the context of jenkins.buildacloud.org the system vms are not uploaded to nexus/artifactory and just kept on the jenkins box. Aren’t they published to S3 somewhere? It’d probably be better if users fetch systemvms from S3……though one wonders why users need to fetch systemvms built from master at all… Changing this causes 404s for people example: https://mail-archives.apache.org/mod_mbox/cloudstack-users/201409.mbox/browser Hmm. We can’t have too many people badgering jenkins like this I guess. So let’s improve on that! In the meantime... export BUILD_NUMBER= at the right place in the relevant build job should fix the immediate issue... cheers, Leo
Re: SystemVM file name changes
On Sep 24, 2014, at 5:33 PM, Ian Duffy i...@ianduffy.ie wrote: Aren’t they published to S3 somewhere? It’d probably be better if users fetch systemvms from S3…… Both not a clue what the S3 urls are though. Based on http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201402.mbox/%3c3f6b7b36-32a2-4f11-aeff-36a400d67...@gmail.com%3E It ought to be https://s3.amazonaws.com/systemvm-templates.cloudstack.org/master/32/systemvmtemplate-master-xen.vhd.bz2 https://s3.amazonaws.com/systemvm-templates.cloudstack.org/master/64/systemvm64template-master-xen.vhd.bz2 and so forth but I’m getting permission denied. Hugo, any idea? Isn't this what we use /etc/cloudstack-release for? You could include extra information in there if you wish, version, branch, git sha1, etc. which would give great traceability. Sounds like a plan. I’ll see about submitting a patch tonight. The issue is that /etc/cloudstack-release is parsed in a location or two (I forget the details) so it kind-of has to match existing regexes. Grmbl is all I have to say. In the meantime... export BUILD_NUMBER= at the right place in the relevant build job should fix the immediate issue... Did this. Next builds _should_ be back to their old urls. g’night! Leo
SystemVM file name changes
Hi All, I noticed the filename of the generated systemvms changed on http://jenkins.buildacloud.org/job/build-systemvm64-master/ They now include a [0-9]* before the hypervisor name representing the build number. Why do we do this? Its annoying for external resources linking to the last successful build.
