On 20/08/10 09:47, Martin Pool wrote: > Along with maxb and garyvdm I've been updating the bzr beta ppa for > our 2.2 release. This is not absolutely the most fun thing I've ever > done; in fact it's pretty tedious.
Hi Martin :-) > It's the kind of tedious work that doesn't require much creative input > but is not very automated, seems to have a bunch of snags that make it > not trivially automatable, and with long latency, so you can't just > sit down and do it then know you're done. It's about 10 minutes of > actual thinking spread over a couple of days of spade work. I spent 3 hours last night in which I made 29 uploads to the bzr/proposed PPA - I didn't find it particularly tedious, and the most time consuming parts were wrestling the dulwich testsuite into working with older git versions in older distroseries, and familiarizing myself with the packaging branches in use. I think the critical reason I have a different viewpoint on the tedium-level involved is that I already _have_ automated the repetitive bits, and I'm strongly familiar with the general packaging process. > We have about ten packages to rebuild, and we want to do do that > across four distroseries (hardy, jaunty, karmic, lucid, maverick). So > anything that happens once per package build is quite a lot of work; > and there's a fair amount of human-in-the-loop latency per package. > Doing them all seems to be multiple days work, even with some amount > of automation. Ideally we would update on every release but to be > able to do that we have to scale better. > > Most of what we do is just add a rebuild line into the debian > changelog, push, and upload, but things occasionally go wrong per > package. This should be very automatable. A lot of the packages I touched weren't using packaging branches at all - ironically, this actually made my life easier, when I could simply take the package from maverick/sid, apply boilerplate debian/changelog tweaks, and upload. Indeed, I scripted this (which was easy for me since it was a trivial derivation of a script I use for other PPAs). The script is attached, for the curious, but basically it does the following: Take a .dsc file as input. Unpack the .dsc. Use the dch tool to tweak the changelog. Pack up the modified source, and generate a .changes file. Optionally sign it. Optionally upload it. So with all that, I actually had a single-line invocation to take the debian sid version of a package sitting on my local disk, and upload it to the PPA, for a given series. Of course, I swiftly ran into the problem of source format 3.0 - so I enhanced the script to automatically transform source format 3.0 to source format 1.0 when targeting jaunty or below. Now, certainly, this isn't a magic bullet - it's highly dependent on the Debian sid source package building without modifications in all the target series - but for a lot of bzr plugins, that's the case. > It can be a bit hard to work out which packaging branch is used for > the version in our ppa. Launchpad has a concept of series branches > for We can have a convention for plugins that we package, but it's > just a convention and because it's evolved over time it's not always > consistent. For instance if Gary has uploaded bzr-gtk - > 0.99.0-1~hardy4 but it's failing, I can get the source for that > package and try to fix it, but I can't easily work out what branch > contains that source. (Well, I can guess it's one of the recently > updated branches in <https://code.edge.launchpad.net/bzr-gtk> but it's > not great.) I think this just requires that the project set a convention and use it. I have a thread on the bazaar ML proposing we do so. > Some external dependencies of bzr (like subvertpy, testtools, and > subunit) are packaged into other ppas. To make the install experience > better we copy all the dependency packages into our ppa. This works > pretty well. It does introduce another possible level of > inconsistency or confusion though: not everything in our ppa will have > a branch in the same place, and not everything is meant to be updated > in the same way. (bzr packages you should directly upload; subvertpy > you should prefer to copy from somewhere else.) Well, interesting that you should mention that. For subvertpy and dulwich, we're now going with the solution of building them directly in the bazaar PPAs, as Jelmer has advised that he doesn't intend that those PPAs aim to package the latest release across all distroseries with any promptness. I'm inclined to suggest we do the same for testtools and subunit, especially given that the versioning of those other PPAs seems to be violating the best practice of not making PPA version strings that look like ones that would be in the main archive. > Maverick's bzr has its copy of python-configobj cut out, but that > library is not shipped on hardy. Dealing with things like this make > me wonder a lot whether it's a good use of time to keep supporting > many old platforms. Definitely a concern, but in this case a simple backport of the python-configobj package ought to sort matters. > The lag seems to come in mostly through Soyuz delays: there's a delay > before your package is accepted, then a delay that was running at > several hours yesterday before it's actually built or rejected. I do > realize that there's a lot of demand for Soyuz build machines and that > people are working on improving scheduling and performance. But still > it does mean there's this long period where you have mental > work-in-progress, and can't wholeheartedly move on to something else. > If there's a mistake either in your work, or the scripts, you have to > page in the state you had a while ago. The long lag is pretty > unreasonable when bzr has barely any compilation steps and builds in > well under a minute on a laptop. This is absolutely and definitely a problem. For example, last night I uploaded dulwich packages, which FTBFS due to a silly limitation in the bit of sbuild that processes build-deps. And I had to wait until the next day to find out about this, and then wait another day for my fixed packages to build. > One approach to the lag in soyuz builds is to test build every package > locally, before uploading. This seems like a bit of an unnecessary > kludge, but perhaps it's the most sensible thing for now. Since we > want to build on various different distroseries we have to use some > kind of virtual environment. There are various options including > pbuilder, but none that I've found are really plug-and-play. I have a > wrapper script passed on by Robert that gives it some better defaults. > I think this doesn't yet take into account that some build > dependencies might need to come from either the destination or other > ppas, but we could possibly add that. I'm not sure it's a kludge, so much as a totally sensible part of preparing an upload. However, given I keep a full set of cowbuilder chroots updated for all active distroseries anyway, what could be of great annoyance to many, is fairly trivial to me. Furthermore, there's the gotcha of what to do when you want to build something - say bzrtools - which depends on another package in a ppa - like the bzr of the right version. In my case my .pbuilderrc contains magic so that I pass a ppas=bzr/proposed envvar, pass --override-config to the cowbuilder invocation, and it Just Does It. Again, a case of having the tooling already available saving the day. > It seems like we have to keep track of which builds have passed or > failed by collecting a set of nearly-identical mails. There's a > "latest updates" portlet in eg > <https://edge.launchpad.net/~bzr/+archive/proposed> but it only shows > the last 5 which is not really enough (http://pad.lv/620903). https://edge.launchpad.net/~bzr/+archive/proposed/+packages is fit for the purpose. > Perhaps we could standardize a way to do local test builds in a way as > close as possible to what Launchpad does, including how to get > bzr-builddeb to use it for a a test build before uploading. pbuilder (well, cowbuilder, which makes pbuilder fast enough to be pleasant) does a very nice job most of the time. sbuild (what's actually used on the buildds) is also an option, if a little trickier to set up, and requiring LVM for the kind of copy-on-write behaviour that cowbuilder gives. (Though I hear sbuild has recently added union-fs support.) > Our process at the moment is to upload everything into > <https://launchpad.net/~bzr/+archive/+proposed> before then promoting > it to the regular archive, so that we don't break dependencies between > packages. (Uploading a bzr newer than the plugins support might make > them uninstallable, and unlike the main archive(?) ppas don't > themselves check for this.) Nope, the main archive has no more magic here. > I think the basic approach of using a > staging archive is ok. We ought to be able to script something that > does both the check and the promotion if it succeeds. edos-debcheck would be something to look into for the check part. launchpadlib scripts can easily accomplish the promotion part. > I discovered that you can copy binaries from maverick into lucid, > assuming it's a pure python package that doesn't need to be rebuilt. > This makes things a little easier. We could add a tool to hydrazine > to do it. This always makes me a little nervous. Yes, it'll often work for the development release to the next-older release. But, it's quite hard to reliably spot when it'll stop working. For this reason, I tend to favour a rebuild anyway. Max.
#!/bin/sh
set -e
usage() {
echo >&2 "usage: $0 [options] file.dsc"
echo >&2 " -s Sign"
echo >&2 " -u DPUT Upload"
echo >&2 " -f Force dput"
echo >&2 " -D SERIES Distroseries (REQUIRED)"
exit 2
}
sign=
dput=
forcedput=
series=
while getopts su:fD: OPTION; do
case $OPTION in
s) sign=1;;
u) dput=$OPTARG;;
f) forcedput=1;;
D) series=$OPTARG;;
\?) usage;;
esac
done
shift $(($OPTIND - 1))
if [ $# -ne 1 ] || [ -z "$series" ]; then
usage
fi
dscfile=$1
shift
tmpdir="tmp.backport.$series.$$"
# Extract
dpkg-source -x "$dscfile" "$tmpdir"
# Tweak changelog
cd "$tmpdir"
dch --distributor not-ubuntu -D "$series" -b -l "~bazaar1~$series" "Rebuild in
PPA for $series."
# Trivial source format downgrade magic
if [ -f debian/source/format ]; then
case $series in
hardy|jaunty)
if [ -d debian/patches ]; then
echo "AUTOMATIC REVERT TO SOURCE FORMAT 1.0, PATCHES BUNDLED AS
diff.gz"
dch -a "Reverted to source format 1.0, all of debian/patches/
bundles as the .diff.gz."
rm -r .pc debian/patches
rm debian/source/format
else
echo "AUTOMATIC REVERT TO SOURCE FORMAT 1.0"
dch -a "Trivially reverted to source format 1.0."
rm debian/source/format
fi
;;
esac
fi
cd ..
# Package up again
dpkg-source -i -b "$tmpdir"
# Generate a .changes
cd "$tmpdir"
package=`dpkg-parsechangelog | sed -nr 's/^Source: //p'`
version=`dpkg-parsechangelog | sed -nr 's/^Version: //p'`
dpkg-genchanges -S -sd > "../${package}_${version}_source.changes"
cd ..
# Clean up
rm -rf "$tmpdir"
if [ "$sign" ]; then
debsign "${package}_${version}_source.changes"
fi
if [ "$dput" ]; then
dput ${forcedput+-f} "$dput" "${package}_${version}_source.changes"
fi
signature.asc
Description: OpenPGP digital signature
-- ubuntu-distributed-devel mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
