[OK this took WAY longer than expected. I am stopping here, but I would really appreciate if others could help with comments, extra examples, etc.]
In the past couple of years, I have informally helped a few different people adopt the new workflow[1], but that does not scale well. So, I am putting some of my ideas in a big email to try to document this better, to discuss some problems with the new system, and hopefully to help more people switch to this workflow. I will put notes prefixed with [TEAM], which would really benefit from wider input. [1] https://go-team.pages.debian.net/workflow-changes.html Emanuel & Tong: excuse me for adding some extra stuff here, hopefully it does not confuse you much. I would recommend you follow my suggestions where this text deviates from the published doc, because it will make things easier for you, and we can easily change that later if the team decides something else. === Basics of packaging with the new workflow ========================================= I am writing this to complement the new workflow page published in [1]. So, please read that first, and hopefully this text will get you up and running quickly. Note that there are some fuzzy things there, and there are also problems and I and others have found after a few years of using this workflow. Basic concepts -------------- * we keep unchanged upstream git history in our git repo. Debian packaging is a descendant of that history, merging at every new upstream release. * we don't use upstream tarballs or pristine-tar, those are created from git tags. [TEAM] Michael noted some issues sometimes with tars created directly from git in the document, but this has never happened to me with packages that were not using pristine-tar. Has anybody experienced this? [/TEAM] * many times we need to do repackaging of upstream sources: removing vendored libraries, upstream "debian/" directory, or non-dfsg material. We also do that in branches descending from upstream history. Tags ---- Tags are very important for making things simpler, as many of our tools will use them automatically. If we are packaging version 1.33 of a project, we should have these tags: * upstream/1.33 must point to the exact commit in the upstream repository we are packaging. * upstream/1.33+ds (or +dfsg): if we need to repackage, this must point to the commit after all the repackaging changes are applied, and becomes the relevant tag for tools. The tag without the "+ds+ suffix becomes irrelevant, but is good for documentation. * debian/1.33-1 (or debian/1.33+ds-1): after all the packaging work is done, and only after the resulting package is uploaded to the archive, we use "debcommit -r" to generate this tag automatically. Other tags must NOT be pushed to salsa, so be careful not to use "git push --tags" unless you have pruned other tags first. The git config suggested at the end helps achieve this. Branches -------- The usual branch structure (and hierarchy) is as follows: - upstream +- unvendored (also named repackaged, dfsg, etc) +- debian/sid +- debian/<distribution target> Upstream branches ----------------- The "upstream" branch follows upstream release tags or commit of the snapshot to be packaged. Ideally there should be no commits from us. [TEAM: problem] In the last year we have had many instances where -with uptreams that *do* releases- the release tags couldn't be fast-forwarded, and keeping the "upstream" branch updated required force-pushing (which is very complicated, and requires extra privileges in salsa) or adding merge commits, which defeat the purpose of having a pristine upstream branch. [TEAM: my proposal for consideration of the team] I can think of 3 solutions for this problem: * Drop the upstream branch altogether for upstreams that do release tags, and rely on tags as explained in the previous section. * Un-protect the upstream branches in Salsa and always force-fetch and force-push. * Only use the upstream branch locally and force-fetch as needed, only push tags to Salsa. For this introduction, I will ignore this problem, as it only affects a small number of packages. [/TEAM] The "unvendored"/"repackaged"/etc branch contains the changes to the upstream codebase before it is packaged. On each new upstream release or snapshot, the upstream branch is merged onto it, and any new modifications applied. [TEAM: problem] Another issue is the naming inconsistency of the repackaged branches, and the impossibility of using upstream/foo branches when there is already a branch called "upstream". [TEAM: proposal] One solution that came in discussions in #debian-prometheus is to instead use these branch names, which allow for future new uses: upstream/master upstream/repackaged etc. Of course, "upstream/master" might be absent if we also go with the previous proposal. [/TEAM] Packaging branches ------------------ These follow DEP-14 format, which is the used to automatically configure sbuild, pbuilder , or cowbuilder so that the build is performed in the right environment. Packaging work happens in the "debian/<distribution>" branch, being "debian/sid" the branch where most of the work happens. When we need to backport packages, or produce updates to stable releases, we use branches named like "debian/buster-backports", which must branch off the relevant commit in "debian/sid" history. Tooling configuration --------------------- To make things more uniform and simple, the "debian/gbp.conf" file should define the parameters needed for the building of the package with no extra fuzz. This file will differ in the "debian-branch" parameter for each packaging branch, and the "upstream-branch" will vary depending on the package. Normally, it should just contain this: ##### [DEFAULT] debian-branch = debian/sid debian-tag = debian/%(version)s #upstream-branch = upstream/repackaged upstream-branch = upstream/master upstream-tag = upstream/%(version)s [buildpackage] dist = DEP14 ##### Any local configuration, such as which builder to use, paths for resulting files, etc. should be placed in $HOME/.gbp.conf. Changelog creation ------------------ Normally, you should format your commit messages so they can be used unchanged to generate the debian/changelog file, and never modify the debian/changelog direclty when doing other changes to avoid merge conflicts. There are also some tags you can add to modify the automatic generation, see the gbp-dch man page for details. When you start working on a new Upstream release, you need to create a new entry in the changelog with the right version before you can attempt building, as gbp will use that to determine the tag where the upstream source can be found. You can leave this uncommitted until the packaging is ready: $ dch -v 1.33+ds-1 "New upstream release." To fill the changelog with all the commit messages, use: $ gbp dch debian/ This will only take into account commits since the last time the changelog was modified in git unless you pass the "--since" option. Recommended local configurations -------------------------------- This is not normative, as it only affects your local environment, but makes things easier. Environment variables: $HOME/.bashrc or equivalent: === export [email protected] export DEBFULLNAME="Martina Ferrari" === Devscripts configuration: $HOME/.devscripts === DEBCHANGE_RELEASE_HEURISTIC=changelog DEBCOMMIT_SIGN_TAGS=yes DEBSIGN_KEYID=<your GPG key id> DEBUILD_DPKG_BUILDPACKAGE_OPTS="-i -ICVS -I.svn" USCAN_DOWNLOAD=no USCAN_VERBOSE=yes === Git-buildpackage configuration: $HOME/.gbp.conf === [DEFAULT] pristine-tar = False sign-tags = True color = auto [buildpackage] export = WC ignore-new = True notify = False export-dir = ../build-area/ # Make sure source-only packages are generated too (options for pbuilder and # sbuilder). #builder = sbuild --source-only-changes -s -v -A pbuilder = True pbuilder-options = --source-only-changes # Run a full lintian check and sign both the source-only as well as the # arch-specific changes files. postbuild = echo Running lintian and debsign: && \ cd "$GBP_BUILD_DIR" && \ lintian -I --pedantic --no-tag-display-limit && \ debsign --debs-dir `dirname "$GBP_CHANGES_FILE"` && \ debsign --debs-dir `dirname "$GBP_CHANGES_FILE"` -S --no-re-sign [dch] full = True meta = True customizations = /usr/share/doc/git-buildpackage/examples/wrap_cl.py [clone] repo-user = DEBIAN repo-email = DEBIAN debian-branch = debian/sid # Rename remote to "debian" and set-up automatic fetch & push rules. postclone = git remote rename origin debian && \ git config --add remote.debian.push 'refs/heads/debian/*' && \ git config --add remote.debian.push 'refs/heads/upstream' && \ git config --add remote.debian.push 'refs/tags/debian/*' && \ git config --add remote.debian.push 'refs/tags/upstream/*' && \ git config --add remote.debian.fetch 'refs/tags/*:refs/tags/*' === Workflow examples ----------------- These examples assume you are using the recommended configurations. I am not yet using the experimental new features of dh-make-golang, which I only learn today they existed :) 1. Create a repository from scratch. # Use dh-make-golang to create a basic template (but we will discard the git repo). $ dh-make-golang make -dep14 -wrap-and-sort at -type library \ golang.org/x/arch $ mv golang-golang-x-arch/debian generated-debian $ rm -rf golang-golang-x-arch # Create salsa project. $ PKG=new-workflow-demo $ dh-make-golang create-salsa-project $PKG # Set up local git repository, and add configuration normally added by gbp clone hooks. $ git clone -o debian \ [email protected]:go-team/packages/$PKG.git $ git config --add remote.debian.push 'refs/heads/debian/*' $ git config --add remote.debian.push 'refs/heads/upstream/*' $ git config --add remote.debian.push 'refs/tags/debian/*' $ git config --add remote.debian.push 'refs/tags/upstream/*' $ git config --add remote.debian.fetch 'refs/tags/*:refs/tags/*' # Add upstream remote, using info extracted by dh-make-golang $ SRC=$(sed -n 's/Source:\s*\(\S\+\)/\1/p' \ ../generated-debian/copyright) $ git remote add --fetch --no-tags upstream $SRC # Decide the commit where the packaging takes place: either a snapshot in upstream's master branch, or a specific tag. # Use git-log and git-ls-remote to find them. $ git ls-remote --tags upstream $ git log remotes/upstream/master # Define variables based on those findings. $ COMMIT=368ea8f $ VERSION=0.0~git20191126.368ea8f $ GIT_VERSION=$(echo $VERSION | sed 's/[~:]/_/g') # Create upstream/master at that commit and tag. $ git checkout -b upstream/master $COMMIT $ git tag upstream/$GIT_VERSION # If needed, do repackaging, and tag the new source. $ git checkout -b upstream/repackaged refs/heads/upstream/master $ git rm -rf vendor $ git commit -m "Remove vendor/ directory." $ git tag upstream/${GIT_VERSION}+ds # Create the packaging branch (either from upstream/master or upstream/repackaged). $ git checkout -b debian/sid # Push initial branches to salsa; order is important, as it determines the default branch. $ git push --set-upstream debian debian/sid # And all the others, according to the push configuration. $ git push --set-upstream debian # Use auto-generated packaging as starting point. $ mv ../generated-debian/ debian/ $ git add debian/ $ git commit -m "Initial packaging prepared by dh-make-golang." $ git push debian And now you are ready to go. I actually ran these commands, and you can see the result in https://salsa.debian.org/go-team/packages/new-workflow-demo/ 2. Clone an existing repository from salsa. # The postclone hook in gbp-clone configures the debian remote. $ PKG=prometheus-node-exporter $ gbp clone [email protected]:go-team/packages/$PKG # Add the upstream remote, fetch commits, but ignore tags. $ SRC=$(sed -n 's/Source:\s*\(\S\+\)/\1/p' debian/copyright) $ git remote add --fetch --no-tags upstream $SRC
