On 19/01/17 07:00, Robie Basak wrote:
Hi Peter,

On Thu, Jan 19, 2017 at 02:16:27AM +0000, peter green wrote:
Some time ago I put together what I call the "autoforwardporter". The
aim of this is to take downstream changes and apply them on top of new
debian uploads.
Can you expand on this?

Here is my attempt at describing how the system works.

Throughout the process "local" versions are distinguished from "debian" 
versions by looking for tags (in our case +rpi and +rvt) in the version number. The +rpi tag is 
used where we have local modifications, the +rvt tag is used where we previously had local 
modifications, have since removed them but for version numbering reasons we can't just use the 
Debian version directly.

The tag is normally located at the end of the version number, but occasionally 
it is located earlier in the version number for version ordering reasons.

The system relies on a whitelist of source package names that it should 
consider processing (some local changes are short term hacks that should 
probbablly not be carried forward)

When the system tries to make a forwardport it creates a logfile. It will only 
try to make a forwardport with a given version number once unless the logfile 
is manually removed.

Stage1: local repo on autoforwardporter box

We use reprepro to maintain local copies of both our own suites and the corresponding suite from Debian on the box that runs the 
autoforwardporter. On our own side we have a "main" suite and a "staging suite" (similar in concept at least to 
ubuntu's "proposed"). For stretch these suites are called "stretch" (our main suite) "stretch-staging" (our 
staging suite) and "stretch-deb" (debian stretch)

Stage1: import.

I am using dgit import-dsc for this with a custom driver script that takes a directory 
full of dscs, imports and tags them. If the package has a "dgit:" header we 
just let dgit import it with whatever history is in the git repo.

If the dsc does not use dgit then the import script figures out the history 
shape it should use from debian/changelog. For locally modified packages the 
immediate parent must be present or the import fails. For non locally modified 
packages we are far more lenient allowing version skipping and even orphan 

Stage2: decide whether to forward port and figure out version numbers

For each package in the whitelist we check the version numbers. Figure out whether 
there are local changes in the staging or main suites that should be brought 
forward to a new version from Debian and if-so what the new version number should 
be. The code is quite tricky because it has to handle some weird corner cases (for 
example if we have a version 1.0+rpi1 and debian introduces 1.0+deb9u1 then the 
new version needs to be numbered 1.0+rpi1+deb9u1 since 1.0+deb9u1+rpi1 < 
1.0+rpi1 )

Stage3: potential psuedomerge.

First we grab the changelog from our old local version and look for the first 
entry in the changelog that does not have one of our local tags on it. This 
tells us the version from Debian that it was based on.

In most cases the old version from Debian will be an ancestor of the new 
version from Debian but sometimes it isn't. In this case we do a psuedomerge to 
create a commit that has the tree from the new debian version and has the old 
debian version as an ancestor. This avoids trying to merge changes from the old 
Debian version into the new Debian version when we do the main merge.

Stage 4: main merge

This bit is a simple git merge. The "ours" side of the merge is our old version and the 
"theirs side" is the new version from Debian with the psudomerge if needed.

We use the "diff3" conflictstyle so our later fixup tools can get the context 
they need.

Stage 5: post merge fixup

It is necessary to fix up debian/changelog so it doesn't contain conflict markers and so 
it has the right versoin number. Exactly how this is done is a matter of taste, 
personally I squash together all the "local" changes into a single changelog 
entry on top of the Debian changelog. Ubuntu seem to be a fan of mergechangelogs 
(personally I find the result of mergechangelogs to be a confusing mess).

It is also frequently needed to fix up debian/patches/series . The script tries 
to retain order with local patches sorted after debian ones.

I expect further fixup will be needed, in particular I am expecting to have to 
add some further quilt-related hackery. I also want to take a look at 
debian/control merge failures, at least the simple cases.

Stage 6: commit results to working branch for archival

We commit the results to a working branch and push them to a suitable (local) 
repo to preserve them. If there were merge failures that could not be 
auto-resolved we commit the files with conflict markers still in them and put a 
list at the top of debian/changelog. This allows a human to clone the working 
branch, finish the fixup manually and amend the merge commit.

Stage 7: build

Assuming we were able to automatically resolve the merge conflicts we build a 
source package with dgit and then call sbuild to build that source package.

Assuming it built succesfully we copy it to our "output" directory.

Stage 8: dgit push

Assuming the source package builds successfully we do a dgit push to add the 
dgit: tag to the dsc and push the git stuff to our dgit server. We suppress the 
actual uploading for the moment as I like to make some final manual checks 
before uploading.

If the dgit push succeeds we re-copy the package with the dgit: header added to our 
"output" directory.

vcs-pkg-discuss mailing list

Reply via email to