Re: Preferred git branch structure when upstream moves from tarballs to git

[Sean Whitton]

Hello,

On Thu 09 May 2019 at 01:23PM +01, Ian Jackson wrote:

> For your workflow, I see broadly two possibilities, depending what
> representation you use for upstream source code:
>
>  (i) You use upstream tarballs as your representation of the upstream
>  source code, and do not use upstream git branches.
>
>  If this is the case then the benefits for everyone in me
>  implementing #903392 for you, and you then using `dgit
>  push-source', are - frankly - limited.  The resulting dgit git
>  branch would still contain imports of .orig tarballs, rather than
>  the actual upstream git history.
>
>  The user of `dgit clone' would get your *packaging* history,
>  sure, but this is a much more minor benefit.

A benefit not mentioned here is that it gets another class of DDs typing
`dgit push-source` for their uploads.  If getting all DDs typing `dgit
push-source` to upload is going to be how we make source packages
obsolete, that's significant.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Preferred git branch structure when upstream moves from tarballs to git

[Sean Whitton]

Hello Ansgar,

On Tue 07 May 2019 at 06:30PM +02, Ansgar wrote:

> So in a way the problem is that the documentation needs to exist.  git-
> debrebase or dgit (to a lesser extend) implement a Debian-specific
> version control system on top of Git.  git-debrebase makes this most
> obvious: the Git history it generates is pretty much unlike anything
> you would ever get from regular Git usage (counterexamples welcome), it
> mangles your Git commits into a specific format it expects
> (laundering), mangles rebased branches into fast-forward merges with
> pseudomerges; all this does not happen in regular Git usage.
>
> dgit has less of these problems, but still does things different from
> regular git usage (again pseudomerges to make rebases fast-forward for
> example); there are other issues as well such as its suggested use for
> NMUs feeling rather unfriendly as it moves the packaging to another VCS
> than the maintainer uses with a disjoint history...

dgit does the bare minimum required to interoperate with the Debian
archive.  If you don't use dgit, the need to handle the archive's
idiosyncrasies (e.g. the concept of NMUs) doesn't go away, but instead
is all done by a human.  And this introduces the risk of overwriting
NMUs, missing security fixes etc.

I am very happy that dgit is watching my back when it comes to dealing
with the Debian archive.  As an experienced DD I have a pretty solid
grasp of how ftp-master works, but it is useful to have a tool checking
my assumptions as I go.

> It illustrates my other main gripe with dgit/git-debrebase: it makes it
> harder to share in-progress work, in fact git-debrebase discourages
> people from doing so:
>
> | Note that each time you conclude a debrebase you introduce a
> | pseudomerge into your git history, which may make it harder to read.
>
> | A simple convention you can use to minimise the number of
> | pseudomerges is to git debrebase conclude only right before you
> | upload or push to salsa.debian.org.
>
> Yes, one can avoid some of the problems by pushing a non-fast-forward,
> non-interchange branch.  But that differs from the regular workflow
> and, again, requires more advanced Git knowledge.

This, and other things in your mail to which I'm replying, are
legitimate criticisms of git-debrebase, but not of dgit.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Preferred git branch structure when upstream moves from tarballs to git [and 1 more messages]

[Ian Jackson]

Holger Levsen writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git [and 1 more messages]"):
> On Thu, May 09, 2019 at 01:46:19PM +0100, Ian Jackson wrote:
> > debcheckout *certainly* doesn't do this.  It just gives you the
> > current master which may not have been uploaded anywhere.
> 
> nope:
> 
> $ debcheckout munin
> declared git repository at https://salsa.debian.org/debian/munin.git -b debian
> git clone https://salsa.debian.org/debian/munin.git -b debian munin ...
> [...]

Sorry, I was speaking loosely.

I meant it gives you the current head of the maintainer(s)' main
Debian sid packaging branch.  Which may be (and often is) ahead of the
archive, because things have been pushed to it and not uploaded yet.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git [and 1 more messages]

[Ian Campbell]

On Thu, 2019-05-09 at 12:51 +, Holger Levsen wrote:
> On Thu, May 09, 2019 at 01:46:19PM +0100, Ian Jackson wrote:
> > debcheckout *certainly* doesn't do this.  It just gives you the
> > current master which may not have been uploaded anywhere.
> 
> nope:
> 
> $ debcheckout munin
> declared git repository at https://salsa.debian.org/debian/munin.git
> -b debian
> git clone https://salsa.debian.org/debian/munin.git -b debian munin
> ...
> [...]

I don't think that one counterexample invalidates what Ian says in the
common case, it is certainly true that the most usual thing you get
from debcheckout is master regardless that some (minority of)
maintainers might have tried to be smarter.

In any case for me it gets the `debian-experimental` branch, not really
sure why and it's probably not what I expected/wanted[0]:

   $ debcheckout munin
   declared git repository at https://salsa.debian.org/debian/munin -b 
debian-experimental
   git clone https://salsa.debian.org/debian/munin -b debian-experimental munin 
...
   Cloning into 'munin'...
   warning: redirecting to https://salsa.debian.org/debian/munin.git/
remote: Enumerating objects: 53812, done.
remote: Counting objects: 100% (53812/53812), done.
remote: Compressing objects: 100% (13784/13784), done.
remote: Total 53812 (delta 37431), reused 53214 (delta 36851)
Receiving objects: 100% (53812/53812), 12.51 MiB | 1.31 MiB/s, done.
Resolving deltas: 100% (37431/37431), done.
warning: redirecting to https://salsa.debian.org/debian/munin.git/

I think at best all this example really does is mutate Ian's "It just gives you 
the
current master which may not have been uploaded anywhere" into "It just
gives you some branch which may not have been uploaded anywhere"?

Ian.

[0] and the manpage doesn't seem to suggest a way to request a
particular suite.

Re: Preferred git branch structure when upstream moves from tarballs to git [and 1 more messages]

[Holger Levsen]

On Thu, May 09, 2019 at 01:46:19PM +0100, Ian Jackson wrote:
> debcheckout *certainly* doesn't do this.  It just gives you the
> current master which may not have been uploaded anywhere.

nope:

$ debcheckout munin
declared git repository at https://salsa.debian.org/debian/munin.git -b debian
git clone https://salsa.debian.org/debian/munin.git -b debian munin ...
[...]

(not replying to the other points because enotime, sorry.)


-- 
tschau,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: Preferred git branch structure when upstream moves from tarballs to git [and 1 more messages]

[Ian Jackson]

Holger Levsen writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git [and 1 more messages]"):
> On Wed, May 08, 2019 at 04:14:00PM +0100, Ian Jackson wrote:
> 
>  * Debian should provide source code as git branches which:
>   - can be built using a standard set of runes
>   - will produce the same binaries as official Debian ones 
>   - can be reliably located
>   - can be easily modified (using standard git commands)
>   - contain the git histories we are actually using ourselves
> 
> There is only one way to do this.  It is `dgit push[-source]'.
> Vcs-Git and Salsa do not provide this. 
> 
> ---end quote---
> 
> and I'm not sure I agree this is true, to me Vcs-Git and Salsa do provide all
> of this, *if* Vcs-Git is set.

No, sadly not.

   - can be built using a standard set of runes

This is not true of Vcs-Git because the format of the Vcs-Git
repository is not standardised.

   - will produce the same binaries as official Debian ones 

This is not true of Vcs-Git because it is not trivial (or even, in the
general case, systematically possible) to find the right git tags
corresponding to a particular upload.  DEP-14 does help here but not
everyone agrees with it: eg, git-dpm has slightly nonstandard version
mangling and its maintainer has resisted changing; packaging-only
monorepos cannot use DEP-14 tags because DEP-14 tags aren't
package-qualified.

debcheckout *certainly* doesn't do this.  It just gives you the
current master which may not have been uploaded anywhere.

   - can be reliably located

Vcs-Git does help find the repo, but finding the right branch/commit
is difficult (see above).  Also, Vcs-Git gives wrong answers if there
was an upload (eg an NMU) which was not pushed to salsa.  If you use
debcheckout you might miss an important security update this way!

   - can be easily modified (using standard git commands)

This is not true of Vcs-Git because of the repository format problem.
The most common tree format is patches-unapplied, which does not meet
this requirement.  git-grep does not work properly; git-blame does not
work properly; git-merge does not work properly; git-cherry-pick does
not work properly. etc. etc.

   - contain the git histories we are actually using ourselves

Vcs-Git does meet this requirement.

> So, IOW, I can see problems with individual packages here but not with the
> general workflow/tool of using vcs-git: and debcheckout.

See also my response to some similar points raised by Simon McVittie:

  https://lists.debian.org/debian-devel/2019/05/msg00085.html

Regards,
Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

(adding #903392)

Ben Finney writes ("Re: Preferred git branch structure when upstream moves from 
tarballs to git"):
> What I remain unconvinced of is the worth of abandoning the clean
> separation between an upstream source repository (whether Git repository
> or not) and a VCS repository for Debian packaging (typically in Git).

I disagree with this but I'm not really interested in arguing you out
of it.  As Ian Campbell says, dgit doesn't want to change your
workflow.  It wants to help you help *users* and *downstreams* to
change theirs, from tarballs+diffs to git.

My interest is to make that possible, *without* maintainers having to
change their workflows.

For your workflow, I see broadly two possibilities, depending what
representation you use for upstream source code:

 (i) You use upstream tarballs as your representation of the upstream
 source code, and do not use upstream git branches.

 If this is the case then the benefits for everyone in me
 implementing #903392 for you, and you then using `dgit
 push-source', are - frankly - limited.  The resulting dgit git
 branch would still contain imports of .orig tarballs, rather than
 the actual upstream git history.

 The user of `dgit clone' would get your *packaging* history,
 sure, but this is a much more minor benefit.

 (ii) You use upstream git history, and generate .orig tarballs
 with (say) git-deborig.

 Implementing #903392 for this case should be straightforward,
 provided that your .orig tarballs and the git branch actually are
 identical.  Furthermore, it would be very useful to someone who
 `dgit clone's one of your packages: they would get a full and
 proper git history which would work exactly as they expect.

 The question here would be: how do you manage the upstream git
 branch ?  dgit push would need to automatically find (or be told,
 but that would be a UI nuisance) the upstream git commit.

 (And if that upstream git commit is in a separate tree somewhere,
 dgit would have to find that tree, too.)

If you are roughly in situation (ii) then I am quite interested in
supporting your use of dgit push, eg by working on #903392.  Please
engage in that bug, and encourage others in a similar position to do
the same.

If you are currently in situation (i) then your adoption of dgit is a
low priority for me, since to provide really useful output from `dgit
clone' it would be necessary for you to change your workflow.  (I do
have a wip branch but it is a mess and not tested.  So if someone
wants to work on this, let me know.)

I am not really interested in arguing that people's workflows are
wrong; it is a very religious kind of topic.  Let us save that for a
bar conversation some time ...

HTH.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Ansgar writes ("Re: Preferred git branch structure when upstream moves from 
tarballs to git"):
> On Thu, 2019-05-09 at 10:33 +0100, Ian Campbell wrote:
> > I think there's a common misconception here which has cropped up
> > several times in this thread. (NB: I've not used dgit in anger yet, but
> > I hope what follows is correct).

Everything Ian said is completely correct.  Thanks, Ian :-).

> > That misconception is that "dgit repo" == "maintainer's repo", which is
> > not actually the case. As a maintainer you can (if you want) just use
> > "dgit push" (instead of dput) while only caring about (and interacting
> > with) the "maintainer's repo", never touching or looking at dgit's view
> > of the world (apart from perhaps noticing and ignoring some `dgit/*`
> > branches in your _local_ repo, I don't beleive you are required to push
> > those to anywhere, including your maintainer repo).
> 
> I think that is very confusing, yes.
> 
> By now I have come to understand that whatever "dgit" produces should
> be just regarded as a build artifact, just like binary packages or for
> some people the .dsc or for some workflows debian/patches/*.

As a maintainer you can think of it like, and treat it as, a build
product.  The user or downstream can think of it like, and use it as,
the source code.  It all depends on your point of view.

Fundamentally the circle being squared here is this:

 * Maintainers think "the source code" means some Debian-specific
   thing.  Often a strange or dangerous thing - even the most usual, a
   patches- unapplied git branch, is dangerous if you don't know how
   to handle it.  Furthermore the maintainers don't even agree.

 * Users think "the source code" means a normal git branch you can
   build and do `git grep' and `git commit' and `git merge' with.

Luckily it turns out that you can construct the 2nd from the 1st, in
most cases.  (Indeed `gbp pq import' sort of does that and actually
for maintainers using gbp pq, the patch queue branch is the preferred
form for most *modifications* and the master branch is an interchange
format used for distribution; luckily the two are interconvertibel.)

And you can do it in such a way that the 2nd *contains* the 1st, so
that anyone who has the converted version also has the unconverted
version, if they care.  In practice, it is IME very rare to want this.

> I wonder why not the "maintainer view" is published (for the part
> making Git repositories "reliably locatable") and any mangling is
> applied by "dgit clone" (instead of "dgit push")?  ("dgit push" can
> check that the mangling works.)

dgit push makes, and publishes, a maintainer view tag, as well as a
dgit view tag.  So you can indeed get the maintainer view.

So far no-one who is a user of `dgit clone' and `dgit fetch' has
requested a feature for presenting the maintainer view as a matter of
course.  Such a feature would have some difficulties.  What is the
`maintainer view' if the maintainer uses gbp pq, but the most recent
upload was an NMU not done with dgit ?

If you say "the .dsc import" then this "maintainer view" format
oscillates across versions.  If you say "whatever you get when
converting the .dsc back to the maintainer's format" then a
backwards-converter would be required.

Or to put it another way, while everyone is required, when uploading,
to provide something that looks like a .dsc, not everyone who uploads
is required to provide something in some other maintainer-determined
format.

> Then whatever is published is what is the actual preferred form of
> modification (as it is what the maintainer uses),

The key question about "preferred form for modification" is "preferred
by who", of course.  As I say above, maintainers and (at least a big
subset of) users/downstreams have different opinions.

Happily since the maintainer's view is always an ancestor of the
corresponding dgit view, publishing the dgit view implies publishing
the maintainer's view; and finding it in the history is
straightforward because there is an appropriate tag.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ansgar]

On Thu, 2019-05-09 at 10:33 +0100, Ian Campbell wrote:
> On Thu, 2019-05-09 at 11:19 +1000, Ben Finney wrote:
> > What I remain unconvinced of is the worth of abandoning the clean
> > separation between an upstream source repository (whether Git repository
> > or not) and a VCS repository for Debian packaging (typically in Git).
> 
> I think there's a common misconception here which has cropped up
> several times in this thread. (NB: I've not used dgit in anger yet, but
> I hope what follows is correct).
> 
> That misconception is that "dgit repo" == "maintainer's repo", which is
> not actually the case. As a maintainer you can (if you want) just use
> "dgit push" (instead of dput) while only caring about (and interacting
> with) the "maintainer's repo", never touching or looking at dgit's view
> of the world (apart from perhaps noticing and ignoring some `dgit/*`
> branches in your _local_ repo, I don't beleive you are required to push
> those to anywhere, including your maintainer repo).

I think that is very confusing, yes.

By now I have come to understand that whatever "dgit" produces should
be just regarded as a build artifact, just like binary packages or for
some people the .dsc or for some workflows debian/patches/*.

Though directing users to build artifacts when they request the source
seems counterproductive to me...

I wonder why not the "maintainer view" is published (for the part
making Git repositories "reliably locatable") and any mangling is
applied by "dgit clone" (instead of "dgit push")?  ("dgit push" can
check that the mangling works.)

Then whatever is published is what is the actual preferred form of
modification (as it is what the maintainer uses), but if so desired one
can still get a "dgit view".  (Though for contributing changes to the
maintainer, one should probably base them on the maintainer view...)

In this case the published history also matches the "git histories we
are actually using ourselves", a design goal not met currently; one
could also apply the mangling feature to repositories not published on
the dgit server.

Ansgar

Re: Preferred git branch structure when upstream moves from tarballs to git [and 1 more messages]

[Colin Watson]

On Wed, May 08, 2019 at 03:26:32PM +, Holger Levsen wrote:
> thanks for the pointer, but I don't see the string debcheckout in that
> message and vcs-git only once, where you write:
> 
> ---begin quote---
> 
> >From these we can conclude:
> 
>  * Debian should provide source code as git branches which:
>   - can be built using a standard set of runes
>   - will produce the same binaries as official Debian ones 
>   - can be reliably located
>   - can be easily modified (using standard git commands)
>   - contain the git histories we are actually using ourselves
> 
> There is only one way to do this.  It is `dgit push[-source]'.
> 
> Vcs-Git and Salsa do not provide this. 
> 
> ---end quote---
> 
> and I'm not sure I agree this is true, to me Vcs-Git and Salsa do provide all
> of this, *if* Vcs-Git is set.
> 
> And if it's not set, it's either because the package is not in git or because
> d/control is lacking information, aka the package is buggy.

I think the thing you're missing is Ian's "can be reliably located"
point.  As you say, debcheckout works if those conditions are met, but
they're very often not met.  Given how long debcheckout and Vcs-Git have
been around, it seems a reasonable conclusion that they're unlikely to
reach 100% any time soon (and there's the long-standing problem with
stale metadata in stable releases and so on that means it isn't even a
ratchet, and won't be unless some as-yet-unfinished work to make it be
an archive override or similar happens instead).  Without wishing to
speak for Ian, I think this is what he's abbreviating as "do not provide
this".

By contrast, "dgit clone" works regardless of whether these conditions
are met, although it results in a better git history if the maintainer
used "dgit push" or "dgit push-source".  That does seem like an
important distinction.

> So, IOW, I can see problems with individual packages here but not with the
> general workflow/tool of using vcs-git: and debcheckout.
> 
> (or what am I missing?)

Maybe another way to look at it is that sometimes it makes more sense to
look at a problem systemically, including new tools, than by annotating
thousands of source packages one by one.

[FWIW: I recently started using "dgit push-source" for my uploads,
partly as a result of this thread although I'd been meaning to look into
it for ages.  I've filed a couple of bugs and been confused about one or
two things that Ian has kindly helped me with, but by and large it's
been much more straightforward than I'd feared.]

-- 
Colin Watson   [cjwat...@debian.org]

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Campbell]

On Thu, 2019-05-09 at 11:19 +1000, Ben Finney wrote:
> What I remain unconvinced of is the worth of abandoning the clean
> separation between an upstream source repository (whether Git repository
> or not) and a VCS repository for Debian packaging (typically in Git).

I think there's a common misconception here which has cropped up
several times in this thread. (NB: I've not used dgit in anger yet, but
I hope what follows is correct).

That misconception is that "dgit repo" == "maintainer's repo", which is
not actually the case. As a maintainer you can (if you want) just use
"dgit push" (instead of dput) while only caring about (and interacting
with) the "maintainer's repo", never touching or looking at dgit's view
of the world (apart from perhaps noticing and ignoring some `dgit/*`
branches in your _local_ repo, I don't beleive you are required to push
those to anywhere, including your maintainer repo).

The "dgit repo" provides a view onto precisely what has been uploaded
to the archive, no more no less. Where it can (i.e. where maintainers
are using "dgit push" etc) it tries to stitch in the "maintainer's
repo" (which I think is part #1 of where this misconception arises) as
a subset but it doesn't follow that using dgit implies any constraint
on the maintainer's repo (or at least that is the aim, AIUI there are
some bugs and wrinkles which make this less than 100% true in reality,
e.g. #908747)

There is a caveat though (which I think is part #2 of the
misconception) which is that dgit sometimes does need to understand
tooling which maintainer's are using in the "maintainer's repo" (i.e.
it needs to understand gbp pq and debrebase etc to work with repos
maintained that way). But "dgit needs to understand the maintainer's
repo" does not imply the inverse "the maintainer's repo must be shaped
somehow by the use of dgit". In fact AIUI not having the use of dgit
put constraints on how a maintainer structures their repo is an
explicit design goal of dgit.

It follows that there are some "maintainer's repo" layouts/strategies
which dgit is not currently aware of and is therefore unable to cope
with (and I beleive "packaging only repo" workflow is one of those).
However AIUI from what Ian has said elsewhere that this is purely down
to prioritisation and time, and given either big demand or adequate
time this workflow will be supported by dgit. There is no philosophical
objection (in dgit at least) to such workflows nor a desire to force
(nor nudge) maintainers away from using whatever workflow they wish to.

Ian.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ben Finney]

Ian Jackson  writes:

> So far I have gained the impression that the kind of people who are
> using packaging-only git trees tend to be very conservative, and very
> comfortable with .dsc/tarball/patch/quilt based tools, and not really
> convinced of the usefulness of git.

Let me counter that impression, then: I am well convinced of the
usefulness of Git, both generally and for Debian packaging in
particular. I use it every day for all manner of development work.

What I remain unconvinced of is the worth of abandoning the clean
separation between an upstream source repository (whether Git repository
or not) and a VCS repository for Debian packaging (typically in Git).

The worth of such a clean separation is that it does not run afoul of
the often wide differences between upstream developer team workflow
and Debian package maintainer team workflow.

So it's not conservatism or unwillingness to learn, in my case at least.
I don't see that there's sufficient benefit to be had trying to weld
their work together into one Git repository, when we have existing tools
and workflows that don't demand that conflation.

I believe I've made an informed assessment of the benefits and costs of
a combined-upstream-with-Debian-packaging single VCS repository, and
tentatively rejected it. That choice is not irrevocable and I could be
convinced by sufficient benefit, but AFAICT the costs remain what they
are.

> I have even experienced what feels to me like significant hostility.

Yes, I've seen that at times. I'm sure you get more that I haven't seen.
It's sad to see when it happens.

> If I am wrong and there are more users to be had by implementing this
> feature then I will definitely consider giving it a higher priority.

I'll try to find time to respond on the bug report you cited.

> BTW, dgit is capitalised thus.

Can't promise I'll remember that; as with DPut, it's easier to see the
portmanteau term with distinct capitalisation, in my opinion.

-- 
 \  “Advertising is the price companies pay for being unoriginal.” |
  `\—Yves Béhar, _New York Times_ interview 2010-12-30 |
_o__)  |
Ben Finney

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Simon McVittie writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git"):
> On Sat, 04 May 2019 at 21:10:34 +0100, Ian Jackson wrote:
> > So far I have gained the impression that
> > the kind of people who are using packaging-only git trees tend to be
> > very conservative, and very comfortable with .dsc/tarball/patch/quilt
> > based tools, and not really convinced of the usefulness of git.
> 
> A less common, but IMO valid, reason to use packaging-only git trees
> is that your upstream source code consists of large non-text objects
> that aren't managed efficiently by git. The openarena-data family of
> packages are (I think) the only place I still use packaging-only git
> trees, and the nexuiz-data package is an example of the sort of monster
> that will typically be created if you don't.

Yes.

These are a minority, though, I think.  At some point, we will have to
come up with ways of handling these.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Simon McVittie writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git"):
> On Tue, 07 May 2019 at 19:25:39 +0100, Ian Jackson wrote:
> > What I am primarily advocating for in this thread is that maintainers
> > should choose `dgit push-source' over `dput' (where this is possible).
> > This is the only way for a maintainer to provide users with the git
> > history in a sensible form (ie, a form which does not require the user
> > to know what special git practices the maintainer has adopted).
> 
> I think you're implicitly asserting here that the tree that exists in
> a dgit commit (upstream source as conveyed by the orig tarball, with
> debian/ added, with patches applied if any) is the only sensible form
> for the git history of a Debian source package, and I don't think that's
> something that has consensus.

Sorry, I was not clear.  My view is much more nuanced than that.
I think that
   - we should publish one consistent form to our users
   - this is the only sensible form for that.

I do not think that this is the only sensible form for working within
Debian.  I am not trying to abolish patches-unapplied.

Happily it is possible to convert most other forms to that.
(That's kind of implied by being able to build the thing.)

> I realise that you think (some or all of) those other "shapes" are not
> sensible,

No, many of these (most of them) are completely fine.  My word
`sensible' there was only in the context of a form to provide to those
users who want to change how their computer behaves, without learning
a lot of obscure Debian-specific source code management stuff.

> I don't think it's coincidence that many of the larger teams in Debian,
> in particular Perl and Python, have ended up with patches-unapplied.

Indeed not.  I am not trying to change that.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Simon McVittie writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git"):
> On Wed, 08 May 2019 at 12:18:41 +0100, Ian Jackson wrote:
> > Indeed, you yourself say you avoid gbp but for many packages, the
> > Vcs-Git header gives you a patches-unapplied format which requires[1]
> > gbp pq to switch to a patches-applied view before you build it.
> 
> This step is not required: for 3.0 (quilt) packages, dpkg-source is happy
> with either patches-applied or patches-unapplied source directories,
> as long as you're at one extreme or the other

Yes.  Perhaps indeed that's what the person I was responding to,
there, does.

However, this is actually quite dangerous.  You definitely do not want
to rely on it.  If dpkg-source's heuristic decides the patches have
already been applied, it will not apply them again.

The effect is that you might, for example, do this

  1. debcheckout
  2. dpkg-buildpackage -uc -b
  3. examine result, looks good
  4. git clean -xdff && git reset --hard # [1]
  5. git cherry-pick 23891dcaf
  6. dpkg-buildpackage -uc -b
  7. dpkg -i

and if you are unlucky, the 2nd build in step 6 silently didn't apply
the patches, even though the 1st build in step 2 did.  Whether you are
going to be unlucky is complicated.

[1] step 4 is needed because our build systems are often a mess.

There are other bizarre things about the state you get from
debcheckout with such a package.  For example, `git grep'
can sometimes fail to find the thing you are looking for:

  bubblewrap$ git --no-pager grep '^#!.*python3'
  # .oO{ But my .deb says python3, wtf ? }

This is all very well if you are a seasoned Debian person and know
your way round the hall of mirrors.  For someone with a passing
familiarity with git and no real Debian knowledge, it is a minefield.

To see what this looks like for a user, take a look at

   https://manpages.debian.org/stretch-backports/dgit/dgit-user.7.en.html

and imagine what you would have to write in an equivalent manpage
based on debcheckout.  debcheckout might give you an error message.
It might give you a CVS working tree!

Even if it succeeds and gives you git, it might give you a patches-
applied git-dpm or gdr branch.  It might give you a patches-unapplied
gbp pq git branch.  It might give you a multi-package packaging-only
monorepo.  It might give you the top of a topgit pile.  It might give
you a polyphonic multidimensional delta rhinoceros.

> Easier than the tool you have to use anyway (directly or indirectly)
> to build any package?

gbp pq import
  - makes git grep work
  - makes git log work
  - makes git blame work
  - leaves the tree clean
  - means you can actually edit files and git-commit them!

I don't say it's perfect.  Indeed I wrote a competing tool which has
an entirely different data model and different workflow, but that is
not really germane...

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git [and 1 more messages]

[Holger Levsen]

On Wed, May 08, 2019 at 04:14:00PM +0100, Ian Jackson wrote:
> > > [Ian Jackson:]
> > > > You should use dgit for the benefit of users.  See my other mail which
> > > > answers why Vcs-Git and debcheckout is not enough.
> > > 
> > > could you be so kind and provide a pointer, this thread is rather long
> > > already? (Maybe this is also worth an FAQ entry somewhere..)
> 
> I meant this message:
>   https://lists.debian.org/debian-devel/2019/05/msg00065.html

thanks for the pointer, but I don't see the string debcheckout in that
message and vcs-git only once, where you write:

---begin quote---

>From these we can conclude:

 * Debian should provide source code as git branches which:
  - can be built using a standard set of runes
  - will produce the same binaries as official Debian ones 
  - can be reliably located
  - can be easily modified (using standard git commands)
  - contain the git histories we are actually using ourselves

There is only one way to do this.  It is `dgit push[-source]'.

Vcs-Git and Salsa do not provide this. 

---end quote---

and I'm not sure I agree this is true, to me Vcs-Git and Salsa do provide all
of this, *if* Vcs-Git is set.

And if it's not set, it's either because the package is not in git or because
d/control is lacking information, aka the package is buggy.

So, IOW, I can see problems with individual packages here but not with the
general workflow/tool of using vcs-git: and debcheckout.

(or what am I missing?)


-- 
tschau,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: Preferred git branch structure when upstream moves from tarballs to git [and 1 more messages]

[Ian Jackson]

Holger Levsen writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git"):
> could you be so kind and provide a pointer, this thread is rather long
> already? (Maybe this is also worth an FAQ entry somewhere..)

Jonathan Carter writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git"):
> I'd like to second that request, I mean to go through this thread again
> when I have some free moments, but a FAQ would help a lot to assimilate
> all this information.

A FAQ is a really good idea.  Thank you for the suggestion.

In the meantime,

> > [Ian Jackson:]
> > > You should use dgit for the benefit of users.  See my other mail which
> > > answers why Vcs-Git and debcheckout is not enough.
> > 
> > could you be so kind and provide a pointer, this thread is rather long
> > already? (Maybe this is also worth an FAQ entry somewhere..)

I meant this message:
  https://lists.debian.org/debian-devel/2019/05/msg00065.html

> (I'm in a similar boat to Holger with my experience to gbp, I tried
> going through the documentation on the wiki a few times and just got
> frustrated. I just use packaging and Vcs-fields and tags and nothing
> fancier than that, which works for the few dozen packages I have and I
> think the workflow is trivial enough so that downstream consumers
> wouldn't have any problem whatsoever figuring out how to make use of it,
> but I suppose as that list grows it will become important to have
> worflows that scale up a bit more.)

I confess I always find these kind of comments (which I hope you won't
mind me parodying as "I don't use anything") a bit confusing because
they don't explain how you perform the key task that tools like gbp pq
are intended to help with.

I think what is going on here is this: to you, what you do for that is
so completely obvious it doesn't occur to you that I don't know what
it is.  There is such a gulf here between different people's workflows
that we are having trouble communicating.

If you are maintaining a package in "3.0 (quilt)", you are maintaining
a series of patches to the upstream part of the package.  The question
is: what tool(s) do you use to change what changes a patch makes to
the upstream source, edit a patch's accompanying message (if you give
them messages), add a new patch, drop a patch, rebase the patches onto
a new upstream version, and so on.

The answer clearly isn't "nothing" since you're not fiddling with the
individual bits in your computer's RAM by your electrotelekinetic
superpowers :-).  At least if you are then awesome but maybe you are
beyond need for version control software.

I guess the answer probably isn't even "I edit the diffs in the
patches, in my text editor".  At least I hope not since editing diffs
in a text editor is still horrific even with a really good text
editor.  I think that probably many people who say "I don't use
anything fancy" are using quilt, or maybe raw diff and patch ?  Maybe
you are editing debian/series with your text editor and occasionally
using rm and mv on patch files ?  Or git-mv and git-rm ?

I realise the above might sound facetious.  That's not my intent.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Holger Levsen]

On Wed, May 08, 2019 at 03:57:08PM +0100, Ian Jackson wrote:
> > both.
> For packaging-only, dgit does not properly support that.  Do you have
> the upstream source in git form too, and if so how do you combine
> them ?  Do you manipulate patches using quilt ? [1]

hm/i'm sorry, I cannot really think of such an example of a package I'm
maintaining. I've definitly encountered such packages though...

> > could you be so kind and provide a pointer, this thread is rather long
> > already? (Maybe this is also worth an FAQ entry somewhere..)
> Sorry, I should have done that the first time.  I'll reply to
> Jonathan and you.

thank you!

> [1] I have to say that I think quilt is just awful.  Perhaps your
> mileage varies, [...]

well, only sometimes. but then, it also does work.
 
> I know it can be work to learn new stuff, and of course it's a pain if
> lots of people are advising you to learn different and mutually
> incompatible new stuff, but anyone who is still using quilt will
> probably benefit a lot from learning how to use gitish tools instead.
 
most of the time I deal with quilt is when I want to modify existing
packages, usually not from sid, but from stable or older, and then it
feels more straightforward to deal with quilt. often I just dget the
sources (from an old release (*)), run git init ; git add * : git commit -m
init and fiddle qith quilt & git until it applies nicely.

that way, git really supports me nicely, without forcing me to learn
anything. there's so much to learn all the time...


(*) debcheckout is largely unusable for stretch and older as alioth is gone,
so... meh.

-- 
tschau,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Holger Levsen writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git"):
> On Wed, May 08, 2019 at 12:18:41PM +0100, Ian Jackson wrote:
> > I'm not sure what git branch format you are using.  (Maybe you said it
> > earlier in this thread.)  Packaging-only ?  git-merge ?
>  
> both.

For packaging-only, dgit does not properly support that.  Do you have
the upstream source in git form too, and if so how do you combine
them ?  Do you manipulate patches using quilt ? [1]

If you use git-merge and have all the source (including upstream
source) in a single git tree there is indeed no need to use gbp pq.
The manpage dgit-maint-merge(7) explains how to publish such a thing
with dgit.

> > You should use dgit for the benefit of users.  See my other mail which
> > answers why Vcs-Git and debcheckout is not enough.
> 
> could you be so kind and provide a pointer, this thread is rather long
> already? (Maybe this is also worth an FAQ entry somewhere..)

Sorry, I should have done that the first time.  I'll reply to
Jonathan and you.

Regards,
Ian.

[1] I have to say that I think quilt is just awful.  Perhaps your
mileage varies, but I thought quilt was awful by comparison with
*CVS*, when I first encountered it...

I know it can be work to learn new stuff, and of course it's a pain if
lots of people are advising you to learn different and mutually
incompatible new stuff, but anyone who is still using quilt will
probably benefit a lot from learning how to use gitish tools instead.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Matthew Vernon writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git"):
> Scott Kitterman  writes:
> > For what it's worth, when you start calling me names (unethical)
> > because I'm not using your shiny new toy, my immediate reaction is to
> > decide to ignore further discussion on the topic.

I'm sorry, that was not my intention.

> I think there's a considerable difference between:
> 
> "I think ethically, everyone should do X, and I will try and persuade
> you that this is correct"
> 
> And
> 
> "If you disagree with me that everyone should do X then you are
> unethical"

I meant the former.  I definitely don't mean to say that all the
people not doing what I think is best, are bad people.

Regards,
Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Simon McVittie]

On Sat, 04 May 2019 at 21:10:34 +0100, Ian Jackson wrote:
> So far I have gained the impression that
> the kind of people who are using packaging-only git trees tend to be
> very conservative, and very comfortable with .dsc/tarball/patch/quilt
> based tools, and not really convinced of the usefulness of git.

A less common, but IMO valid, reason to use packaging-only git trees
is that your upstream source code consists of large non-text objects
that aren't managed efficiently by git. The openarena-data family of
packages are (I think) the only place I still use packaging-only git
trees, and the nexuiz-data package is an example of the sort of monster
that will typically be created if you don't.

I suppose that's a form of not really being convinced of the usefulness
of git: I'm convinced that git is useful for program source code and
metadata, but I'm not convinced that it's useful for multiple gigabytes
of models, textures etc. that are added and deleted more often than they
are edited. If there was a convenient way to use a lookaside object store
like git-lfs or git-annex for source packages, then I'd use that.

Arguably this is an instance of the same problem as some of those we have
around DFSG interpretation and whether "software" refers to programs or
to all blobs of bytes: our processes are designed for program source
code, and non-program content doesn't always fit very well into those
processes; but we can't just not ship the non-program content, because
we're trying to provide a self-contained software distribution with no
external dependencies.

smcv

Re: Preferred git branch structure when upstream moves from tarballs to git

[Simon McVittie]

On Tue, 07 May 2019 at 19:25:39 +0100, Ian Jackson wrote:
> What I am primarily advocating for in this thread is that maintainers
> should choose `dgit push-source' over `dput' (where this is possible).
> This is the only way for a maintainer to provide users with the git
> history in a sensible form (ie, a form which does not require the user
> to know what special git practices the maintainer has adopted).

I think you're implicitly asserting here that the tree that exists in
a dgit commit (upstream source as conveyed by the orig tarball, with
debian/ added, with patches applied if any) is the only sensible form
for the git history of a Debian source package, and I don't think that's
something that has consensus.

Alternative "shapes" that other people advocate include: debian/ only
(a straightforward translation of typical packaging-in-svn to git);
patches-unapplied trees (as seen in e.g. the Perl, Python, GNOME teams);
debian/ overlaid onto upstream's git repository (as opposed to
upstream's tarball code-drops), with any differences between the orig
tarball and upstream's git repository papered over by
debian/source/local-options (as seen in the Xorg team).

I realise that you think (some or all of) those other "shapes" are not
sensible, and I realise that dgit's design requires a single canonical
result of importing a Debian source package, and I agree that as a
general operating principle it's undesirable that different packages
have their contents represented in different ways; but I don't think
it's necessarily helpful to assert that the representation chosen for
dgit is the only one that can be considered sensible, and I think you're
in danger of causing people who disagree with you on this point to not
consider dgit, which seems like the opposite of what you want.

I think the reason people are conflating the contents of the tree with
the workflow that was used to arrive at that tree is that the workflow
is not completely decoupled from the tree. This is most obvious with
git-dpm, which leaves a mechanically-generated file in debian/ that it
cannot work without (and that is only practical to maintain by using
git-dpm), but it's also true for other representations: for instance
gbp-pq normally uses a patches-unapplied tree as the interchange format
(requiring dgit to compensate for this difference in assumptions),
whereas git-debrebase requires patches-applied if I understand correctly.

The major interoperating "families" are:

* patches-applied: git-debrebase or dgit-maint-merge (and git-dpm would
  be in this category if it didn't require extra metadata in debian/).

* patches-unapplied: gbp pq, or patches managed out-of-band with quilt
  or by dropping git format-patch output into d/patches, either with full
  source in git or with only debian/ in git (built with
  gbp buildpackage --overlay, or by unpacking the orig tarball into the
  source directory and avoiding committing it to git).

(Native packages, and non-native packages with no patches, can be seen as
a special case of either family - if you don't have any patches then it
doesn't matter whether you have applied them or not.)

I don't think it's coincidence that many of the larger teams in Debian,
in particular Perl and Python, have ended up with patches-unapplied. It's
a straightforward continuation of how we used to maintain packages in
non-git VCSs like svn, it only requires learning to use a tool like
gbp pq or quilt if you have enough non-orthogonal patches that they
need to be rebased on each other to apply cleanly, and it's a fairly
close match for how most non-Debian source packaging formats work
(for instance Fedora SRPMs, Arch Linux PKGBUILDs and Gentoo ebuilds all
involve committing unified diffs to their respective VCSs alongside the
equivalent of debian/control).

smcv

Re: Preferred git branch structure when upstream moves from tarballs to git

[Matthew Vernon]

Scott Kitterman  writes:

> On May 7, 2019 8:50:57 PM UTC, Sam Hartman  wrote:
>>> "Ian" == Ian Jackson  writes:
>>Ian> The latter point is because using dgit push is an ethical
>>Ian> imperative, not because the two somehow have some deep
>>   Ian> technical linkage.  IMO almost *any* tutorial being written now
>>Ian> about how to do Debian packaging, and which mentions git at
>>Ian> all, ought to say to use dgit.
>>
>>I think this is at the crux of our disagreement.
>>I'm going to need to think about this for myself.
>>I think it's safe to say there is not a project consensus that dgit is
>>an ethical imperative.
>>And yes understanding that you believe that's the case makes it a lot
>>easier to understand the rest of your mail.
>>
>>As I said I'll need to think about whether I agree with that.  A lot of
>>my thinking in this discussion is that I consider dgit a nice to have,
>>not a requirement.
>
> For what it's worth, when you start calling me names (unethical)
> because I'm not using your shiny new toy, my immediate reaction is to
> decide to ignore further discussion on the topic.

I think there's a considerable difference between:

"I think ethically, everyone should do X, and I will try and persuade
you that this is correct"

And

"If you disagree with me that everyone should do X then you are
unethical"

...and that conflating the two as you appear to be doing so here is
quite unhelpful.

Regards,

Matthew

-- 
"At least you know where you are with Microsoft."
"True. I just wish I'd brought a paddle."
http://www.debian.org

Re: Preferred git branch structure when upstream moves from tarballs to git

[Simon McVittie]

On Wed, 08 May 2019 at 12:18:41 +0100, Ian Jackson wrote:
> Indeed, you yourself say you avoid gbp but for many packages, the
> Vcs-Git header gives you a patches-unapplied format which requires[1]
> gbp pq to switch to a patches-applied view before you build it.

This step is not required: for 3.0 (quilt) packages, dpkg-source is happy
with either patches-applied or patches-unapplied source directories,
as long as you're at one extreme or the other (applying half the patch
series doesn't work). Here's a smallish package that I happen to maintain
using gbp-pq, built without using gbp:

~/tmp$ debcheckout bubblewrap
...
~/tmp$ cd bubblewrap
~/tmp/bubblewrap$ dpkg-buildpackage -b
...
 dpkg-source --before-build .
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying debian/Use-Python-3-for-test-demo-code.patch
... a build happens ...
 dpkg-genchanges --build=binary >../bubblewrap_0.3.3-1_amd64.changes
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-source: info: unapplying debian/Use-Python-3-for-test-demo-code.patch
dpkg-buildpackage: info: binary-only upload (no source included)

(This probably only works for 3.0 (quilt) source packages, but gbp pq
makes little sense for any other format.)

> ([1] there are other ways to apply the patches but this is the
> easiest.)

Easier than the tool you have to use anyway (directly or indirectly)
to build any package?

smcv

Re: Preferred git branch structure when upstream moves from tarballs to git

[Jonathan Carter]

On 2019/05/08 13:23, Holger Levsen wrote:
>> You should use dgit for the benefit of users.  See my other mail which
>> answers why Vcs-Git and debcheckout is not enough.
> 
> could you be so kind and provide a pointer, this thread is rather long
> already? (Maybe this is also worth an FAQ entry somewhere..)

I'd like to second that request, I mean to go through this thread again
when I have some free moments, but a FAQ would help a lot to assimilate
all this information.

(I'm in a similar boat to Holger with my experience to gbp, I tried
going through the documentation on the wiki a few times and just got
frustrated. I just use packaging and Vcs-fields and tags and nothing
fancier than that, which works for the few dozen packages I have and I
think the workflow is trivial enough so that downstream consumers
wouldn't have any problem whatsoever figuring out how to make use of it,
but I suppose as that list grows it will become important to have
worflows that scale up a bit more.)

> thanks. (also for the work and thoughts you put into dgit!)

Ditto.

-Jonathan

-- 
  ⢀⣴⠾⠻⢶⣦⠀  Jonathan Carter (highvoltage) 
  ⣾⠁⢠⠒⠀⣿⡁  Debian Developer - https://wiki.debian.org/highvoltage
  ⢿⡄⠘⠷⠚⠋   https://debian.org | https://jonathancarter.org
  ⠈⠳⣄  Be Bold. Be brave. Debian has got your back.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Holger Levsen]

On Wed, May 08, 2019 at 12:18:41PM +0100, Ian Jackson wrote:
> I'm not sure what git branch format you are using.  (Maybe you said it
> earlier in this thread.)  Packaging-only ?  git-merge ?
 
both.

> You should use dgit for the benefit of users.  See my other mail which
> answers why Vcs-Git and debcheckout is not enough.

could you be so kind and provide a pointer, this thread is rather long
already? (Maybe this is also worth an FAQ entry somewhere..)

> > I *do* appreciate the idea of dgit and the work put into it, I think it
> > goes into the right direction, [...]
> Yes, I quite understand this point of view.

thanks. (also for the work and thoughts you put into dgit!)


-- 
tschau,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Holger Levsen writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git"):
> as a data point: I never really got my head around gbp, too many times
> it came in the way, did things I didnt expect to do (or couldnt easily
> figure out what it did), so I basically (try to) avoid packages which
> require gbp in some way.

Fair enough.

I'm not sure what git branch format you are using.  (Maybe you said it
earlier in this thread.)  Packaging-only ?  git-merge ?

> dgit *is* another layer of complexity on top of debuild and git (and other
> tools), as is gbp, and frankly (or sadly?), for the dgit case I dont see
> much benefit why I should use it rather than dput: my packages are
> maintained in git and have the correct Vcs-Git* headers set, so
> debcheckout will do the right thing and give you the full git history.

You should use dgit for the benefit of users.  See my other mail which
answers why Vcs-Git and debcheckout is not enough.

Indeed, you yourself say you avoid gbp but for many packages, the
Vcs-Git header gives you a patches-unapplied format which requires[1]
gbp pq to switch to a patches-applied view before you build it.

([1] there are other ways to apply the patches but this is the
easiest.)

> I *do* appreciate the idea of dgit and the work put into it, I think it
> goes into the right direction, but to me it's not sliced bread yet, but
> rather a fancy new machine for slicing bread which is more complicated
> and harder to grasp, so I'm still on that old slicing machine, which I
> know since many years and which I can easily fix if the bread gets stuck
> or the knifes need resharpening or some such.

Yes, I quite understand this point of view.

Regards,
Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Holger Levsen]

On Tue, May 07, 2019 at 07:25:39PM +0100, Ian Jackson wrote:
> A maintainer who is currently using gbp pq can (and IMO generally
> should) use `dgit push' to do their uploads, rather than dput/dupload.
 
as a data point: I never really got my head around gbp, too many times
it came in the way, did things I didnt expect to do (or couldnt easily
figure out what it did), so I basically (try to) avoid packages which
require gbp in some way.

dgit *is* another layer of complexity on top of debuild and git (and other
tools), as is gbp, and frankly (or sadly?), for the dgit case I dont see
much benefit why I should use it rather than dput: my packages are
maintained in git and have the correct Vcs-Git* headers set, so
debcheckout will do the right thing and give you the full git history.


I *do* appreciate the idea of dgit and the work put into it, I think it
goes into the right direction, but to me it's not sliced bread yet, but
rather a fancy new machine for slicing bread which is more complicated
and harder to grasp, so I'm still on that old slicing machine, which I
know since many years and which I can easily fix if the bread gets stuck
or the knifes need resharpening or some such.


-- 
tschau,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Sam Hartman writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git"):
> Ian Jackson  writes:
> > The latter point is because using dgit push is an ethical
> > imperative, not because the two somehow have some deep
> > technical linkage.  IMO almost *any* tutorial being written now
> > about how to do Debian packaging, and which mentions git at
> > all, ought to say to use dgit.
> 
> I think this is at the crux of our disagreement.
> I'm going to need to think about this for myself.

OK...

> I think it's safe to say there is not a project consensus that dgit is
> an ethical imperative.

Indeed.

> And yes understanding that you believe that's the case makes it a lot
> easier to understand the rest of your mail.

Good.

> As I said I'll need to think about whether I agree with that.  A lot of
> my thinking in this discussion is that I consider dgit a nice to have,
> not a requirement.

I guess I have stated the reasons already, but to recap:

 * Users should be able to easily change the software that runs
   on their computer.

 * "Easily" requires source code which:
  - can be built using a standard set of runes
  - will produce the same binaries as official Debian ones
  - can be reliably located
  - can be easily modified
   apt provides this, if you think tarballs+diffs are enough, but:

 * Nowadays people want the source code as the git history.
   When upstream and Debian are both using git, then arguably
   tarballs+diffs is not really the preferred form for modification.
   I.e. the .dsc source package is not really the source code.

>From these we can conclude:

 * Debian should provide source code as git branches which:
  - can be built using a standard set of runes
  - will produce the same binaries as official Debian ones 
  - can be reliably located
  - can be easily modified (using standard git commands)
  - contain the git histories we are actually using ourselves

There is only one way to do this.  It is `dgit push[-source]'.

Vcs-Git and Salsa do not provide this.  Occasionally people propose
alternative technical approaches.  So in theory there might be some
other way of achieving the goal.  But people have been proposing
technical approaches to this question for a very long time - a decade
at least.  Joey Hess and I came up with a design which was actually
implementable, politically practical, deployable and operable, and I
have implemented it, negotiated for it, deployed it, and am now
operating it.

dgit push involves minimal change to maintainer workflows and does not
gore anyone's ox, unlike all most of the (at best vapourware)
proposals before and since.


> If you accept the premis that everyone should be using dgit, I agree
> with you.
> I don't think the gbp documentation has adopted that premis.

Indeed.


> > The only reason dgit needs to get involved in your build
> > process is because of the .gitignore bug.  (#908747)
> 
> Well, and dgit wants to construct my dsc, which means it wants to
> construct my changes, which... means it wants to call sbuild or
> something else for me.

The only reason dgit wants to construct your dsc is #908747.  If you
can avoid #908747 some other way then you can build things however you
like and dgit push will work just fine. [1]

Of course if you are doing source-only uploads, as we should be doing
all the time (we're still shipping maintainers' binaries?!!11eleven)
then you do not need to really "build" anything and `dgit push-source'
suffices.  Of course it makes a .dsc but that is an affordance rather
than an inconvenience.


I recognise that your message was in some sense "I will go away and
think" rather than an invitation for me to further engage in advocacy
to you.  I hope you don't mind that I've done so anyway.  Feel free
not to reply.  But I did want to say these things for the benefit of
other readers.

And yes, do please file bugs if you disagree with dgit about things.
I welcome inchoate bug reports, although I don't promise not to reply
with "that doesn't sound right, please send steps to reproduce" :-).

Regards,
Ian.


[1] I would love it if dgit never needed to get involved in building.
There's tons of code in dgit to deal with the quirks of everyone's
favourite build tools.  I appreciate that wrapping the build tools in
yet another layer is very undesirable.

All of this is only necessary because of #908747.

Why didn't I try to get #908747 fixed ?  Well, it was one of dgit's
original principles that it would not require anyone else's permission
or assistance, at least not for anything remotely controversial.
Without that principle the project would have been totally stalled.

Looking at the history of #908747 you can see that I was right.

The result is that dgit contains many many ugly workarounds.  Each of
these 

Re: Preferred git branch structure when upstream moves from tarballs to git

[Scott Kitterman]

On May 7, 2019 8:50:57 PM UTC, Sam Hartman  wrote:
>>>>>> "Ian" == Ian Jackson  writes:
>
>Ian> Sam Hartman writes ("Re: Preferred git branch structure when
>Ian> upstream moves from tarballs to git"):
>>> I'm aware I'm making a typing error, and speaking in
>>> generalities.  I agree that my statement is true for debrebase,
>>> but I meant the dgit ecosystem.
>
>Thanks so much for both this  mail and your previous one.
>As an aside, frustration is part of the game.
>What I'm happy about is that we're working to respect each other and
>that we're reaching understanding.
>I expect to be frustrated; it's part of life and being passionate about
>what we do.
>I'm happy when people work with me as you're doing and we get past
>that.
>Thank you very much for working with me.
>
>
>Ian> The latter point is because using dgit push is an ethical
>Ian> imperative, not because the two somehow have some deep
>   Ian> technical linkage.  IMO almost *any* tutorial being written now
>Ian> about how to do Debian packaging, and which mentions git at
>Ian> all, ought to say to use dgit.
>
>I think this is at the crux of our disagreement.
>I'm going to need to think about this for myself.
>I think it's safe to say there is not a project consensus that dgit is
>an ethical imperative.
>And yes understanding that you believe that's the case makes it a lot
>easier to understand the rest of your mail.
>
>As I said I'll need to think about whether I agree with that.  A lot of
>my thinking in this discussion is that I consider dgit a nice to have,
>not a requirement.

For what it's worth, when you start calling me names (unethical) because I'm 
not using your shiny new toy, my immediate reaction is to decide to ignore 
further discussion on the topic.

Scott K

Re: Preferred git branch structure when upstream moves from tarballs to git

[Sam Hartman]

>>>>> "Ian" == Ian Jackson  writes:

Ian> Sam Hartman writes ("Re: Preferred git branch structure when
Ian> upstream moves from tarballs to git"):
>> I'm aware I'm making a typing error, and speaking in
>> generalities.  I agree that my statement is true for debrebase,
>> but I meant the dgit ecosystem.

Thanks so much for both this  mail and your previous one.
As an aside, frustration is part of the game.
What I'm happy about is that we're working to respect each other and
that we're reaching understanding.
I expect to be frustrated; it's part of life and being passionate about
what we do.
I'm happy when people work with me as you're doing and we get past that.
Thank you very much for working with me.


Ian> The latter point is because using dgit push is an ethical
Ian> imperative, not because the two somehow have some deep
Ian> technical linkage.  IMO almost *any* tutorial being written now
Ian> about how to do Debian packaging, and which mentions git at
Ian> all, ought to say to use dgit.

I think this is at the crux of our disagreement.
I'm going to need to think about this for myself.
I think it's safe to say there is not a project consensus that dgit is
an ethical imperative.
And yes understanding that you believe that's the case makes it a lot
easier to understand the rest of your mail.

As I said I'll need to think about whether I agree with that.  A lot of
my thinking in this discussion is that I consider dgit a nice to have,
not a requirement.


>> Dgit and debrebase are not really separate in terms of teaching.
>> If you look at the documentation for debrebase, you'll find that
>> there are a lot of cross references from debrebase docs to dgit.

Ian> dgit and gbp pq are not separate in terms of teaching either!

If you accept the premis that everyone should be using dgit, I agree
with you.
I don't think the gbp documentation has adopted that premis.

>> Dgit is harder overall even if you ignore debrebase because it
>> has a lot of moving parts that in my experience sometimes fail
>> and because dgit wants to get involved in your build process,
>> wants to be aware of your patch management, etc.

Ian> When dgit fails it is, almost always, because the "source code"
Ian> you are uploading to the Debian archive is not the same as what
Ian> you are actually working with as the source code in your own
Ian> workflow.

For me I'm finding I need to include or not include some overwrites, or
running into trouble with keys or random stuff like that.
I find it takes me two or three attempted pushes to get it right and
about half the time dgit's right that I should adjust something and half
the time I'm fairly sure I disagree with it.
And yeah, I should file bugs, and I have sometimes, and will file more
in the future.

Ian> The only reason dgit needs to get involved in your build
Ian> process is because of the .gitignore bug.  (#908747)

Well, and dgit wants to construct my dsc, which means it wants to
construct my changes, which... means it wants to call sbuild or
something else for me.


Anyway, thanks; I think your message clarified where I need to think
more.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Sam Hartman writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git"):
> I'm aware I'm making a typing error, and speaking in generalities.  I
> agree that my statement is true for debrebase, but I meant the dgit
> ecosystem.

I don't think this "the dgit ecosystem" is a helpful framing.  It is
very misleading, not to say entirely false.  Ecosystems are about
interactions, notably synergies.  dgit synergises about as well with
git-dpm or gbp pq or indeed raw git as it does with git-debrebase.

So "git-debrebase" is not part of "the dgit ecosystem".  "The dgit
ecosystem" is primarily git, gbp pq, and git-dpm.  git-debrebase is a
minority interest for dgit (because many people use gbp pq and few
people use git-debrebase).

 dgit   git-debrebase

Direct   dput, dupload, gbp pq, git-dpm
Competitors  apt-get source git-merge

Works well   gbp pq, git-dpm,   git
with git-merge, git

Who should   Everyone who can   Brave people
use it, for  who like it
which packages   Hopefully, Packages with
  all packages   nontrivial delta

Adoption FUDMaintainer needs
barriers Bugs in .dsc tools  to completely
 Maintainer selfishness   change workflow

Primary  Users and  Maintainers
beneficiaries downstreamswithin Debian
of adoption   outside Debian

Opinionated  To the most minimalCompletely
  extent possible

Maturity Very matureQuite new

Ethics ofUse of "dgit push" Few ethical
adoption  is IMO an ethical  implications
   imperative

Team adoptionEach individualWhole team
 decision uploader can adopt must agree
   dgit push, or not

Repository   No change needed,  Must convert branch
 conversion   use existing masterfrom previous tool

Future,  Obsolete when  Rather better
after source  source packagesif no source
packages  go awaypackages needed


Lumping these two things together with some kind of "ecosystem" label,
does more to obscure than it does to illuminate.

Basically the only things they have in common are:
  * They are to do with git and Debian source code
  * I wrote them
  * The tutorials for git-debrebase say to use dgit

The latter point is because using dgit push is an ethical imperative,
not because the two somehow have some deep technical linkage.  IMO
almost *any* tutorial being written now about how to do Debian
packaging, and which mentions git at all, ought to say to use dgit.


> Dgit and debrebase are not really separate in terms of teaching.
> If you look at the documentation for debrebase, you'll find that there
> are a lot of cross references from debrebase docs to dgit.

dgit and gbp pq are not separate in terms of teaching either!

There are just as many cross-references from dgit-maint-gbp(7) to dgit
documentation as there are from dgit-maint-debrebase(7).


> Dgit is harder overall even if you ignore debrebase because it has a lot
> of moving parts that in my experience sometimes fail and because dgit
> wants to get involved in your build process, wants to be aware of your
> patch management, etc.

When dgit fails it is, almost always, because the "source code" you
are uploading to the Debian archive is not the same as what you are
actually working with as the source code in your own workflow.

Other tools don't notice this (or even have special code to achieve
it!)  IMO this is a violation of our principles which you should be
concerned to fix and which you should applaud a tool for spotting.

The only reason dgit needs to get involved in your build process is
because of the .gitignore bug.  (#908747)

> git-dpm is harder than debrebase because it is less polished and
> involves more explicit branches in some ways.

I have little interest in advocating git-debrebase.  Obviously I think
it's great, and I may advocate its use in a team I'm in, but if other
people disagree then fine.

> Please understand I think dgit and debrebase are great technologies.
> I'm moving towards using them more and more, and when I'm acting as a
> downstream rather than a Debian developer, dgit clone is the best thing
> since sliced bread.

Thanks for the compliment.

But, you will have noticed that dgit clone sometimes doesn't give you
a good history.  That happens when the maintainer uses dput or dupload
instead of dgit push.

That is why I am engaging in this thread.  I want to see `dgit clone'
produce the best answer much m

Re: Preferred git branch structure when upstream moves from tarballs to git

[Sune Vuorela]

On 2019-05-07, Ansgar  wrote:
> The concept behind 3.0 (quilt) is much easier to explain: extract
> tarballs, optionally apply some patches provided.  With the bonus that
> users who have used other distributions before might already know about
> most of this which lowers barrier of entry.  Implementation details
> like constructing a .dsc are indeed messy.

I currently work with some yocto guys trying to get some of their
recipes adapted to better multilib support. I've been pointing to
debian/rules and debian/patches/* as places to find inspiration for
"weird things". The fact that the packages in question have been in 3.0
(quilt) with either a only-debian repository or full upstream source
have been very easy to glance over.

So from an external-ish PoV, current state of things is very nice. 

But of course, it is not our primary customers. But probably our
secondary ones.

/Sune

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Sam Hartman writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git"):
> I was going to argue that I thought teaching gbp pq was easier than
> dgit, and that's true up until you have to update a patch.

I'm sorry not to really deal with your whole mail, but this (and a few
more statements like it in your email) is a category error.

dgit is not an alternative to gbp pq.  dgit is (amongst other things)
a tool a maintainer can use *together with* gbp pq, to publish their
git history.

A maintainer who is currently using gbp pq can (and IMO generally
should) use `dgit push' to do their uploads, rather than dput/dupload.

Ansgar was pointing to the git-debrebase manpage.  git-debrebase *is*
an alternative to gbp pq.  Both git-debrebase and gbp pq are
alternatives to git-dpm and to use of raw quilt.


Block diagrams:

For the maintainer:

  gbp pq  |  git-debrebase  |  git-dpm  |  quilt  | ...
   -
 dgit push[-source] |   dupload / dput  

   Maintainer must choose one from top row ("git workflow"), and
   then one from the bottom row ("upload/publication tool").
   Most but not yet quite all gitish workflows are supported by
   dgit push.

What I am primarily advocating for in this thread is that maintainers
should choose `dgit push-source' over `dput' (where this is possible).
This is the only way for a maintainer to provide users with the git
history in a sensible form (ie, a form which does not require the user
to know what special git practices the maintainer has adopted).


For the user/downstream:

   git   ||   git + quilt, gbp, ???   ||  dpkg-source
 ||(depends on package)   ||   quilt
   --||---||--
   dgit clone||git clone salsa||  apt-get
/fetch   ||aka debcheckout||source

   User must choose one stack.


For the NMUer:

   git   ||   git + quilt, gbp, ???   ||  dpkg-source
 ||(depends on package)   ||   quilt
   --||---||
   dgit clone||   debcheckout ||  apt-get source
   ---||
   dgit push |dput / dupload  ||  dput

   NMUer must choose a stack, although if they use a gitish
   workflow they can mix-and-match upload tools.


Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Sam Hartman]

> "Ian" == Ian Campbell  writes:

Ian> On Tue, 2019-05-07 at 11:01 -0400, Sam Hartman wrote:
>> That said, I think Ansgar has some really valid points.  I think
>> that dgit (or git-dpm) are the hardest work flows to teach.

Ian> I think perhaps you meant `git-debrebase` rather than `dgit`
Ian> throughout the remainder of your mail, since comparing `dgit`
Ian> and `git-dpm` is (AFAIUI) a type error of sorts since they are
Ian> not serving the same purpose -- `dgit` replaces `dput` not your
Ian> patch managment system so comparing it to `git-dpm`, a patch
Ian> management system, is somewhat apples to oranges.

I'm aware I'm making a typing error, and speaking in generalities.  I
agree that my statement is true for debrebase, but I meant the dgit
ecosystem.

git-dpm is harder than debrebase because it is less polished and
involves more explicit branches in some ways.
Dgit is harder overall even if you ignore debrebase because it has a lot
of moving parts that in my experience sometimes fail and because dgit
wants to get involved in your build process, wants to be aware of your
patch management, etc.

Dgit and debrebase are not really separate in terms of teaching.
If you look at the documentation for debrebase, you'll find that there
are a lot of cross references from debrebase docs to dgit.
For example if you want to  talk about handling new upstreams  or orig
tarballs with debrebase you need to talk about something else--either
from the dgit ecosystem or the gbp ecosystem or something.

Please understand I think dgit and debrebase are great technologies.
I'm moving towards using them more and more, and when I'm acting as a
downstream rather than a Debian developer, dgit clone is the best thing
since sliced bread.  Since I've mostly stopped eating bread perhaps I
should come up with a new metaphor, but it's really neat regardless of
what metaphor I use.

Still, this stuff is hard.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ansgar]

On Tue, 2019-05-07 at 12:51 +0100, Ian Jackson wrote:
> Ansgar Burchardt writes ("Re: Preferred git branch structure when
> upstream moves from tarballs to git"):
> > Sam Hartman  writes:
> > > OK, I didn't hear that as an answer but think I'm coming to the
> > > same
> > > conclusion that Ian did after reading your mail.
> > > It sounds like you are talking about having the Debian packaging
> > > in a
> > > separate repository from upstream.
> > > Do I correctly understand the model you are talking about?
> ...
> > I'll point to [1] for what complexity this avoids.  Try explaining
> > that
> > to someone without advanced Git knowledge...
> > 
> >   [1] 
> > https://manpages.debian.org/unstable/git-debrebase/git-debrebase.5.en.html
[...]

> In this message you are using the fact that I have written a
> comprehensive data model specification, against me.  This is a
> seriously bad thing to do.
> 

I think you underestimate how much knowledge you require from users
right from the beginning...

So in a way the problem is that the documentation needs to exist.  git-
debrebase or dgit (to a lesser extend) implement a Debian-specific
version control system on top of Git.  git-debrebase makes this most
obvious: the Git history it generates is pretty much unlike anything
you would ever get from regular Git usage (counterexamples welcome), it
mangles your Git commits into a specific format it expects
(laundering), mangles rebased branches into fast-forward merges with
pseudomerges; all this does not happen in regular Git usage.

dgit has less of these problems, but still does things different from
regular git usage (again pseudomerges to make rebases fast-forward for
example); there are other issues as well such as its suggested use for
NMUs feeling rather unfriendly as it moves the packaging to another VCS
than the maintainer uses with a disjoint history...

Sure, dealing with patch series is not nice either way, but I don't
think the added complexity of git-debrebase (or dgit) is worth what it
provides.  Alternatives require less intimiate knowledge of git and
provide a gentler learning curve; some basic patch queue manipulation
even doesn't require special tools at all (adding/removing patches that
already exist in the right format and apply).

Other distributions have intentionally be trying to move to less
distro-specific tools to reduce barrier of entry; dgit and git-
debrebase both move in the opposite direction (more highly distro-
specific tools).

> You should be applauding me for providing serious documentation for
> advanced users.

According to git-debrebase(1) the documentation I referred to is basic
documentation:

> You should read this manpage in cojnunction with "TERMINOLOGY" in 
> git-debrebase(5), which defines many important terms used here.

"TERMINOLOGY" then refers to most other sections.

> The obvious counter is the (still comprehensive, but user-facing)
> tutorial manpage.  For example,
>   
> https://manpages.debian.org/stretch-backports/dgit/dgit-maint-debrebase.7.en.html#EDITING_THE_DEBIAN_PACKAGING
> et seq.

It illustrates my other main gripe with dgit/git-debrebase: it makes it
harder to share in-progress work, in fact git-debrebase discourages
people from doing so:

| Note that each time you conclude a debrebase you introduce a
| pseudomerge into your git history, which may make it harder to read.

| A simple convention you can use to minimise the number of
| pseudomerges is to git debrebase conclude only right before you
| upload or push to salsa.debian.org. 

Yes, one can avoid some of the problems by pushing a non-fast-forward,
non-interchange branch.  But that differs from the regular workflow
and, again, requires more advanced Git knowledge.

> An equivalently detailed and frank document about dpkg-source would
> be a nightmare.

The concept behind 3.0 (quilt) is much easier to explain: extract
tarballs, optionally apply some patches provided.  With the bonus that
users who have used other distributions before might already know about
most of this which lowers barrier of entry.  Implementation details
like constructing a .dsc are indeed messy.

With most other Debian packaging workflows one can also limit oneself
to more basic Git commands for regular usage, in the most minimalistic
case only commit/push/pull (not even merge); these also match what
users will know if they have used Git at all (be it for packaging in
other distribution, upstream projects, or even just writing their
thesis).

You might say that git-debrebase is not required, but what about users
that want to contribute to a package maintained using it?  They will
have to deal with the complexity...  This also differs from other
packaging workflows that usually don't make specific tools mandatory.

(Not all tools can always be made optional of course, but one should do
some consideration if one makes new tools mandatory.)

Ansgar

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Campbell]

On Tue, 2019-05-07 at 11:01 -0400, Sam Hartman wrote:
> That said, I think Ansgar has some really valid points.  I think that
> dgit (or git-dpm) are the hardest work flows to teach.

I think perhaps you meant `git-debrebase` rather than `dgit` throughout
the remainder of your mail, since comparing `dgit` and `git-dpm` is
(AFAIUI) a type error of sorts since they are not serving the same
purpose -- `dgit` replaces `dput` not your patch managment system so
comparing it to `git-dpm`, a patch management system, is somewhat
apples to oranges.

Ian.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Sam Hartman]

>>>>> "Ian" == Ian Jackson  writes:

Ian> Ansgar Burchardt writes ("Re: Preferred git branch structure
    Ian> when upstream moves from tarballs to git"):
>> Sam Hartman  writes: > OK, I didn't hear
>> that as an answer but think I'm coming to the same > conclusion
>> that Ian did after reading your mail.  > It sounds like you are
>> talking about having the Debian packaging in a > separate
>> repository from upstream.  > Do I correctly understand the model
>> you are talking about?
Ian> ...
>> I'll point to [1] for what complexity this avoids.  Try
>> explaining that to someone without advanced Git knowledge...
>> 
>> [1]
>> 
https://manpages.debian.org/unstable/git-debrebase/git-debrebase.5.en.html

Ian> I am getting increasingly unhappy with your tone, and with the
Ian> thrust of your arguments, in this thread.  Your messages feel
Ian> very hostile to 

I'd like to try and step in.

Ian, it sounds like you're frustrated.
I'd like to understand exactly why.

Is it that  you're frustrated  because you've spent a lot of time
working on documenting what you think is the simplest, best option, and
others are disagreeing?

I do find the work you've done valuable.  But like Ansgar I find that it
demonstrates there is a lot of complexity in the dgit work flows.  I
don't think this is a criticism of your work; I think that the reason
people are pointing to your work is that it clearly demonstrates the
complexity.

I also know that it's easy to sweep the complexity that we're familiar
with under the rug.  I hear your concern that people advocating other
approaches may be hiding complexity and that we need to be careful not
to hide complexity simply because it is undocumented.


That said, I think Ansgar has some really valid points.  I think that
dgit (or git-dpm) are the hardest work flows to teach.
One of my coworkers in a debian developer.
I could teach him dgit (although I think he already knows it
reasonably).

I would not expect to succeed at teaching dgit or debrebase to the other
two engineers at my company, although I'm kind of tempted to try as a
discussion point.

I was going to argue that I thought teaching gbp pq was easier than
dgit, and that's true up until you have to update a patch.

I do think I could teach the other engineers separate debian repository.
That's certainly true assuming that upstream doesn't need to be changed.
I think that so long as I said redo your patches from scratch every new
upstream release I could even teach separate debian repositories.

By teach dgit I mean teach dgit well enough to add an upstream patch,
and migrate that across to upstream releases as they change.

I think that patches applied git workflows have a lot to add.  I know I
don't want separate debian repositories as my preferred workflow.  But I
do think there really is a lot of complexity here, and I think that it
is a lot easier to get someone started with some of the simpler
workflows.

That said, I also hear your argument that for downstreams who are
want to just add a patch or two (and throw it all away and start over
when they take a new upstream), patches applied workflows are easier.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Ansgar Burchardt writes ("Re: Preferred git branch structure when upstream 
moves from tarballs to git"):
> Sam Hartman  writes:
> > OK, I didn't hear that as an answer but think I'm coming to the same
> > conclusion that Ian did after reading your mail.
> > It sounds like you are talking about having the Debian packaging in a
> > separate repository from upstream.
> > Do I correctly understand the model you are talking about?
...
> I'll point to [1] for what complexity this avoids.  Try explaining that
> to someone without advanced Git knowledge...
> 
>   [1] 
> https://manpages.debian.org/unstable/git-debrebase/git-debrebase.5.en.html

I am getting increasingly unhappy with your tone, and with the thrust
of your arguments, in this thread.  Your messages feel very hostile to
me.

In this message you are using the fact that I have written a
comprehensive data model specification, against me.  This is a
seriously bad thing to do.  You should be applauding me for providing
serious documentation for advanced users.

The obvious counter is the (still comprehensive, but user-facing)
tutorial manpage.  For example,
  
https://manpages.debian.org/stretch-backports/dgit/dgit-maint-debrebase.7.en.html#EDITING_THE_DEBIAN_PACKAGING
et seq.

An equivalently detailed and frank document about dpkg-source would be
a nightmare.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ansgar Burchardt]

Sam Hartman  writes:
>> "Ansgar" == Ansgar   writes:
>
> Ansgar> Sam Hartman  writes:
> >> I'm having a bit of trouble here and am hoping you can help us
> >> out.  Ian asked what git workflow it is that you're talking about
> >> where people can deal with commit push and pull and don't need to
> >> know more.
> >> 
> >> I didn't see a clear answer to that.  Which debian packaging
> >> workflow do you believe has that property?
>
> Ansgar> There is no need for a vendor branch if only the packaging
> Ansgar> information is kept in the repository, i.e. only debian/.
> Ansgar> In particular there is no need for branches or merges for
> Ansgar> regular updates (i.e. not based on an older release).
>
> OK, I didn't hear that as an answer but think I'm coming to the same
> conclusion that Ian did after reading your mail.
> It sounds like you are talking about having the Debian packaging in a
> separate repository from upstream.
> Do I correctly understand the model you are talking about?

Sure.

I'll point to [1] for what complexity this avoids.  Try explaining that
to someone without advanced Git knowledge...

  [1] https://manpages.debian.org/unstable/git-debrebase/git-debrebase.5.en.html

Note that CVS already had a mechanism for tracking upstream releases
(vendor branches); I'm not sure how widely they were used, but as far as
I understand they were usually seen as advanced use of CVS.  Git makes
maintaining such vendor branches easier (better merging), and some
workflows for Debian packaging result in trivial merges, but it is very
confusing when something unexpected happens.

Ansgar

Re: Preferred git branch structure when upstream moves from tarballs to git

[Sam Hartman]

> "Ansgar" == Ansgar   writes:

Ansgar> Sam Hartman  writes:
>> I'm having a bit of trouble here and am hoping you can help us
>> out.  Ian asked what git workflow it is that you're talking about
>> where people can deal with commit push and pull and don't need to
>> know more.
>> 
>> I didn't see a clear answer to that.  Which debian packaging
>> workflow do you believe has that property?

Ansgar> There is no need for a vendor branch if only the packaging
Ansgar> information is kept in the repository, i.e. only debian/.
Ansgar> In particular there is no need for branches or merges for
Ansgar> regular updates (i.e. not based on an older release).

OK, I didn't hear that as an answer but think I'm coming to the same
conclusion that Ian did after reading your mail.
It sounds like you are talking about having the Debian packaging in a
separate repository from upstream.
Do I correctly understand the model you are talking about?

I'm not trying to be dense here.  I'm 85% sure I'm correct here but
would like te bring that up to 100%

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ansgar]

Sam Hartman  writes:
> I'm having a bit of trouble here and am hoping you can help us out.
> Ian asked what git workflow it is that you're talking about where people
> can deal with commit push and pull and don't need to know more.
>
> I didn't see a clear answer to that.
> Which debian packaging workflow do you believe has that property?

There is no need for a vendor branch if only the packaging information
is kept in the repository, i.e. only debian/.  In particular there is no
need for branches or merges for regular updates (i.e. not based on an
older release).

This is also what pretty much every other distribution seems to be
doing, for example Fedora, Gentoo, Ports (BSD), Arch Linux, Void Linux.

It would be easier if there was even more separation between packaging
and upstream source, e.g. base/src (upstream source), base/debian
(packaging information), and possibly other directories below base for
build artifacts (instead of unpredictable locations under base/debian).
Which leads back to the beginning of the subthread[1].

  [1] https://lists.debian.org/debian-devel/2019/04/msg00462.html

Ansgar

Re: Preferred git branch structure when upstream moves from tarballs to git

[Sam Hartman]

>>>>> "Ansgar" == Ansgar   writes:

Ansgar> On Fri, 2019-05-03 at 17:39 +0100, Ian Jackson wrote:
>> Ansgar writes ("Re: Preferred git branch structure when upstream
>> moves from tarballs to git"):
>> > On Fri, 2019-05-03 at 15:59 +0100, Ian Jackson wrote:
    >> > > Ansgar writes ("Re: Preferred git branch structure when
>> upstream > > moves from tarballs to git"): > > > How do you
>> update the package to a new upstream release?
>> > > 
>> > > The same way you would with whatever tools you are currently
>> > > using for > > that task.  dgit is not a git delta management
>> tool.  For the > > maintainer, dgit replaces dput, not gbp pq or
>> git-dpm or quilt.
>> > 
>> > The question was to illustrate that dgit forces you to learn
>> about > branches, merging, ...  There is no workflow with dgit
>> that doesn't > require this; when not using dgit, this isn't
>> required.
>> 
>> This is so wrong it is very perplexing to me.

Ansgar> Yet you confirm exactly what I am saying...

I'm having a bit of trouble here and am hoping you can help us out.
Ian asked what git workflow it is that you're talking about where people
can deal with commit push and pull and don't need to know more.

I didn't see a clear answer to that.
Which debian packaging workflow do you believe has that property?

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Ben Finney writes ("Re: Preferred git branch structure when upstream moves from 
tarballs to git"):
> Ian Jackson  writes:
> > [...] I have not taught dgit how to convert [separate Debian
> > packaging-only repository and upstream source repository] into a git
> > tree that contains the [combined] source code.
> 
> One interpretation of `convert [separate repositories] into [one Git
> tree]' is that it's a flag-day, point of no return conversion that
> precludes the separate-Debian-packaging-repository workflow from that
> point forward. If so, that wouldn't be enough to convince me to use
> DGit, because I consider the separate-Debian-packaging-repository
> workflow superior for many cases.

Sorry that I have apparently been unclear.

That's not how dgit copes with "special" maintainer git practices.
Rather, the maintainer's thing (whatever that is) is converted anew
*at each upload* (ie at each dgit push).  The converted version is
published to dgit users (using an appropriate history structure[1].

The maintainer does not see the output of the conversion, unless they
go looking for it.[2]  These conversions operate in "split view mode"
where the maintainer's branch(es) are left alone.

Indeed you can try dgit for one upload and go back to using dput for
your next upload.  You can even have a packaging team some of whose
members use dgit push and some don't.  [3]

> Contrariwise, on the assumption we are instead talking about a
> `conversion to a Git tree' that would seamlessly let DGit support the
> ongoing workflow of Debian packaging in a repository separate from any
> upstream code:
> 
> > That feature would be the wishlist bug
> >   #903392 "want support for packaging-only maintainer views"
> > which I've already mentioned once in this thread, where I explained
> > why implementing it is a low priority.
> 
> While acknowledging you are under no obligation to implement features
> demanded by others, I'll point out that for as long as DGit lacks
> support for that workflow, you won't get much user feedback from people
> whose Debian packages need that workflow and therefore are not
> knowledgeable DGit users.

Are you a user who uses this workflow, and who would strongly consider
using dgit if this git tree format were properly supported ?

If so then I would like to talk to you more.
Particular questions I would like to ask are:
  * Do you have the upstream source code available in git form ?
If you do then it should be used, since that is much better
than a tarball import.  How might it be found, automatically ?
  * If so, is the .orig tarball you are using exactly identical to
what you have in git ?
Ideally, CC the answers to #903392 :-).

(If we can't get the upstream source in git form then the the result,
for a user of `dgit clone', would be significantly less useful: it
would still involve tarball imports.  The real benefits emerge when
the user can not just `git grep' the whole thing, but get a useful
`git log' that shows something other than "imported upstream 1.2.3",
can `git merge' some other upstream code, etc.)

> So for as long as that persists, it will take special consious effort to
> remember that there's a class of Debian package maintainers who would
> like to become knowledgeable about DGit but can't yet; that the current
> DGit user base is not representative of that class of potential user.
> 
> I trust you already know this, but it bears explicit expression from
> time to time :-)

I'm aware of this, thanks.  So far I have gained the impression that
the kind of people who are using packaging-only git trees tend to be
very conservative, and very comfortable with .dsc/tarball/patch/quilt
based tools, and not really convinced of the usefulness of git.
I have even experienced what feels to me like significant hostility.

If I am wrong and there are more users to be had by implementing this
feature then I will definitely consider giving it a higher priority.

BTW, dgit is capitalised thus.

Regards,
Ian.

[1] The dgit view history for an upload is arranged so that it is ff
from the converted version of the previous one upload; and it has the
maintainer's history as an ancestor (which makes git log etc. work).

[2] The git objects representing the conversions are retained in the
maintainer's git tree, but they are reachable there only from
dgit-specific refs.  If the maintainer deletes those refs nothing will
go wrong, but if those objects were garbage collected a future dgit
push would end up downloading them again from the dgit git server,
which is not desirable.

[3] If you are *not* using the split view feature, your dgit push will
usuall involve stitching the dgit history (with perhaps a .dsc import
or two) into your branch.  If you then go back to using dput that will
remain harmlessly in your hi

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ben Finney]

Ian Jackson  writes:

> […] I have not taught dgit how to convert [separate Debian
> packaging-only repository and upstream source repository] into a git
> tree that contains the [combined] source code.

One interpretation of “convert [separate repositories] into [one Git
tree]” is that it's a flag-day, point of no return conversion that
precludes the separate-Debian-packaging-repository workflow from that
point forward. If so, that wouldn't be enough to convince me to use
DGit, because I consider the separate-Debian-packaging-repository
workflow superior for many cases.

Contrariwise, on the assumption we are instead talking about a
“conversion to a Git tree” that would seamlessly let DGit support the
ongoing workflow of Debian packaging in a repository separate from any
upstream code:

> That feature would be the wishlist bug
>   #903392 "want support for packaging-only maintainer views"
> which I've already mentioned once in this thread, where I explained
> why implementing it is a low priority.

While acknowledging you are under no obligation to implement features
demanded by others, I'll point out that for as long as DGit lacks
support for that workflow, you won't get much user feedback from people
whose Debian packages need that workflow and therefore are not
knowledgeable DGit users.

So for as long as that persists, it will take special consious effort to
remember that there's a class of Debian package maintainers who would
like to become knowledgeable about DGit but can't yet; that the current
DGit user base is not representative of that class of potential user.

I trust you already know this, but it bears explicit expression from
time to time :-)

-- 
 \ “Our task must be to free ourselves from our prison by widening |
  `\our circle of compassion to embrace all humanity and the whole |
_o__)   of nature in its beauty.” —Albert Einstein |
Ben Finney

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ansgar]

On Fri, 2019-05-03 at 17:39 +0100, Ian Jackson wrote:
> Ansgar writes ("Re: Preferred git branch structure when upstream
> moves from tarballs to git"):
> > On Fri, 2019-05-03 at 15:59 +0100, Ian Jackson wrote:
> > > Ansgar writes ("Re: Preferred git branch structure when upstream
> > > moves from tarballs to git"):
> > > > How do you update the package to a new upstream release?
> > > 
> > > The same way you would with whatever tools you are currently
> > > using for
> > > that task.  dgit is not a git delta management tool.  For the
> > > maintainer, dgit replaces dput, not gbp pq or git-dpm or quilt.
> > 
> > The question was to illustrate that dgit forces you to learn about
> > branches, merging, ...  There is no workflow with dgit that doesn't
> > require this; when not using dgit, this isn't required.
> 
> This is so wrong it is very perplexing to me.

Yet you confirm exactly what I am saying...

> This does not work right now with dgit because I have not taught dgit
> how to convert such a thing into a git tree that contains the actual
> source code.

I'm not talking about a hypothetical, different version of dgit which
might have a different set of features.  Anything is possible for such
a hypothetical version.

> Using dgit is a significantly higher entry barrier than not using
> > it;
> 
> Using dgit is a much lower entry barrier if you already know git, and
> don't know about Debian source packages.  Like most people nowadays.

I know plenty of people who more or less only know git commit, push and
pull.  Nothing more advanced.

> Using dgit is only a higher entry barrier if (i) you aren't
> comfortable with "intermediate" use of git (ii) you are already an
> expert on dpkg-source, quilt, etc.

Thank you for confirming that dgit raises entry barriers.  I still
believe that is opposite of what distributions should do, see the
popularity of AUR or similar projects.

> Any maintainer who feels (as I often do) that the preferred form for
> modification of their package includes the full git history, should
> already be using dgit, since how else can they always discoverably
> publish the complete corresponding source ?

There is this fancy Vcs-Git field.  Simple to discover.

Now, if dgit could just use that repository...

> [1] Possibly the user will get the maintainer's git history plus some
> autogenerated dgit conversion commits.  For example, for a package
> whose maintainer uses gbp pq, the user gets a thing that has many
> similarities to the gbp pq patch queue branch.  In particular, the
> patches are applied, each as a git commit in the user's history.

And then has to take care to not push those artificial commits
somewhere if that is not the preferred workflow... That seems rather
backwards to me.

Ansgar

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Ansgar writes ("Re: Preferred git branch structure when upstream moves from 
tarballs to git"):
> On Fri, 2019-05-03 at 15:59 +0100, Ian Jackson wrote:
> > Ansgar writes ("Re: Preferred git branch structure when upstream moves from 
> > tarballs to git"):
> > > How do you update the package to a new upstream release?
> > 
> > The same way you would with whatever tools you are currently using for
> > that task.  dgit is not a git delta management tool.  For the
> > maintainer, dgit replaces dput, not gbp pq or git-dpm or quilt.
> 
> The question was to illustrate that dgit forces you to learn about
> branches, merging, ...  There is no workflow with dgit that doesn't
> require this; when not using dgit, this isn't required.

This is so wrong it is very perplexing to me.

Maybe Russ has it, when he says:

  Are you using a separate repository for only the debian directory and
  merging that with an unpacked upstream tarball without using a version
  control system to do so?  I don't know how that works with dgit if
  that's the case.

?

In answer to Russ's implied question.

This does not work right now with dgit because I have not taught dgit
how to convert such a thing into a git tree that contains the actual
source code.  That feature would be the wishlist bug
  #903392 "want support for packaging-only maintainer views"
which I've already mentioned once in this thread, where I explained
why implementing it is a low priority.

If Russ has described guessed right then even so it is *not* true that
"dgit forces you to learn about branches, merging,...".

It's just that dgit does not yet know how to convert your git branch
into the kind of git branch that other people want.  If it did (that
would be me implementing #903392) then you would still not need to
learn anything you do not already know.  You would just say `dgit
push', from your existing working tree, and dgit would do its thing
automatically.


Maybe you are interpreting "please use dgit" to include "please switch
to a git tree containing the upstream source code".  That is not
correct.

It is true that switching to a git tree containing the upstream source
code (and the upstream git history) does involve multiple branches.
While obviously Russ and I think git trees should containin the
upstream source code, no-one is asking you to do that "because dgit".


> Using dgit is a significantly higher entry barrier than not using it;

Using dgit is a much lower entry barrier if you already know git, and
don't know about Debian source packages.  Like most people nowadays.

Indeed dgit's key benefit is that it helps you avoid having to learn
about Debian source packages.  This is useful because Debian source
packages are, by modern standards, crazy, and seriously defective.

Using dgit is only a higher entry barrier if (i) you aren't
comfortable with "intermediate" use of git (ii) you are already an
expert on dpkg-source, quilt, etc.


> > If you are trying to get the source code, there is a choice to be
> > made.
> > 
> > You can choose to
> > 
> >  (1) Get a predictable standardised tree format, which can directly be
> >  built, cherry-picked, etc.,
> > 
> >  And, always have what is actually in the archive.
> > 
> >  But maybe an unhelpful history.  Especially, if the maintainer
> >  does not use dgit push.
> > 
> >  (dgit clone)
> 
> This doesn't give you the real source according to people who state
> that "source code" nowadays means the Git repository used by the
> maintainer.  dgit doesn't differ from apt-get source & gbp import-dsc
> there, at least I don't see one.

Yes, people who state that the source code nowadays usually means the
git repository consider that a Debian source package is not the proper
source code either.  The Debian "source" archive is not really a
source archive.

Any maintainer who feels (as I often do) that the preferred form for
modification of their package includes the full git history, should
already be using dgit, since how else can they always discoverably
publish the complete corresponding source ?

> (This might be different for packages where the maintainer is using
> dgit, didn't forget to push and didn't make any changes since the last
> upload, i.e. when debcheckout would give the same result.)

If the maintainer uses `dgit push' then the user who uses `dgit clone'
gets the maintainers' git history. [1]  That's the whole point.

Whether they `forgot to push' (to salsa I guess) is not very relevant.
The source code that ought to be published along with the .debs is not
some random other more recent version.  It is the source code that
produced the .debs that the user is running.  dgit gives the user
exactly that.

[1

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ansgar]

On Fri, 2019-05-03 at 15:59 +0100, Ian Jackson wrote:
> Ansgar writes ("Re: Preferred git branch structure when upstream moves from 
> tarballs to git"):
> > On Thu, 2019-05-02 at 09:15 -0700, Russ Allbery wrote:
> > > Ansgar  writes:
> > > > Having to know about branches, merging, dealing with multiple remotes,
> > > > ... *is* an entry barrier compared to not having to know about it.  Now
> > > > you have to teach people that before you even get to how to write a
> > > > build recipe.
> > > 
> > > I'm confused.  I'm a happy user of dgit and don't have to think about any
> > > of those things as part of using dgit.  I choose to use branches, but I
> > > certainly wouldn't have to, and merging, multiple remotes, and so forth
> > > don't seem related to using dgit at all.
> > 
> > How do you update the package to a new upstream release?
> 
> The same way you would with whatever tools you are currently using for
> that task.  dgit is not a git delta management tool.  For the
> maintainer, dgit replaces dput, not gbp pq or git-dpm or quilt.

The question was to illustrate that dgit forces you to learn about
branches, merging, ...  There is no workflow with dgit that doesn't
require this; when not using dgit, this isn't required.

Using dgit is a significantly higher entry barrier than not using it; I
wouldn't recommend using it for that reason.

> The "dgit" repository is also separate from the "real" repository;
> > if
> > you just use "dgit clone ${something}" you won't get the current
> > "master" branch (unless it happens to be identical to the last
> > release), or totally different if the maintainer doesn't use dgit.
> > 
> > The history is also strange if you "dgit clone" a repository where
> > the
> > maintainer used dgit in the past, but no longer does.  Now you have
> > a
> > commit tree with multiple roots which is also confusing for people.
> > 
> > All of this is very uncommon outside the dgit world.
> 
> If you are trying to get the source code, there is a choice to be
> made.
> 
> You can choose to
> 
>  (1) Get a predictable standardised tree format, which can directly be
>  built, cherry-picked, etc.,
> 
>  And, always have what is actually in the archive.
> 
>  But maybe an unhelpful history.  Especially, if the maintainer
>  does not use dgit push.
> 
>  (dgit clone)

This doesn't give you the real source according to people who state
that "source code" nowadays means the Git repository used by the
maintainer.  dgit doesn't differ from apt-get source & gbp import-dsc
there, at least I don't see one.

(This might be different for packages where the maintainer is using
dgit, didn't forget to push and didn't make any changes since the last
upload, i.e. when debcheckout would give the same result.)

>  (2) Have an unpredictable (and usually not even specified) tree
>  format, which may (and usually will) require use of
>  git-workflow-specific tools to even build it.
> 
>  Have to understand the maintainer's branch structure to try to
>  figure out which version corresponds to what is in the archive.
> 
>  Have the maintainer's history.  This is usually in a format
>  useful to the maintainer, but it is not standardised.  It may be
>  the packaging history for a whole packaging ecosystem.  It may be
>  a linear history of a gbp pq branch.  It may be a git-dpm history
>  containing multiple rebases of the same patch series.
> 
>  (debcheckout)

It has the advantage of having the history if one cares about it.  Why
would one not prefer that over the result of importing the .dsc
somewhere?

Maybe dgit is useful for some people to maintain their packages in Git;
but it's not helpful for packages where the maintainer is using it and
some complications are just not a good idea IMHO (like using a separate
Git repository with potentially different contents and not warning
users when fabricating history different from the real VCS history).

Ansgar

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Ansgar writes ("Re: Preferred git branch structure when upstream moves from 
tarballs to git"):
> On Thu, 2019-05-02 at 09:15 -0700, Russ Allbery wrote:
> > Ansgar  writes:
> > > Having to know about branches, merging, dealing with multiple remotes,
> > > ... *is* an entry barrier compared to not having to know about it.  Now
> > > you have to teach people that before you even get to how to write a
> > > build recipe.
> > 
> > I'm confused.  I'm a happy user of dgit and don't have to think about any
> > of those things as part of using dgit.  I choose to use branches, but I
> > certainly wouldn't have to, and merging, multiple remotes, and so forth
> > don't seem related to using dgit at all.
> 
> How do you update the package to a new upstream release?

The same way you would with whatever tools you are currently using for
that task.  dgit is not a git delta management tool.  For the
maintainer, dgit replaces dput, not gbp pq or git-dpm or quilt.


Or do you mean, when doing an NMU campaign ?  I wasn't aware that
Debian encouraged NMU campaigns involving new upstream versions.

If you want to do an NMU involving a new upstream version then
*somehow* you are going to have to do some patch rebasing.  There are
broadly speaking three options

 - Figure out what workflow the maintainer is using, and follow it
   (learning their tooling and their workflow if necessary).  This is
   a pain.  It's especially a pain if you're less of a Debian expert.
   Debian experts already know N different git packaging workflows.

 - Mess about with tarballs and patches.  Yuk.

 - Choose your faviour a git packaging rebase tool and convert
   whatever the maintainer published into the format needed by your
   tool.  IME this works OK and generally produces a `3.0 (quilt)'
   .dsc that no-one will object to.

 - Use git merge and convert to 1.0 or single-debian-patch.
   (No-one will thank you for this!)

Because NMUing new upstream versions is rather a minority sport, I
haven't prioritised documentation or tooling for this.  But when
necessary, I would do it with git-debrebase convert-from-dgit-view
followed by git-debrebase new-upstream.  This is IME very effective
but it's my tool and I would say that.

> The "dgit" repository is also separate from the "real" repository; if
> you just use "dgit clone ${something}" you won't get the current
> "master" branch (unless it happens to be identical to the last
> release), or totally different if the maintainer doesn't use dgit.
> 
> The history is also strange if you "dgit clone" a repository where the
> maintainer used dgit in the past, but no longer does.  Now you have a
> commit tree with multiple roots which is also confusing for people.
> 
> All of this is very uncommon outside the dgit world.

If you are trying to get the source code, there is a choice to be
made.

You can choose to

 (1) Get a predictable standardised tree format, which can directly be
 built, cherry-picked, etc.,

 And, always have what is actually in the archive.

 But maybe an unhelpful history.  Especially, if the maintainer
 does not use dgit push.

 (dgit clone)

 (2) Have an unpredictable (and usually not even specified) tree
 format, which may (and usually will) require use of
 git-workflow-specific tools to even build it.

 Have to understand the maintainer's branch structure to try to
 figure out which version corresponds to what is in the archive.

 Have the maintainer's history.  This is usually in a format
 useful to the maintainer, but it is not standardised.  It may be
 the packaging history for a whole packaging ecosystem.  It may be
 a linear history of a gbp pq branch.  It may be a git-dpm history
 containing multiple rebases of the same patch series.

 (debcheckout)

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Ben Hutchings writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git"):
> On Thu, 2019-05-02 at 11:23 +0100, Ian Jackson wrote:
> > Sorry for shouting, but, really.  It is kind of frustrating to have
> > designed and implemented and deployed a complex piece of software
> > which solves a lot of problems, and constantly hear people
> >  - proposing solutions which do not address the primary difficulties
> >  - merely lamenting that our current practices are so bad
> >  - stating that the problems are intractable and cannot be solved
> >  - saying that for this we need to agree a uniform git workflow
> 
> We still need to resolve the issues with merging that I raised in
> October.  (I say "we" because I realise you are likely to need me or
> someone else to spend time explaining and testing the specific
> scenarios that didn't work.)

That is to do with git-debrebase, not to do with dgit.

AIUI the current kernel team workflow has a bare-packaging git tree so
*if* you were to want to adopt dgit now you would want #903392 in
dgit, "want support for packaging-only maintainer views", implemented.


Packaging-only maintainer views are in a surprisingly large minority
but that minority largely consists of people who are using and keen on
quite old tools and workflows, and do not very much want to change.

>From the pov of a user/downstream who does `dgit clone': The benefit a
tarball import plus packaging git history (made by `dgit push' during
conversion from a packaging-only maintainer view), compared to a .dsc
import (tarball import, with patch queue converted to git branch, made
by `dgit clone' itself), is rather minor.

So that's why support packaging-only maintainer views is fairly low on
my todo list.


As for the linux kernel:

The proper git history is regarded by most people as the PFM.  So you
would really want #903392 fixed in a way that let dgit reuse upstream
git tags (and dgit would have to check them against your orig
tarballs).  This doesn't seem like something that many other people
would want but maybe it is ?

But all of this seems a lot of work to do for a quite suboptimal
answer when git-debrebase looks like it might be better, even though
there is still work to be done on git-debrebase.  The work on
git-debrebase will have much wider applicability.


So yes, the Debian linux kernel team have a good reason for not using
dgit yet.  Sorry if I gave a contrary impression.  There will be some
other odd cases too no doubt, but what I said was true for most normal
packages maintained using tools like native git, gbp pq, git+quilt, or
git-dpm.

Regards,
Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Russ Allbery]

Ansgar  writes:
> On Thu, 2019-05-02 at 09:15 -0700, Russ Allbery wrote:

>> I'm confused.  I'm a happy user of dgit and don't have to think about
>> any of those things as part of using dgit.  I choose to use branches,
>> but I certainly wouldn't have to, and merging, multiple remotes, and so
>> forth don't seem related to using dgit at all.

> How do you update the package to a new upstream release?

I think I understand what I was missing.  Are you using a separate
repository for only the debian directory and merging that with an unpacked
upstream tarball without using a version control system to do so?  I don't
know how that works with dgit if that's the case.

For whatever it's worth, I'm in the camp that believes the combination of
the Debian packaging and the imported upstream releases should be merged
into a single coherent Git history, because the packaging changes are
heavily dependent on the upstream changes and have an important sequencing
between upstream imports that is a key component of the total history of
the package.  Having to manually stitch together two separate histories is
exactly the kind of tedious chore that I think computers are good at and
humans are bad at, so I'd rather let Git do it.  YMMV, of course.

> The "dgit" repository is also separate from the "real" repository; if
> you just use "dgit clone ${something}" you won't get the current
> "master" branch (unless it happens to be identical to the last release),
> or totally different if the maintainer doesn't use dgit.

Yes, that's true, and somewhat inherent in the model.  I don't know that
that's avoidable without standardizing the actual Git tree used by every
maintainer for in-progress work, which is a much harder lift.  dgit stays
out of the business of investigating how the maintainer does work, at the
cost of only having visibility to the commits that have been pushed to the
archive.

-- 
Russ Allbery (r...@debian.org)   

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ansgar]

On Thu, 2019-05-02 at 09:15 -0700, Russ Allbery wrote:
> Ansgar  writes:
> 
> > Having to know about branches, merging, dealing with multiple remotes,
> > ... *is* an entry barrier compared to not having to know about it.  Now
> > you have to teach people that before you even get to how to write a
> > build recipe.
> 
> I'm confused.  I'm a happy user of dgit and don't have to think about any
> of those things as part of using dgit.  I choose to use branches, but I
> certainly wouldn't have to, and merging, multiple remotes, and so forth
> don't seem related to using dgit at all.

How do you update the package to a new upstream release?

The "dgit" repository is also separate from the "real" repository; if
you just use "dgit clone ${something}" you won't get the current
"master" branch (unless it happens to be identical to the last
release), or totally different if the maintainer doesn't use dgit.

The history is also strange if you "dgit clone" a repository where the
maintainer used dgit in the past, but no longer does.  Now you have a
commit tree with multiple roots which is also confusing for people.

All of this is very uncommon outside the dgit world.

Ansgar

Re: Preferred git branch structure when upstream moves from tarballs to git

[Russ Allbery]

Ansgar  writes:

> Having to know about branches, merging, dealing with multiple remotes,
> ... *is* an entry barrier compared to not having to know about it.  Now
> you have to teach people that before you even get to how to write a
> build recipe.

I'm confused.  I'm a happy user of dgit and don't have to think about any
of those things as part of using dgit.  I choose to use branches, but I
certainly wouldn't have to, and merging, multiple remotes, and so forth
don't seem related to using dgit at all.

Maybe you're using dgit in a way that's suboptimal for your workflow?

-- 
Russ Allbery (r...@debian.org)   

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ben Hutchings]

On Thu, 2019-05-02 at 11:23 +0100, Ian Jackson wrote:
[...]
> Sorry for shouting, but, really.  It is kind of frustrating to have
> designed and implemented and deployed a complex piece of software
> which solves a lot of problems, and constantly hear people
>  - proposing solutions which do not address the primary difficulties
>  - merely lamenting that our current practices are so bad
>  - stating that the problems are intractable and cannot be solved
>  - saying that for this we need to agree a uniform git workflow
[...]

We still need to resolve the issues with merging that I raised in
October.  (I say "we" because I realise you are likely to need me or
someone else to spend time explaining and testing the specific
scenarios that didn't work.)

Ben.

-- 
Ben Hutchings
It is easier to write an incorrect program
than to understand a correct one.




signature.asc
Description: This is a digitally signed message part

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ansgar]

On Thu, 2019-05-02 at 13:45 +0100, Ian Jackson wrote:
> Ansgar Burchardt writes ("Re: Preferred git branch structure when
> upstream moves from tarballs to git"):
> > On Tue, 2019-04-30 at 16:00 -0700, Sean Whitton wrote:
> > > As a package maintainer, if you don't keep the whole source package in
> > > git, you're giving up on a lot of the power of git.
> > 
> > I think keeping entry barriers low is more important than being able to
> > use all the power of Git.  That's sadly one of the main problems with
> > Dgit: it raises entry barriers by making packaging more complicated. 
> 
> A source-only upload with dgit is fewer commands, and more reliable,
> than doing so with sbuild/dput.

Complexity is not the number of commands to use.

Having to know about branches, merging, dealing with multiple remotes,
... *is* an entry barrier compared to not having to know about it.  Now
you have to teach people that before you even get to how to write a
build recipe.

(Also for source-only uploads you don't need sbuild at all.)

> Doing an NMU campaign with dgit is massively easier than doing so with
> .dsc-based tools.

Why should that be "massively easier" with dgit?  Without dgit you get
source, change the source package, build the source package, upload the
modified source package.  No matter what workflow/VCS/* the maintainer
uses.

Ansgar

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Ansgar Burchardt writes ("Re: Preferred git branch structure when upstream 
moves from tarballs to git"):
> On Tue, 2019-04-30 at 16:00 -0700, Sean Whitton wrote:
> > As a package maintainer, if you don't keep the whole source package in
> > git, you're giving up on a lot of the power of git.
> 
> I think keeping entry barriers low is more important than being able to
> use all the power of Git.  That's sadly one of the main problems with
> Dgit: it raises entry barriers by making packaging more complicated. 

A source-only upload with dgit is fewer commands, and more reliable,
than doing so with sbuild/dput.

Doing an NMU campaign with dgit is massively easier than doing so with
.dsc-based tools.

It is true that sometimes you can see errors from dgit about
discrepancies between your git tree and your source package, which you
do not notice if you only use gbp and dput.  What is happening there
is that you are uploading a different thing to what you have in git,
and not noticing.

We don't tell people to not use lintian because it produces error
messages, do we ?

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ansgar Burchardt]

On Tue, 2019-04-30 at 16:00 -0700, Sean Whitton wrote:
> 
> On Tue 30 Apr 2019 at 08:05AM +02, Ansgar wrote:
> 
> > As an example: to update to a new upstream release, I ideally just have
> > to drop the new upstream tarball, update d/changelog and am done.
> > Compare with [1] which is much more complicated, even ignoring the extra
> > complexity using dgit adds compared to just using git.
> > 
> >   [1] 
> > https://manpages.debian.org/stretch-backports/dgit/dgit-maint-merge.7.en.html#NEW_UPSTREAM_RELEASES
> 
> As a package maintainer, if you don't keep the whole source package in
> git, you're giving up on a lot of the power of git.

I think keeping entry barriers low is more important than being able to
use all the power of Git.  That's sadly one of the main problems with
Dgit: it raises entry barriers by making packaging more complicated. 
Packaging shouldn't be complicated: it's just a build recipe plus some
metadata.

> The most
> significant thing is that you cannot manipulate quilt patches as commits
> on a branch.  It is also much more involved to cherry pick commits from
> upstream branches, and quickly obtain diffs between Debian's version of
> the code and arbitrary other branches, to mention a few other things.

Most packages don't need that either: for me most changes are either
fairly static (no merge conflict) or are just backports of upstream
commits (in which case they can just be removed when using a new
upstream version).

It does get easier when most fixes are applied upstream instead of
staying only in Debian :-)

> I also think that you're doing a disservice to downstream users.  If
> you're trying to fix a bug in the packaged version of some software on
> your computer, you don't care about the distinction between Debian's
> packaging scripts and the upstream code.

Either the bug is in upstream code, then you just need the upstream
source (and the patch should be pushed upstream anyway).  Or it is in
the (ideally smalll) Debian-specific parts which hopefully don't need a
long history to understand.

If you have large, invasive changes from upstream, you effectively fork
the package.  Maybe one should release it as a "fork" then so non-
Debian distributions can benefit from the changes in the fork.  That is
arguable a disservice to users...

Ansgar

Re: Preferred git branch structure when upstream moves from tarballs to git

[Gard Spreemann]

My initial question resulted in a lot of useful advice and opinions, and
spurred quite an interesting discussion. Thanks to everyone who
contributed, and apologies for not contributing myself.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Philipp Kern writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git"):
> On 5/2/2019 7:11 AM, Ben Finney wrote:
> > Conversely, I think it does a disservice to downstream users to mix in
> > Debian packaging changes with upstream changes. The separation is useful
> > and much easier to maintain when the repositories are separate.
> 
> To be honest the greatest disservice is that we cannot standardize on a
> single way so that a downstream can just pull all of them and build from
> them. Instead you need humans analyzing how every single one of them works.

A downstream can get this with dgit already.  dgit provides a
downstream, or a user, with *every* package in a *standardised* git
tree format.

The shape of the history depends on what the maintainer did.  If the
maintainer used dgit, as they should, then the dgit git tree seen by
the user or downstream is the maintainer's history (possibly with some
additional git commits for format conversion).  If the last person to
upload the package just did `dput' then the downstream gets a .dsc
import.

This is a principal reason why every maintainer should be using dgit
to do their uploads.

Also, and I am going to repeat myself because this is constantly
misunderstood, for most maintainers [1],
YOU DO NOT NEED TO CHANGE YOUR GIT WORKFLOW to upload with dgit

That is, EVERYTHING TO SOLVE THIS PROBLEM [2] ALREADY EXISTS.
Including the tooling to convert the various kinds of maintainer git
branch into something standard and suitable for users and downstreams.

The problem is not to design it, the problem is to get people to use
it.

Sorry for shouting, but, really.  It is kind of frustrating to have
designed and implemented and deployed a complex piece of software
which solves a lot of problems, and constantly hear people
 - proposing solutions which do not address the primary difficulties
 - merely lamenting that our current practices are so bad
 - stating that the problems are intractable and cannot be solved
 - saying that for this we need to agree a uniform git workflow

We had the design conference in Vaumarcus in 2013.  Joey Hess and I
came up with the basic design principles on a big piece of cardboard
with a bunch of us crowded round a table.  I went and implemented it
right away and uploaded 0.1 while still at Debconf.

dgit has been useable for use by gbp pq users (and people with
equivalent patch management workflows, including people using
git+quilt) since 2.0 in October 2016.  You can upload with dgit to
buster using dgit 3.11 from stretch (although you probably prefer the
8.3 from stretch-backports).  This stuff is stable, mature, documented
software.

Ian.

[1] Support for bare-packaging git trees is a wishlist item with some
experimental patches, which I would complete if someone told me that
this was the one thing stopping them using dgit.

[2] I mean the problem of providing every Debian downstream and user
with a useable and correct and standardised git branch which can be
used for building the package, developing patches, and sharing the
results.  The problem of streamlining the Debian maintainer's upload
process to be more like "git push" remains.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Philipp Kern]

On 5/2/2019 7:11 AM, Ben Finney wrote:
> Conversely, I think it does a disservice to downstream users to mix in
> Debian packaging changes with upstream changes. The separation is useful
> and much easier to maintain when the repositories are separate.

To be honest the greatest disservice is that we cannot standardize on a
single way so that a downstream can just pull all of them and build from
them. Instead you need humans analyzing how every single one of them works.

Kind regards
Philipp Kern

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ben Finney]

Sean Whitton  writes:

> Hello,
>
> On Tue 30 Apr 2019 at 08:05AM +02, Ansgar wrote:
>
> > As an example: to update to a new upstream release, I ideally just
> > have to drop the new upstream tarball, update d/changelog and am
> > done.
>
> As a package maintainer, if you don't keep the whole source package in
> git, you're giving up on a lot of the power of git.

I can't speak for Ansgar, but “you don't keep the whole source package
in Git” is not implied by keeping Debian packaging separate. It's not
accurate to the workflow, at least as I described (I don't know Ansgar's
case, but nothing he described implies that either).

Rather, I keep the Debian packaging source separate from the upstream
source. That doesn't preclude Git access to the upstream source, and I
frequently use a Git repository cloned from upstream for that.

So, the Debian-packaging-in-a-separate-repository does not give up any
of the power of Git.

> The most significant thing is that you cannot manipulate quilt patches
> as commits on a branch. It is also much more involved to cherry pick
> commits from upstream branches, and quickly obtain diffs between
> Debian's version of the code and arbitrary other branches, to mention
> a few other things.

The full power of Git is available when I manipulate upstream source to
refresh my Debian patches. Indeed, it's even neater to refresh those
patches by going straight from the only-upstream-source repository.

> I also think that you're doing a disservice to downstream users. If
> you're trying to fix a bug in the packaged version of some software on
> your computer, you don't care about the distinction between Debian's
> packaging scripts and the upstream code. It's all going to be turned
> into a .deb once you've fixed your problem. You want the history of
> the whole thing. Thus, a git history that contains both the upstream
> git history and the Debian maintainer's changes to the packaging
> scripts is going to be very useful. A git history of only the Debian
> packaging scripts is much less useful.

Conversely, I think it does a disservice to downstream users to mix in
Debian packaging changes with upstream changes. The separation is useful
and much easier to maintain when the repositories are separate.

-- 
 \ “Time is the great legalizer, even in the field of morals.” |
  `\ —Henry L. Mencken |
_o__)  |
Ben Finney

Re: Preferred git branch structure when upstream moves from tarballs to git

[Sean Whitton]

Hello,

On Wed 01 May 2019 at 09:53PM +0200, Adam Borowski wrote:

> On Wed, May 01, 2019 at 08:14:24AM +0200, Thomas Goirand wrote:
>> It's basically useless, since upstream repository is just one command
>> away, with the upstream URL documented in debian/rules.
>
> This assumes the upstream URL lives forever.  But what if the upstream
> repo's hoster (let's call it "alioth") goes offline?  It's not an unheard
> of thing: Gitorious, Google Code, etc...
>
> To me, having the pristine code in a preferred form for modification in the
> same place as your packaged version follows the spirit of GPL from 3 decades
> ago, or the design of .dsc of 2⅔ decades ago that calls for shipping the
> pristine upstream tarball and Debian patches separately.  And today, I
> really wouldn't call a flat tarball the preferred form for modification.

Perhaps I'm missing something, but zigo said he was pushing upstream's
tags to his repo.  Doesn't that cover this?

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Preferred git branch structure when upstream moves from tarballs to git

[Adam Borowski]

On Wed, May 01, 2019 at 08:14:24AM +0200, Thomas Goirand wrote:
> It's basically useless, since upstream repository is just one command
> away, with the upstream URL documented in debian/rules.

This assumes the upstream URL lives forever.  But what if the upstream
repo's hoster (let's call it "alioth") goes offline?  It's not an unheard
of thing: Gitorious, Google Code, etc...

To me, having the pristine code in a preferred form for modification in the
same place as your packaged version follows the spirit of GPL from 3 decades
ago, or the design of .dsc of 2⅔ decades ago that calls for shipping the
pristine upstream tarball and Debian patches separately.  And today, I
really wouldn't call a flat tarball the preferred form for modification.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁
⢿⡄⠘⠷⠚⠋⠀ Have you accepted Khorne as your lord and saviour?
⠈⠳⣄

Re: Preferred git branch structure when upstream moves from tarballs to git

[Gabriel F. T. Gomes]

Oh, only now I noticed that this has become a long thread.  Sorry for
the out-of-context reply. :)

On 01 May 2019, Gabriel F. T. Gomes wrote:

>On 29 Apr 2019, Gard Spreemann wrote:
>
>>For one of my packages, I maintain two public git branches: one is
>>upstream/latest, where I've been importing upstream's released tarballs,
>>and the other is debian/sid that contains the packaging.
>>
>>Recently, upstream has finally started using git. What is the
>>recommended way for me to maintain a sane branch structure for the
>>packaging repository while starting to use upstream's git master as the
>>upstream branch to follow?  
>
>I don't know about recommended, but even though the projects I maintain
>have git repositories themselves, I only sync their released tarballs.
>For that, I use git-buildpackage and uscan, more specifically, gbp
>import-orig --uscan [1], which automatically creates a branch structure
>similar to yours.
>
>As far as I know, you are not required to use git-buildpackage, nor to
>sync only from released tarballs, but the link below has some guidance
>on how to sync from upstream repositories, so I hope that helps.
>
>[1] https://wiki.debian.org/PackagingWithGit#Importing_upstream_as_tarballs-1
>
>>(My first thought is to track upstream's master as upstream/latest-git
>>or something, and start merging from that into debian/sid, but I don't
>>know if there's a better way.)  
>
>The link above describes a very similar approach.
>

Re: Preferred git branch structure when upstream moves from tarballs to git

[Gabriel F. T. Gomes]

On 29 Apr 2019, Gard Spreemann wrote:

>For one of my packages, I maintain two public git branches: one is
>upstream/latest, where I've been importing upstream's released tarballs,
>and the other is debian/sid that contains the packaging.
>
>Recently, upstream has finally started using git. What is the
>recommended way for me to maintain a sane branch structure for the
>packaging repository while starting to use upstream's git master as the
>upstream branch to follow?

I don't know about recommended, but even though the projects I maintain
have git repositories themselves, I only sync their released tarballs.
For that, I use git-buildpackage and uscan, more specifically, gbp
import-orig --uscan [1], which automatically creates a branch structure
similar to yours.

As far as I know, you are not required to use git-buildpackage, nor to
sync only from released tarballs, but the link below has some guidance
on how to sync from upstream repositories, so I hope that helps.

[1] https://wiki.debian.org/PackagingWithGit#Importing_upstream_as_tarballs-1

>(My first thought is to track upstream's master as upstream/latest-git
>or something, and start merging from that into debian/sid, but I don't
>know if there's a better way.)

The link above describes a very similar approach.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Thomas Goirand]

On 4/30/19 9:55 AM, Adam Borowski wrote:
> On Tue, Apr 30, 2019 at 09:04:09AM +0200, Thomas Goirand wrote:
>> What I found the most easy way is to just use whatever branching names
>> upstream is using, and *never* store them in Salsa. If you need upstream
>> repository, you can simply run a ./debian/rules fetch-upstream-remote,
>> and most of the time, the only thing you need, is merging a tag. What I
>> push to Salsa are upstream tags only, which is enough.
> 
> Why?  I strongly disagree: having the upstream tree in the same repo
> as packaging is awesome. It lets you mix patches both ways with no effort.
> Having the upstream tree at home but not pushing it deprives contributors
> who are not you of this comfort.

It's basically useless, since upstream repository is just one command
away, with the upstream URL documented in debian/rules.

Cheers,

Thomas Goirand (zigo)

Re: Preferred git branch structure when upstream moves from tarballs to git

[Sean Whitton]

Hello,

On Tue 30 Apr 2019 at 08:05AM +02, Ansgar wrote:

> As an example: to update to a new upstream release, I ideally just have
> to drop the new upstream tarball, update d/changelog and am done.
> Compare with [1] which is much more complicated, even ignoring the extra
> complexity using dgit adds compared to just using git.
>
>   [1] 
> https://manpages.debian.org/stretch-backports/dgit/dgit-maint-merge.7.en.html#NEW_UPSTREAM_RELEASES

As a package maintainer, if you don't keep the whole source package in
git, you're giving up on a lot of the power of git.  The most
significant thing is that you cannot manipulate quilt patches as commits
on a branch.  It is also much more involved to cherry pick commits from
upstream branches, and quickly obtain diffs between Debian's version of
the code and arbitrary other branches, to mention a few other things.

I also think that you're doing a disservice to downstream users.  If
you're trying to fix a bug in the packaged version of some software on
your computer, you don't care about the distinction between Debian's
packaging scripts and the upstream code.  It's all going to be turned
into a .deb once you've fixed your problem.  You want the history of the
whole thing.  Thus, a git history that contains both the upstream git
history and the Debian maintainer's changes to the packaging scripts is
going to be very useful.  A git history of only the Debian packaging
scripts is much less useful.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Campbell]

On Tue, 2019-04-30 at 12:05 +0100, Ian Jackson wrote:
> Sean Whitton writes ("Re: Preferred git branch structure when upstream moves 
> from tarballs to git"):
> > On Mon 29 Apr 2019 at 02:12PM +01, Ian Campbell wrote:
> > > 
> > > I think (but don't know) that the answer is that if your queue drops to
> > > zero you have effectively flipped back to the dgit-maint-merge world.
> > > If you find you want to go the other way then I'm less clear (except
> > > perhaps that is `git debrebase convert-from-dgit-view` and then commit
> > > to non-debian/ as usual?).
> 
> When your queue is empty, dgit-maint-merge and dgit-maint-gbp are
> equivalent and dgit-maint-debrebase is "very close".
> 
> When you go from git-merge to separated-patches, you have to create
> the additional information: the split of the delta into separate
> patches with commit messages.  However, I guess as the user you have
> already figured that out and what you want is to know that you can
> turn git-merge into
>   separated-patches but currently with one patch with
> an autogenerated commit message
> and then you would use normal git-fu to rework that into the desired
> pretty patch series.

Ack, yes.

> > Do you think it would be helpful to add sections to both the -merge and
> > -debrebase manpages saying this stuff?
> 
> I think we should have a separate manpage.  This kind of conversion
> stuff is (hopefully) used rarely.

Assuming some x-refs from the relevant places that seems perfectly
sensible too me.

Ian.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Sean Whitton writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git"):
> On Mon 29 Apr 2019 at 02:12PM +01, Ian Campbell wrote:
> > Are there any docs/advice on switching back and forth between these (or
> > at least switching between dgit-maint-merge and one of the patch
> > queueish ones)?
...
> > I'm not thinking anyone would flip regularly or anything, but just
> > knowing that flipping between them in something approaching a sane way
> > was possible at all would give some confidence to make a choice in the
> > here and now without getting too paralysed by the need to make a
> > decision.

That makes a lot of sense.

> > I think (but don't know) that the answer is that if your queue drops to
> > zero you have effectively flipped back to the dgit-maint-merge world.
> > If you find you want to go the other way then I'm less clear (except
> > perhaps that is `git debrebase convert-from-dgit-view` and then commit
> > to non-debian/ as usual?).

When your queue is empty, dgit-maint-merge and dgit-maint-gbp are
equivalent and dgit-maint-debrebase is "very close".

When you go from git-merge to separated-patches, you have to create
the additional information: the split of the delta into separate
patches with commit messages.  However, I guess as the user you have
already figured that out and what you want is to know that you can
turn git-merge into
  separated-patches but currently with one patch with
an autogenerated commit message
and then you would use normal git-fu to rework that into the desired
pretty patch series.

> Do you think it would be helpful to add sections to both the -merge and
> -debrebase manpages saying this stuff?

I think we should have a separate manpage.  This kind of conversion
stuff is (hopefully) used rarely.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Campbell]

On Mon, 2019-04-29 at 13:50 -0700, Sean Whitton wrote:
> 
> dgit-maint-merge -> dgit-maint-debrebase would indeed be the
> convert-from-dgit-view subcommand.  If I were dealing with this, for
> simplicity, I would start from the point of just having made an upload.
> That way you know your HEAD is a valid dgit view.
> 
> dgit-maint-debrebase -> dgit-maint-merge would be simply to stop using
> the `git debrebase` tool.
> 
> dgit-maint-debrebase -> dgit-maint-merge -> dgit-maint-debrebase might
> get tricky, because `git debrebase` might get confused by your git
> history.  I think, though, that any problems would count as bugs in the
> convert-from-dgit-view subcommand.

Excellent, thanks for confirming/explaining.

> Switching between dgit-maint-gbp and either -merge or -debrebase is
> tricky because you are going from patches-unapplied to patches-applied.
> What I think you would want to do is use convert-from-gbp, and then if
> you wanted -merge, just don't invoke `git debrebase` anymore.  But I am
> not sure.

Yes, I can see how that case is more complex. I think for me at least
that would mean I would simply prefer debrebase anyway so being able to
cycle via gbp wouldn't worry me too much.

> Do you think it would be helpful to add sections to both the -merge and
> -debrebase manpages saying this stuff?

I do, yes, it would knock a potential mental rathole (at least for
me...) out of the picture.

Ian.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Adam Borowski]

On Tue, Apr 30, 2019 at 09:04:09AM +0200, Thomas Goirand wrote:
> What I found the most easy way is to just use whatever branching names
> upstream is using, and *never* store them in Salsa. If you need upstream
> repository, you can simply run a ./debian/rules fetch-upstream-remote,
> and most of the time, the only thing you need, is merging a tag. What I
> push to Salsa are upstream tags only, which is enough.

Why?  I strongly disagree: having the upstream tree in the same repo
as packaging is awesome.  It lets you mix patches both ways with no effort.
Having the upstream tree at home but not pushing it deprives contributors
who are not you of this comfort.

As all the data is already in the repo, having it available as a branch
and tags doesn't cost any non-metadata space.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Did ya know that typing "test -j8" instead of "ctest -j8"
⢿⡄⠘⠷⠚⠋⠀ will make your testsuite pass much faster, and fix bugs?
⠈⠳⣄

Re: Preferred git branch structure when upstream moves from tarballs to git

[Thomas Goirand]

On 4/29/19 11:18 AM, Gard Spreemann wrote:
> Hi,
> 
> For one of my packages, I maintain two public git branches: one is
> upstream/latest, where I've been importing upstream's released tarballs,
> and the other is debian/sid that contains the packaging.
> 
> Recently, upstream has finally started using git. What is the
> recommended way for me to maintain a sane branch structure for the
> packaging repository while starting to use upstream's git master as the
> upstream branch to follow?
> 
> (My first thought is to track upstream's master as upstream/latest-git
> or something, and start merging from that into debian/sid, but I don't
> know if there's a better way.)
> 
> 
>  Best,
>  Gard

What I found the most easy way is to just use whatever branching names
upstream is using, and *never* store them in Salsa. If you need upstream
repository, you can simply run a ./debian/rules fetch-upstream-remote,
and most of the time, the only thing you need, is merging a tag. What I
push to Salsa are upstream tags only, which is enough.

Cheers,

Thomas Goirand (zigo)

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ansgar]

Russell Stuart writes:
> On Tue, 2019-04-30 at 09:25 +0800, Paul Wise wrote:
>> I like this option because it still works well if we ever decide to
>> fix a fundamental flaw in the Debian source package layout.
>
> I suspect whether that's a fundamentally is a matter or personal taste.
>  On this point my taste aligns with yours.
>
> I've used both rpm source format and the Debian one, and IMO the rpm
> one is mostly better.  The primary reason is the one you've mention
> here: they maintain the separation between the source, rpm spec, and
> build areas's far more cleanly than Debian does.  This makes some
> common flaws one often flaws in Debian packages just disappear: like
> cleaning up the source directory after a build.

I also agree with this.  It is also not only RPM, but for example also
Gentoo, Arch and others which contain only the distro-specific parts in
their repositories.

This also makes things easier as developers do not have to know about
branch management, merging, rebasing, ... to start with.  Many people
using Git don't know how to do this.

As an example: to update to a new upstream release, I ideally just have
to drop the new upstream tarball, update d/changelog and am done.
Compare with [1] which is much more complicated, even ignoring the extra
complexity using dgit adds compared to just using git.

Ansgar

  [1] 
https://manpages.debian.org/stretch-backports/dgit/dgit-maint-merge.7.en.html#NEW_UPSTREAM_RELEASES

Re: Preferred git branch structure when upstream moves from tarballs to git

[Russell Stuart]

On Tue, 2019-04-30 at 09:25 +0800, Paul Wise wrote:
> I like this option because it still works well if we ever decide to
> fix a fundamental flaw in the Debian source package layout.

I suspect whether that's a fundamentally is a matter or personal taste.
 On this point my taste aligns with yours.

I've used both rpm source format and the Debian one, and IMO the rpm
one is mostly better.  The primary reason is the one you've mention
here: they maintain the separation between the source, rpm spec, and
build areas's far more cleanly than Debian does.  This makes some
common flaws one often flaws in Debian packages just disappear: like
cleaning up the source directory after a build.

Where the rpm format goes wrong is it then beaks that separation in the
actually .srpm format by putting the upstream source in and rpm bits in
one file, which is of course what Debian gets right.  Sigh.

On the positive side, rpm and deb seem to be gradually converging in a
sort of co-evolution.



signature.asc
Description: This is a digitally signed message part

Re: Preferred git branch structure when upstream moves from tarballs to git

[Paul Wise]

On Tue, Apr 30, 2019 at 9:03 AM Ben Finney wrote:

> My opinionated recommendation: Track the Debian packaging in a separate
> repository (which contains only the ‘debian/’ directory tree). When it's
> time to build the Debian source package for testing, export the upstream
> source to a temporary build directory and export the Debian packaging
> onto that. Build the Debian source package from the result.

I like this option because it still works well if we ever decide to
fix a fundamental flaw in the Debian source package layout. The
directory hierarchy we use is inverted from one that would be logical
based on the relationship between the components of the Debian source
package. The Debian packaging (especially debian/rules) wraps and
controls the interaction of the Debian tools with the upstream source
but the debian/ directory is located inside the upstream source
instead of the upstream source being a subdirectory of the Debian
packaging.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ben Finney]

Gard Spreemann  writes:

> Recently, upstream has finally started using git. What is the
> recommended way for me to maintain a sane branch structure for the
> packaging repository while starting to use upstream's git master as
> the upstream branch to follow?

My opinionated recommendation: Track the Debian packaging in a separate
repository (which contains only the ‘debian/’ directory tree). When it's
time to build the Debian source package for testing, export the upstream
source to a temporary build directory and export the Debian packaging
onto that. Build the Debian source package from the result.

(Of course, Git-BuildPackage supports this workflow, with the ‘--export’
and related options.)

Insulation from the kind of changes in upstream publishing that you
describe, is a significant advantage of this Debian packaging workflow.

-- 
 \ “My girlfriend has a queen sized bed; I have a court jester |
  `\   sized bed. It's red and green and has bells on it, and the ends |
_o__) curl up.” —Steven Wright |
Ben Finney

Re: Preferred git branch structure when upstream moves from tarballs to git

[Sean Whitton]

Hello,

On Mon 29 Apr 2019 at 02:12PM +01, Ian Campbell wrote:

> Are there any docs/advice on switching back and forth between these (or
> at least switching between dgit-maint-merge and one of the patch
> queueish ones)?
>
> When trying to chose I seem to find myself thinking along the lines
> "merge would be fine right now, but whatif..." or "I have a big queue
> now which isn't going anyway but I really hope it will go away
> (eventually)".
>
> I'm not thinking anyone would flip regularly or anything, but just
> knowing that flipping between them in something approaching a sane way
> was possible at all would give some confidence to make a choice in the
> here and now without getting too paralysed by the need to make a
> decision.
>
> I think (but don't know) that the answer is that if your queue drops to
> zero you have effectively flipped back to the dgit-maint-merge world.
> If you find you want to go the other way then I'm less clear (except
> perhaps that is `git debrebase convert-from-dgit-view` and then commit
> to non-debian/ as usual?).

dgit-maint-merge -> dgit-maint-debrebase would indeed be the
convert-from-dgit-view subcommand.  If I were dealing with this, for
simplicity, I would start from the point of just having made an upload.
That way you know your HEAD is a valid dgit view.

dgit-maint-debrebase -> dgit-maint-merge would be simply to stop using
the `git debrebase` tool.

dgit-maint-debrebase -> dgit-maint-merge -> dgit-maint-debrebase might
get tricky, because `git debrebase` might get confused by your git
history.  I think, though, that any problems would count as bugs in the
convert-from-dgit-view subcommand.

Switching between dgit-maint-gbp and either -merge or -debrebase is
tricky because you are going from patches-unapplied to patches-applied.
What I think you would want to do is use convert-from-gbp, and then if
you wanted -merge, just don't invoke `git debrebase` anymore.  But I am
not sure.

Do you think it would be helpful to add sections to both the -merge and
-debrebase manpages saying this stuff?

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Preferred git branch structure when upstream moves from tarballs to git

[Sean Whitton]

Hello,

On Mon 29 Apr 2019 at 11:18AM +02, Gard Spreemann wrote:

> For one of my packages, I maintain two public git branches: one is
> upstream/latest, where I've been importing upstream's released tarballs,
> and the other is debian/sid that contains the packaging.
>
> Recently, upstream has finally started using git. What is the
> recommended way for me to maintain a sane branch structure for the
> packaging repository while starting to use upstream's git master as the
> upstream branch to follow?

An simple approach is to use 'master' for packaging and *not* maintain
upstream branches in your repository.  Instead, you fetch upstream's
release tags and merge those.

Quoting from dgit-maint-debrebase(7), though the point is more widely
applicable than that workflow:

   The idea here is that from Debian's point of view, upstream
   releases are immutable points in history.  We want to make sure
   that we are basing our Debian package on a properly identified
   upstream version, rather than some arbitrary commit on some
   branch.  Tags are more useful for this.

   Upstream's branches remain available as the git remote tracking
   branches for your upstream remote, e.g. remotes/upstream/master.

The reason for not *additionally* maintaining upstream branches is
simply that then you don't have to keep track of whether they are
up-to-date.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Preferred git branch structure when upstream moves from tarballs to git

[Russ Allbery]

Gard Spreemann  writes:

> For one of my packages, I maintain two public git branches: one is
> upstream/latest, where I've been importing upstream's released tarballs,
> and the other is debian/sid that contains the packaging.

> Recently, upstream has finally started using git. What is the
> recommended way for me to maintain a sane branch structure for the
> packaging repository while starting to use upstream's git master as the
> upstream branch to follow?

A lot of people have told you that you have a lot of options, which is of
course true.  But in case you're looking for an opinionated answer rather
than a range of options: in cases like this, I track the upstream master
branch in my repository as master, just as if I had a regular clone of
upstream, and use debian/master (or debian/sid if you want) for the
packaging.

I experimented with a few other approaches, and this one seemed to cause
the least amount of pain.  It means I'm not renaming the upstream branches
when I pull them into my repository (which is possible to do in Git but
tedious and irritating if you get the .git/config runes incorrect or some
tool doesn't pay attention and merges the wrong branch).

-- 
Russ Allbery (r...@debian.org)   

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Campbell]

On Mon, 2019-04-29 at 13:25 +0100, Ian Jackson wrote:
> Look at the intro to each of these and see which one you think might
> be best:
>  https://manpages.debian.org/stretch-backports/dgit/dgit-maint-merge.7.en.html
>  
> https://manpages.debian.org/stretch-backports/dgit/dgit-maint-debrebase.7.en.html
>  https://manpages.debian.org/stretch-backports/dgit/dgit-maint-gbp.7.en.html
>  
> https://manpages.debian.org/stretch-backports/dgit/dgit-maint-native.7.en.html

Are there any docs/advice on switching back and forth between these (or
at least switching between dgit-maint-merge and one of the patch
queueish ones)?

When trying to chose I seem to find myself thinking along the lines
"merge would be fine right now, but whatif..." or "I have a big queue
now which isn't going anyway but I really hope it will go away
(eventually)".

I'm not thinking anyone would flip regularly or anything, but just
knowing that flipping between them in something approaching a sane way
was possible at all would give some confidence to make a choice in the
here and now without getting too paralysed by the need to make a
decision.

I think (but don't know) that the answer is that if your queue drops to
zero you have effectively flipped back to the dgit-maint-merge world.
If you find you want to go the other way then I'm less clear (except
perhaps that is `git debrebase convert-from-dgit-view` and then commit
to non-debian/ as usual?).

Ian.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Gard Spreemann writes ("Preferred git branch structure when upstream moves from 
tarballs to git"):
> Recently, upstream has finally started using git. What is the
> recommended way for me to maintain a sane branch structure for the
> packaging repository while starting to use upstream's git master as the
> upstream branch to follow?

The dgit workflow manpages present a smorgasbord of possibilities to
choose from.  (You can use much of this without dgit, but of course
everyone should be using dgit.)

Look at the intro to each of these and see which one you think might
be best:
 https://manpages.debian.org/stretch-backports/dgit/dgit-maint-merge.7.en.html
 
https://manpages.debian.org/stretch-backports/dgit/dgit-maint-debrebase.7.en.html
 https://manpages.debian.org/stretch-backports/dgit/dgit-maint-gbp.7.en.html
 https://manpages.debian.org/stretch-backports/dgit/dgit-maint-native.7.en.html

Adam mentioned gbp-pq, which is discussed in dgit-maint-gbp(7), and
using plain `git merge', which is discussed in dgit-maint-merge(7).

The dgit-maint-merge(7) manpage intro is particularly good at helping
select a workflow.

dgit-maint-debrebase uses my new git-debrebase tool which you may find
easier than gbp pq.

Ian.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Adam Borowski]

On Mon, Apr 29, 2019 at 11:18:48AM +0200, Gard Spreemann wrote:
> For one of my packages, I maintain two public git branches: one is
> upstream/latest, where I've been importing upstream's released tarballs,
> and the other is debian/sid that contains the packaging.
> 
> Recently, upstream has finally started using git. What is the
> recommended way for me to maintain a sane branch structure for the
> packaging repository while starting to use upstream's git master as the
> upstream branch to follow?
> 
> (My first thought is to track upstream's master as upstream/latest-git
> or something, and start merging from that into debian/sid, but I don't
> know if there's a better way.)

Naming doesn't really matter -- automated tools know only about the
packaging branch, and that's specified in the Vcs-Git field.

So it's mostly about workflow.  Here the opinions differ greatly, and it's a
fine area for flamewars.  There are those who swear by gbp, while for me
that's a monstrosity -- my personal preference is raw git, where updating to
a new upstream is "git merge v3.14.15", with all git goodness like
cherry-pick or bisect working unmolested.  But workflow choices are many.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Did ya know that typing "test -j8" instead of "ctest -j8"
⢿⡄⠘⠷⠚⠋⠀ will make your testsuite pass much faster, and fix bugs?
⠈⠳⣄

Re: Preferred git branch structure when upstream moves from tarballs to git

[Andrey Rahmatullin]

On Mon, Apr 29, 2019 at 11:18:48AM +0200, Gard Spreemann wrote:
> For one of my packages, I maintain two public git branches: one is
> upstream/latest, where I've been importing upstream's released tarballs,
> and the other is debian/sid that contains the packaging.
> 
> Recently, upstream has finally started using git. What is the
> recommended way for me to maintain a sane branch structure for the
> packaging repository while starting to use upstream's git master as the
> upstream branch to follow?
> 
> (My first thought is to track upstream's master as upstream/latest-git
> or something, and start merging from that into debian/sid, but I don't
> know if there's a better way.)
It's hard to answer not knowing your workflows, for example how are you
using those branches and how do you create the orig tarball.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Preferred git branch structure when upstream moves from tarballs to git

[Andrey Rahmatullin]

On Mon, Apr 29, 2019 at 11:40:39AM +0200, Adam Borowski wrote:
> that's a monstrosity -- my personal preference is raw git, where updating to
> a new upstream is "git merge v3.14.15"
gbp allows this too, of course.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Preferred git branch structure when upstream moves from tarballs to git

[Gard Spreemann]

Hi,

For one of my packages, I maintain two public git branches: one is
upstream/latest, where I've been importing upstream's released tarballs,
and the other is debian/sid that contains the packaging.

Recently, upstream has finally started using git. What is the
recommended way for me to maintain a sane branch structure for the
packaging repository while starting to use upstream's git master as the
upstream branch to follow?

(My first thought is to track upstream's master as upstream/latest-git
or something, and start merging from that into debian/sid, but I don't
know if there's a better way.)


 Best,
 Gard