Re: Summary: Git Packaging: Native source formats

[Guillem Jover]

On Tue, 2019-11-05 at 02:34:13 +0100, Guillem Jover wrote:
> I'm planning to add warnings for 1.0 formats when using a version that
> does not match the type of tarball found. Ideally 1.0 formats would
> error out in the same way 3.0 do, but this is impractical, and I guess
> we'll just have to wait it out until 1.0 formats die off. :(

Actually, this involves a very tiny set of packages in the archive:

,--- grep-deb-sources ---
#!/bin/sh
Sources=$(apt-get indextargets --format '$(FILENAME)' 'Identifier: Sources')
/usr/lib/apt/apt-helper cat-file $Sources | grep-dctrl "$@"
`---

- Native 1.0 source + non-native version = 1
  $ grep-deb-sources -sPackage -FFormat '1.0' -a \
  -FVersion '-' -a --not -FFiles '.diff.' | wc -l
- Non-native 1.0 source + native version = 36
  $ grep-deb-sources -sPackage -FFormat '1.0' -a \
  -FVersion '-' -a --not -FFiles '.diff.' | wc -l

Which matches exactly the lintian tag:

  

So this has now an even more fringe usage than I recalled, most of which
I'm pretty confident are due to packaging errors, and where I'd say less
than around 10 might be actually done on purpose.

Thanks,
Guillem

Re: Summary: Git Packaging: Native source formats

[Guillem Jover]

(Even though it was not clear in the initial mail, I'm assuming
 this is about using native format 1.0 with non-native versions.)

On Fri, 2019-08-30 at 12:57:56 -0400, Sam Hartman wrote:
> My take away from this discussion though is that using a native package
> format even when there are upstream sources is an appropriate tool for
> maintainers to use sometimes.

I have to strongly disagree with this. We have 1.0 source packages
that currently do this, some out of error (due to the flakiness of
the format), some due to maintainer choice. Like last time this was
discussed here, I think both usages are wrong.

> I think that is a significant change over our feelings years ago.

My feeling is that repeating all this stuff over and over again is
exhausting, more so when it comes to the git workflow/packaging stuff.

> Back in the day I think there was approach a project consensus that
> using native format packages with  upstreams that were native to Debian
> was basically always wrong.

It's still very much wrong. Of course if you are upstream and are
releasing native source packages, that's obviously fine. A Debian
maintainer could then either just upload them as-is, or turn them
into non-native source packages for upload.

> My reading of this discussion is that it's a tool that maintainers
> should have and it's sometimes reasonable to use that tool.  People have
> done a great job of bringing up athings to think about for maintainers
> considering using that tool.  I've certainly found the discussion
> educational.  It's changed my thinking about when I would choose to use
> native format packages in the future.

It might be a way out maintainers feel is convenient, it does not
mean it's right. Due to at least:

  - This is an aberration of the defined meaning of the terms, and
what they imply. So makes communicating things harder and
confusing.
  - It breaks assumptions and consistency between version, source
format and source content, and the tools dealing with these.
  - It implies releasing tarballs as if we were upstream with
additional content that upstream has never provided (which is
very different to stripping out content and then marking this
in the version).
  - It makes using Debian source tarballs by third-parties unreliable.
  - It makes downloading just the packaging bits more costly.
  - It makes making sure what truly got modified more costly, because
you cannot know easily whether a patch got applied in the actual
tarball.

I've never seen an actual good reason for this practice, except for
the “doing it right is a bit more work”, TBH.

If the maintainer wants to generate a tarball out of git, they can do
that right now with «git archive» and using a version based on a the
upstream version + date + commit-id, then using that as the orig.tar.
It has the same space inefficiency drawback, but at least it is correct.


I'm planning to add warnings for 1.0 formats when using a version that
does not match the type of tarball found. Ideally 1.0 formats would
error out in the same way 3.0 do, but this is impractical, and I guess
we'll just have to wait it out until 1.0 formats die off. :(

Thanks,
Guillem

Re: Git Packaging: Native source formats

[Ansgar]

Sam Hartman writes:
> * One is that you're not using upstream tarballs.  If upstream has
>   tarballs they produce, we're not using them.  I guess we may end up
>   having that part of the conversation now rather than later.
>
>   It's clear that we value integrity of upstream source.  That is we
>   want to make it easy for people to start from some upstream source
>   that is trusted because upstream has produced it and audit just our
>   changes.
>   One way to do this is with an upstream tarball and a diff (or set of
>   diffs or a debian directory).

There are a few other projects consuming upstream tarballs from Debian's
archive.  I've seen source-based distributions (portage, pkgsrc) using
tarballs from there.  It would be friendly to not break this for them if
we can avoid doing so without too much cost.

Upstream tarballs are probably also the easiest way for upstream to
provide a signed version of their software which we have tried to
encourage (for example by including such signatures in Debian's
archive).

Ansgar

Re: Git Packaging: Native source formats

[Marco d'Itri]

On Aug 30, Scott Kitterman  wrote:

> It's not particularly rare for me to poke through other distros package 
> patches when I'm trying to figure out how to solve a problem.  I suspect I'm 
I would even say that maintainers who do not periodically review what 
other distributions do with their own similar packages are negligent, 
especially if we are talking about mature software whose upstream 
maintainer is not active anymore.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: Git Packaging: Native source formats

[Ian Jackson]

Russ Allbery writes ("Re: Git Packaging: Native source formats"):
> [context: git-debrebase, git-dpm]
>
> However, while analyzing a rebased branch isn't as hard for other
> people as a branch with a complex merge history, it does mean that
> upstream has to find a way to extract patches [...]
>
> [context: git-debcherry] 
> This lets you generate the patches for people on demand, but they aren't
> just sitting out there for anyone to look at whenever they want.

Yes.

> Anyway, this is probably only applicable to a minority of upstreams and
> packages, and more upstreams these days would just like all patches as
> PRs.

One way to look at all this is that fishing the patches out of
Debian's (or whatever other downstream's) processes into a
standardised form suitable for an upstream branch is work.

Nowadays most upstreams expect us to do that work.  (The same way we
expect our downstreams to file bugs, not expect us to guddle around in
their systems looking for stuff to take.)

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Git Packaging: Native source formats

[Ian Jackson]

gregor herrmann writes ("Re: Git Packaging: Native source formats"):
> On Thu, 29 Aug 2019 22:03:18 -0400, Theodore Y. Ts'o wrote:
> > The problem I have is that "dgit gbp" doesn't extract the upstream
> > .asc.  
> 
> This sounds like #872864 in git-buildpackage.

Oh, interesting.  In that case Ted might find that using `dgit
push-source', `dgit sbuild' or `dgit pbuilder' work where `dgit
gbp-build' doesn't.  That probably involves manually running
pristine-tar(1) (maybe origtargz would do it the right way?)

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Git Packaging: Native source formats

[James McCoy]

On Fri, Aug 30, 2019 at 09:05:45AM -0700, Russ Allbery wrote:
> Ian Jackson  writes:
> > This is also not that hard, in simple cases.  There is a tool
> > git-debcherry which can do it automatically.  I haven't used it but AIUI
> > if your Debian delta queue has few commits, and doesn't have commits
> > which involve merge conflicts with upstream merges (basically, if each
> > change is carried Debian only for a short time), it will always produce
> > the nice output you would hope for.
> 
> This lets you generate the patches for people on demand, but they aren't
> just sitting out there for anyone to look at whenever they want.
> 
> I suppose it could be provided as an automated service that publishes the
> results, but that would be a piece of infrastructure someone has to run.

Since git-debcherry is used to export the patches when creating the
source package, such a service already exists -- https://sources.debian.org/

Now, git-debcherry doesn't always make the most human consumable patch
series.  For example, the neovim 0.3.4-3[0] upload in which I
cherry-picked a large number of patches for a security fix.  Rather than
having distinct patches for most of those commits, they were mostly
munged into a single "debcherry fixup" patch.

[0]: https://sources.debian.org/src/neovim/0.3.4-3/debian/patches/

Cheers,
-- 
James
GPG Key: 4096R/91BF BF4D 6956 BD5D F7B7  2D23 DFE6 91AE 331B A3DB

Re: Git Packaging: Native source formats

[gregor herrmann]

On Thu, 29 Aug 2019 22:03:18 -0400, Theodore Y. Ts'o wrote:

> The problem I have is that "dgit gbp" doesn't extract the upstream
> .asc.  

This sounds like #872864 in git-buildpackage.


Cheers,
gregor

-- 
 .''`.  https://info.comodo.priv.at -- Debian Developer https://www.debian.org
 : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D  85FA BB3A 6801 8649 AA06
 `. `'  Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
   `-   NP: Rebekka Bakken & Wolfgang Muthspiel: Angela


signature.asc
Description: Digital Signature

Re: Git Packaging: Native source formats

[Vincent Cunningham]

On Fri, 2019-08-30 at 12:20 -0400, Scott Kitterman wrote:
> On Friday, August 30, 2019 12:05:45 PM EDT Russ Allbery wrote: 
> > This lets you generate the patches for people on demand, but they aren't
> > just sitting out there for anyone to look at whenever they want.
> > 
> > I suppose it could be provided as an automated service that publishes the
> > results, but that would be a piece of infrastructure someone has to run.
> > 
> > Anyway, this is probably only applicable to a minority of upstreams and
> > packages, and more upstreams these days would just like all patches as
> > PRs.
> 
> It's not particularly rare for me to poke through other distros package 
> patches when I'm trying to figure out how to solve a problem.  I suspect I'm
> not the only one.  I think having explicit patches available is a really good 
> thing for cross-distro collaboration (in addition to upstream collaboration).
> 
> Scott K

I'll second that looking through patches a distro has supplied can be very
useful. It's very convenient to have explicit patches you can just check quickly
to see if it could solve an issue or not.

Re: Summary: Git Packaging: Native source formats

[Sam Hartman]

> "Ian" == Ian Jackson  writes:


Ian> I certainly agree with this.  I don't think anyone is saying
Ian> that using (say) a merging git workflow with a native source
Ian> package format should be universal, or even the default.

Correct.

My take away from this discussion though is that using a native package
format even when there are upstream sources is an appropriate tool for
maintainers to use sometimes.
I think that is a significant change over our feelings years ago.
Back in the day I think there was approach a project consensus that
using native format packages with  upstreams that were native to Debian
was basically always wrong.

My reading of this discussion is that it's a tool that maintainers
should have and it's sometimes reasonable to use that tool.  People have
done a great job of bringing up athings to think about for maintainers
considering using that tool.  I've certainly found the discussion
educational.  It's changed my thinking about when I would choose to use
native format packages in the future.

So, my high level summary is that it's an option maintainers have, but
there are many things maintainers should consider when electing to use
that option.

I think it would be great to capture the items people brought up here in
a wiki page for people to look at when they consider using native format
packages.

I'd be really greatful if someone else made a first cut at turning this
discussion into such a page, but if it doesn't happen I'll work to find
time.

Thanks for all your great things to consider!


--Sam


signature.asc
Description: PGP signature

Re: Git Packaging: Native source formats

[Simon McVittie]

On Fri, 30 Aug 2019 at 09:05:45 -0700, Russ Allbery wrote:
> [git-debcherry] lets you generate the patches for people on demand, but
> they aren't just sitting out there for anyone to look at whenever they want.

I think that's really important from the transparency point of view that
you touched on in a previous mail. If the upstream has an opinion about
a change (either wanting to apply it to their codebase, or thinking
it's wrong and wanting us to stop applying it) and needs the context
of why we made that change, then I think it's important that we make
it as straightforward as possible for them to see what we've done with
their code. IMO part of a downstream maintainer's job is to present the
changes in a way that an upstream can, if not necessarily agree with,
then at least reason about:

* which of their later changes we backported
* which other changes we made, and why
* how we're compiling it (build-time options, etc.)

This is why I try to encourage my colleagues and co-maintainers to
use DEP-3 on their patches, and reorder debian/patches/series with
backports from upstream before upstreamable patches, which are before
Debian-specific changes.

The changes we *used to* make are somewhat secondary: it's important for
downstream maintainers to be able to dig into the history, and it's
occasionally interesting for upstreams to be able to dig into that history
too, but the version(s) we're currently recommending to our users are the
most important.

> Anyway, this is probably only applicable to a minority of upstreams and
> packages, and more upstreams these days would just like all patches as
> PRs.

For all the upstreamable changes yes, but sometimes we have to make
Debian-specific changes that aren't upstreamable, and we should do that
carefully, avoiding the perception that we're randomly forking software
that the upstream maintainer feels strongly about. I'm quite aware
of this as an upstream whose downstreams sometimes make changes that I
wouldn't have accepted, or even changes that I have already rejected.

(It was a particularly surreal experience when I was maintaining dbus in
Debian with a patch included that I had rejected upstream, although
that's fixed in buster.)

smcv

Re: Git Packaging: Native source formats

[Scott Kitterman]

On Friday, August 30, 2019 12:05:45 PM EDT Russ Allbery wrote:
> Ian Jackson  writes:
> > Russ Allbery writes ("Re: Git Packaging: Native source formats"):
> >> [ discussion of benefits of maintaining the Debian delta as
> >> 
> >>   a linear series of broken-down changes ]
> >> 
> >> There are obviously ways to represent this with a pure Git repository,
> >> but
> >> apart from using a patch system on top of 3.0 (native), at which point I
> >> don't understand why one wouldn't just use 3.0 (quilt), they require
> >> multiple branches and thus aren't available directly in the archive.
> > 
> > This is not true.  There are at least two ways of doing this without
> > using a patch system: git-debrebase and git-dpm.
> > 
> > Both of these use only a single primary git branch which contains both
> > upstream history, and Debian changes to upstream files represented as
> > git commits.
> 
> It's a fair point that I didn't account for a rebase workflow in my
> analysis, and that's definitely an alternative.
> 
> However, while analyzing a rebased branch isn't as hard for other people
> as a branch with a complex merge history, it does mean that upstream has
> to find a way to extract patches to their code from a branch that also has
> packaging-only changes and their upstream changes, and this is non-trivial
> for a lot of people.  It's certainly way harder than just pointing them at
> a directory full of patches.
> 
> > This is also not that hard, in simple cases.  There is a tool
> > git-debcherry which can do it automatically.  I haven't used it but AIUI
> > if your Debian delta queue has few commits, and doesn't have commits
> > which involve merge conflicts with upstream merges (basically, if each
> > change is carried Debian only for a short time), it will always produce
> > the nice output you would hope for.
> 
> This lets you generate the patches for people on demand, but they aren't
> just sitting out there for anyone to look at whenever they want.
> 
> I suppose it could be provided as an automated service that publishes the
> results, but that would be a piece of infrastructure someone has to run.
> 
> Anyway, this is probably only applicable to a minority of upstreams and
> packages, and more upstreams these days would just like all patches as
> PRs.

It's not particularly rare for me to poke through other distros package 
patches when I'm trying to figure out how to solve a problem.  I suspect I'm 
not the only one.  I think having explicit patches available is a really good 
thing for cross-distro collaboration (in addition to upstream collaboration).

Scott K

Re: Git Packaging: Native source formats

[Russ Allbery]

Ian Jackson  writes:
> Russ Allbery writes ("Re: Git Packaging: Native source formats"):

>> [ discussion of benefits of maintaining the Debian delta as
>>   a linear series of broken-down changes ]
>>
>> There are obviously ways to represent this with a pure Git repository, but
>> apart from using a patch system on top of 3.0 (native), at which point I
>> don't understand why one wouldn't just use 3.0 (quilt), they require
>> multiple branches and thus aren't available directly in the archive.

> This is not true.  There are at least two ways of doing this without
> using a patch system: git-debrebase and git-dpm.

> Both of these use only a single primary git branch which contains both
> upstream history, and Debian changes to upstream files represented as
> git commits.

It's a fair point that I didn't account for a rebase workflow in my
analysis, and that's definitely an alternative.

However, while analyzing a rebased branch isn't as hard for other people
as a branch with a complex merge history, it does mean that upstream has
to find a way to extract patches to their code from a branch that also has
packaging-only changes and their upstream changes, and this is non-trivial
for a lot of people.  It's certainly way harder than just pointing them at
a directory full of patches.

> This is also not that hard, in simple cases.  There is a tool
> git-debcherry which can do it automatically.  I haven't used it but AIUI
> if your Debian delta queue has few commits, and doesn't have commits
> which involve merge conflicts with upstream merges (basically, if each
> change is carried Debian only for a short time), it will always produce
> the nice output you would hope for.

This lets you generate the patches for people on demand, but they aren't
just sitting out there for anyone to look at whenever they want.

I suppose it could be provided as an automated service that publishes the
results, but that would be a piece of infrastructure someone has to run.

Anyway, this is probably only applicable to a minority of upstreams and
packages, and more upstreams these days would just like all patches as
PRs.

-- 
Russ Allbery (r...@debian.org)   <http://www.eyrie.org/~eagle/>

Re: Git Packaging: Native source formats

[Ian Jackson]

Russ Allbery writes ("Re: Git Packaging: Native source formats"):
> [ discussion of benefits of maintaining the Debian delta as
>   a linear series of broken-down changes ]
>
> There are obviously ways to represent this with a pure Git repository, but
> apart from using a patch system on top of 3.0 (native), at which point I
> don't understand why one wouldn't just use 3.0 (quilt), they require
> multiple branches and thus aren't available directly in the archive.

This is not true.  There are at least two ways of doing this without
using a patch system: git-debrebase and git-dpm.

Both of these use only a single primary git branch which contains both
upstream history, and Debian changes to upstream files represented as
git commits.

I can't speak for git-dpm, but with git-debrebase and 1.0 native
source format, there would not any patch files.

> Extracting specific changes by comparing only two Git branches with a
> complex merge history is certainly possible to do with native Git tools,
> but I would classify it as an advanced Git skill.  I think there are a lot
> of upstreams using Git for whom that operation would still be quite
> challenging.

This is also not that hard, in simple cases.  There is a tool
git-debcherry which can do it automatically.  I haven't used it but
AIUI if your Debian delta queue has few commits, and doesn't have
commits which involve merge conflicts with upstream merges (basically,
if each change is carried Debian only for a short time), it will
always produce the nice output you would hope for.

> This is one of those cases where knowing your upstream is invaluable.

I certainly agree with this.  I don't think anyone is saying that
using (say) a merging git workflow with a native source package format
should be universal, or even the default.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Git Packaging: Native source formats [and 1 more messages]

[Ian Jackson]

Philipp Kern writes ("Re: Git Packaging: Native source formats"):
> While this may be true on some level, it is also important to be able to
> build packages from checked-out source trees (say, git repositories)
> without an original source present.

Quite.

For example, if one wants to build binaries with sbuild, it is (right
now, anyway) necessary to build a source package because that's how
sbuild transports the source into the build environment.  Right now I,
horrifyingly, have to advise [1] users that in some circumstances they
should run this command:

sbuild -c stretch -A --no-clean-source \
 --dpkg-source-opts='-Zgzip -z1 --format=1.0 -sn'

[1] https://manpages.debian.org/testing/dgit/dgit-user.7.en.html#Using_sbuild

Simon Richter writes ("Re: Git Packaging: Native source formats"):
> On Thu, Aug 29, 2019 at 09:42:50PM +0200, Philipp Kern wrote:
> > Obviously I'm not bound to that format being "3.0 (native)" but some
> > "3.0 (dumb)" that just tars up the whole tree without caring about the
> > version scheme would then be nice to have as a replacement. ;-)
> 
> Are you planning to upload these?

Obviously not to Debian.  I don't think that invalidates the point.
Users are supposed to be able to modify and build software on their
own systems without any expectation that the result will go to Debian.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Git Packaging: Native source formats

[Ian Jackson]

Theodore Y. Ts'o writes ("Re: Git Packaging: Native source formats"):
> On Thu, Aug 29, 2019 at 11:23:01AM +0100, Ian Jackson wrote:
> > I think dgit ought to be compatible with the idea of shipping
> > upstream's .asc's about, but maybe there are bugs.  I don't ever do
> > this so I don't know if it works and I doubt there are tests for it.
> > 
> > So, if you have a package where you want to use dgit push and you find
> > the upstream .asc is not being included, please file bug(s).
> 
> The problem I have is that "dgit gbp" doesn't extract the upstream
> .asc.  Not a big deal, I use /tmp/gbp for my build directory, and I
> manually checkout and populate it with the .asc file.  But building
> from "dgit clone" won't generate same package as I do (which includes
> the .asc file for the orig.tar.gz file.)

When you say

  building from "dgit clone" won't generate same package as I do
  (which includes the .asc file for the orig.tar.gz file.

I'm not an expert on .asc handling but I think what you mean is

  Steps to reproduce

0. mkdir bpd
1. dgit --build-products-dir=../bpd clone FOO
2. cd FOO
3. dch -i wombat
4. git commit -a -m changelog
3. dgit --build-products-dir=../bpd gbp-build

  Expected behaviour

The generated .changes, and the generated .dsc contain
FOO_VSN.orig.tar.gz.asc, just as they are in uploads not
made with dgit.

  Observed behaviour

The .asc is not mentioned in either the generated .dsc or the
generated .changes.

?  That would be a bug.  Can you file it and tell me the value of
`FOO' please :-) ?

(I'm not sure exactly what you mean by "doesn't extract the upstream
.asc" since I don't know what it would be "extracted" from.  I tried
to see if I could reproduce for myself, but a quick check through this
subthread didn't turn up the package name.)

Ian.

PS hardcoding directories in /tmp in one's finger macros usually
amounts to embedding tmpfile race vulnerabilities in one's brain.
Even if one's own machine has per-user /tmp, one's wetware becomes
vulnerable when combined with a differently configured computer.  IMO
one should make and use a ~/tmp or something.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Git Packaging: Native source formats

[Colin Watson]

On Fri, Aug 30, 2019 at 12:29:45AM +0200, Thomas Goirand wrote:
> Now, you're talking about upstream using bzr or hg. These are the very
> few minority (and counting...). We may as well get rid of hg and bzr in
> Debian if it doesn't get fixed so it uses Python 3 only... (well, I
> guess someone will wake up and do the work, so this argument doesn't
> count...).

I appreciate that this is a side note from the point you're making, but
bzr was forked as brz ("breezy"); that fork supports Python 3 and is
compatible with existing bzr branches.  I expect we will indeed need to
remove bzr at some point, but that shouldn't be a major problem for most
upstreams still using it.

-- 
Colin Watson   [cjwat...@debian.org]

Re: Git Packaging: Native source formats

[Colin Watson]

On Thu, Aug 29, 2019 at 01:26:08PM -0700, Russ Allbery wrote:
> While this is not an argument against *ever* using 3.0 (native) or some
> equivalent when packaging upstream software, I have found it to help
> relationships with upstream considerably in the past to represent the
> package as their software as released plus a set of Debian patches.
[...]
> There's also a psychological value for some upstreams to representing all
> changes to upstream as patches.  If they're worried Debian is doing some
> weird nonsense (from their perspective) to their package, they can look at
> debian/rules for build flags and then at debian/patches and see exactly
> what we've done, and feel assured that the rest of the package is exactly
> as they released it.  That's easy for them to do; downloading the full
> package and then diffing against their tree is more awkward and can't be
> done with some casual web browsing.

Yes.  In fact, not only upstreams, but in some cases downstreams and
other people with a stake in how the package is put together.

When I converted Debian's OpenSSH packaging to 3.0 (quilt), it was
partly because a company using my packages was concerned about the
difficulty about picking apart the changes in the Debian diff so that
they could audit the logical changes that I was making; even though they
were a large company with lots of highly-competent staff, just providing
a public VCS that they could look through and pick apart the monolithic
delta for themselves still turned out to be a prohibitively difficult
barrier.  Doing the work to provide cleanly-separated patches (which in
many cases evolve over time) was a useful exercise for me and made
things instantly much more comprehensible for them.

This was a while ago, of course, but it was that experience that
converted me to being convinced that providing broken-out patches in an
easy-to-consume format is an invaluable service to our users.  quilt
itself is very much a lowest common denominator and I certainly think
it's best treated as an export format from richer git history; but just
exporting stuff from git without that annotation of logically-divided-up
differences from upstream isn't an adequate substitute.

> This is one of those cases where knowing your upstream is invaluable.  A
> lot of upstreams won't care in the slightest, some upstreams are Git
> experts and know exactly how to get the data they care about, and others
> are still using Subversion or even CVS and will find anything related to
> Git opaque and frustrating.  Some upstreams actively encourage downstream
> modifications and will seek them out and incorporate them, some upstreams
> are happily oblivious and only look at patches or PRs submitted to them,
> and some upstreams will get actively upset at what they view as
> unmotivated or incompatible changes and may require delicate handling
> (part of which, in my experience, is effortless transparency so that they
> don't feel like anything is being hidden from them).

And also bear in mind the case where doing a git clone maybe takes quite
a while over a slow network link, but going to salsa or
sources.debian.org and having a quick look through debian/patches/ takes
a fraction of the time and is all you need.

-- 
Colin Watson   [cjwat...@debian.org]

Re: Git Packaging: Native source formats

[Simon Richter]

Hi,

On Thu, Aug 29, 2019 at 09:42:50PM +0200, Philipp Kern wrote:

> Obviously I'm not bound to that format being "3.0 (native)" but some
> "3.0 (dumb)" that just tars up the whole tree without caring about the
> version scheme would then be nice to have as a replacement. ;-)

Are you planning to upload these?

   Simon

Re: Git Packaging: Native source formats

[Theodore Y. Ts'o]

On Thu, Aug 29, 2019 at 11:23:01AM +0100, Ian Jackson wrote:
> Theodore Y. Ts'o writes ("Re: Git Packaging: Native source formats"):
> > Or if we end up moving to dgit for everything, and we don't want to
> > use pristine-tar (which I like, but I realize that's not an opinion
> > shared by everyone; some people seem to hate it), and upstream uses a
> > non-git repo (say, bzr, or hg) and still uses signed tar.gz files, I'd
> > argue we need to have a good way to reserve the cryptographic
> > signature of upstream's foo.tar.gz and foo.tar.gz.asc in a dgit-only
> > world.
> 
> I think dgit ought to be compatible with the idea of shipping
> upstream's .asc's about, but maybe there are bugs.  I don't ever do
> this so I don't know if it works and I doubt there are tests for it.
> 
> So, if you have a package where you want to use dgit push and you find
> the upstream .asc is not being included, please file bug(s).

The problem I have is that "dgit gbp" doesn't extract the upstream
.asc.  Not a big deal, I use /tmp/gbp for my build directory, and I
manually checkout and populate it with the .asc file.  But building
from "dgit clone" won't generate same package as I do (which includes
the .asc file for the orig.tar.gz file.)

Cheers,

- Ted

Re: Git Packaging: Native source formats

[Theodore Y. Ts'o]

On Fri, Aug 30, 2019 at 12:29:45AM +0200, Thomas Goirand wrote:
> 
> Pristine-tar forces you to have multiple branches when you may only need
> a single one. It's also not reliable and may easily generate different
> tarballs for the same tag, which defeats its purpose (and no, the
> workaround to remove all timestamps and order files isn't acceptable).

Huh?  When does pristine-tar be unreliable?  At least in my
experience, I can store both the GPG signature and upstream tar file
using pristine tar, and it's never failed to reproduce the exact tar
file such that the GPG signature verifies.

I'm not even sure what you mean by "remove all timestamps and order
files".  The whole point of pristine tar is that you don't have to do
that, and you can preserve the exact binary that was uploaded, and
signed, by the upstream.

> Not only that. If upstream uses git, then I just fetch from it, and use
> the upstream tag as reference to run "git archive", which does the job
> very well.

But not all upstreams use git...

> Now, you're talking about upstream using bzr or hg. These are the very
> few minority (and counting...). We may as well get rid of hg and bzr in
> Debian if it doesn't get fixed so it uses Python 3 only... (well, I
> guess someone will wake up and do the work, so this argument doesn't
> count...).

And if even if they do use git, they may not use signed tags.  Some of
them also do post-processing to generate the tar.gz file which gets
distributed.  (Example: util-linux).  Hence, using "git archive" isn't
a substitute.

I'm arguing that we need to have a sane soluion for those upstreams
that don't use git, or who use git but who don't use signed tags, or
who do post-processing to generate the tar file.  Hopefully you're not
arguing that we should just summarily eject any packages from Debian
which don't use git, and refuse to package any package which isn't
compatible with our new git packaging philosophy?

- Ted

Re: Git Packaging: Native source formats

[Thomas Goirand]

On 8/29/19 1:49 AM, Theodore Y. Ts'o wrote:
> Or if we end up moving to dgit for everything, and we don't want to
> use pristine-tar (which I like, but I realize that's not an opinion
> shared by everyone; some people seem to hate it

/me raises hand. I hate it.

Pristine-tar forces you to have multiple branches when you may only need
a single one. It's also not reliable and may easily generate different
tarballs for the same tag, which defeats its purpose (and no, the
workaround to remove all timestamps and order files isn't acceptable).

Not only that. If upstream uses git, then I just fetch from it, and use
the upstream tag as reference to run "git archive", which does the job
very well.

Now, you're talking about upstream using bzr or hg. These are the very
few minority (and counting...). We may as well get rid of hg and bzr in
Debian if it doesn't get fixed so it uses Python 3 only... (well, I
guess someone will wake up and do the work, so this argument doesn't
count...).

Cheers,

Thomas Goirand (zigo)

Re: Git Packaging: Native source formats

[Russ Allbery]

Sam Hartman  writes:

> Using native source formats (1.0 and 3.0 (native)) is attractive for
> some git workflows.  It means you can just export the git repository and
> don't need to make sure that you use the same upstream tarball when
> upstream versions are the same.  You don't need to synthesize a separate
> upstream tarball.

> It seems particularly attractive when upstream doesn't produce tarballs
> and instead does their development in git.

> I'd like to understand the downsides to doing this:

While this is not an argument against *ever* using 3.0 (native) or some
equivalent when packaging upstream software, I have found it to help
relationships with upstream considerably in the past to represent the
package as their software as released plus a set of Debian patches.  For
at least some upstreams, this is a very natural workflow; they can pull
patches easily at their leisure without requiring the Debian package
maintainer to keep updating a rebased patch submission.

Even with upstreams that switch to a PR workflow, sometimes it takes a
while to convince upstreams to even look at patches, and having a
cleanly-separated patch in the packaging makes it very easy to submit a PR
when upstream is ready.

There are obviously ways to represent this with a pure Git repository, but
apart from using a patch system on top of 3.0 (native), at which point I
don't understand why one wouldn't just use 3.0 (quilt), they require
multiple branches and thus aren't available directly in the archive.

Extracting specific changes by comparing only two Git branches with a
complex merge history is certainly possible to do with native Git tools,
but I would classify it as an advanced Git skill.  I think there are a lot
of upstreams using Git for whom that operation would still be quite
challenging.

There's also a psychological value for some upstreams to representing all
changes to upstream as patches.  If they're worried Debian is doing some
weird nonsense (from their perspective) to their package, they can look at
debian/rules for build flags and then at debian/patches and see exactly
what we've done, and feel assured that the rest of the package is exactly
as they released it.  That's easy for them to do; downloading the full
package and then diffing against their tree is more awkward and can't be
done with some casual web browsing.

This is one of those cases where knowing your upstream is invaluable.  A
lot of upstreams won't care in the slightest, some upstreams are Git
experts and know exactly how to get the data they care about, and others
are still using Subversion or even CVS and will find anything related to
Git opaque and frustrating.  Some upstreams actively encourage downstream
modifications and will seek them out and incorporate them, some upstreams
are happily oblivious and only look at patches or PRs submitted to them,
and some upstreams will get actively upset at what they view as
unmotivated or incompatible changes and may require delicate handling
(part of which, in my experience, is effortless transparency so that they
don't feel like anything is being hidden from them).

-- 
Russ Allbery (r...@debian.org)   

Re: Git Packaging: Native source formats

[Sune Vuorela]

On 2019-08-28, Sam Hartman  wrote:
> * I've heard at least one person claim that native format packages are
>   problematic for downstreams.
>   I'd like to better understand both the theoretical argument here and
>   to understand from downstreams whether it is an issue in practice.
>
>   For downstreams where it is a problem, are you using a git or a
>   non-git workflow?

I've recently (professionally) been involved with a .. sidestream?.
Where going into debian packages, rpm packages and others to fetch their
fixes (as separate files) into a different environment (yocto).

Having the ability to extract separate patches - with clear
documentation - was so much simpler than trying to extract it from
anywhere else.


/Sune

Re: Git Packaging: Native source formats

[Philipp Kern]

On 8/29/2019 8:32 PM, Andrej Shadura wrote:
>> So `3.0 (native)' is not strictly better than 1.0.  dpkg-source
>> refuses to work in the situation where I am saying (and you seem to be
>> agreeing) that it shouldn't even print a warning ...
> 
> I have to disagree with you but I consider this strictly an
> improvement. Allowing native packages with non-native versions
> significantly increases complexity of code handling Debian source
> packages. Not even all Debian tools support this case; arguably it
> should not be supported at all as often leads to malformed packages
> being uploaded to the archive.

While this may be true on some level, it is also important to be able to
build packages from checked-out source trees (say, git repositories)
without an original source present.

For instance at work we check in whole Debian packages as-is (including
their non-native version) to fork and then modify them. Changing the
versioning scheme is pretty disruptive there. For people unfamiliar to
Debian the diff is already represented in the VCS and there is no
technical need to have this conveyed in the intermediate source package
representation that is only needed to feed the build to the build system.

Of course one workaround is to always build from the build tree and to
always specify -b/-B and never build a source package at all.
Unfortunately the various defaults in Debian's toolchain don't make that
as easy as it should be. Some can be addressed through wrapper scripts,
but then it's odd to anyone familiar with Debian.

Obviously I'm not bound to that format being "3.0 (native)" but some
"3.0 (dumb)" that just tars up the whole tree without caring about the
version scheme would then be nice to have as a replacement. ;-)

Kind regards
Philipp Kern

Re: Git Packaging: Native source formats

[Simon McVittie]

On Thu, 29 Aug 2019 at 16:25:35 +0100, Ian Jackson wrote:
> Simon McVittie writes ("Re: Git Packaging: Native source formats"):
> > for
> > version 1.0 source packages, detecting a non-native version with a native
> > source format is the only way to generate any sort of warning about this.
> 
> This is a good point.  Of course it's only a warning and warnings are
> often ignored when you're in a hurry.

Sure, but it's better than nothing.

I was half tempted to suggest making the tuple (version 1.0 source,
non-native version number, native source format) into a fatal error,
but obviously that's a compatibility break - but if there's no reason
to prefer version 1.0 over version 3.0 (native), then maybe it's a
compat break that would be worth it to get rid of one of the traps
inherent in format 1.0 implicitly choosing between native and non-native?

> So we ask the users of
> format 1.0 source packages to add an appropriate setting to
> debian/source/options.
>
> Anyway, it seems to me that if debian/source/format says `3.0
> (native)' then the warning is inappropriate.

Maybe the setting you say is appropriate should also silence such
warnings. Do you know which tools produce them? The Lintian maintainers
are generally receptive to patches for this sort of thing.

smcv

Re: Git Packaging: Native source formats

[Ian Jackson]

Ian Jackson writes ("Re: Git Packaging: Native source formats"):
> Simon McVittie writes ("Re: Git Packaging: Native source formats"):
> > Unlike 1.0 (non-native) vs. 3.0 (quilt), where some people prefer one and
> > some people prefer the other, I am not aware of any advantages of 1.0
> > (native) over 3.0 (native). If 3.0 (native) is indeed strictly better
> > than 1.0 (native), perhaps it would be reasonable to say that packages
> > that intentionally have a non-native version number but a native
> > source format should declare this explicitly, by using "3.0 (native)"
> > in d/source/format? That way, if a version 1.0 source package has a
> > non-native version number, tools can assume that it was meant to have
> > a .orig, and issue warnings; conversely, if a source package with a
> > non-native version number explicitly has "3.0 (native)" format, tools
> > could assume that the maintainer wants what they asked for.
> 
> Perhaps.  I have a vague feeling that there might be (or have once
> been?) some reason to prefer 1.0 (native) to 3.0 (native) but I can't
> bring it to mind, and now that I try to think about it it's all just
> fog.  Maybe I am remembering some years-old abundance of caution.

Private email prompted me to check something, and I was right to
remember that there was a difficulty.  dpkg-source refuses to create a
`3.0 (native)' package with a Debian revision number:

$ dpkg-source  -b .
dpkg-source: error: can't build with source format '3.0 (native)': native 
package version may not have a revision
$

It looks like the dpkg maintainers are adamant that dpkg-source is
correct.  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=737634

So `3.0 (native)' is not strictly better than 1.0.  dpkg-source
refuses to work in the situation where I am saying (and you seem to be
agreeing) that it shouldn't even print a warning ...

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Git Packaging: Native source formats

[Ian Jackson]

Milan Kupcevic writes ("Re: Git Packaging: Native source formats"):
> I've also seen developers deleting a git tag and then creating a new
> git tag using exactly the same name/release number pointing to
> different commit.

It is possible to avoid some of these problems by using a git server
which does not permit tag rewriting and/or branch rewinding.

The dgit git server has these properties: once you have done `dgit
push' and the tag is on the dgit git server, it can't be deleted or
changed.  Likewise, the suite branches that you push to are
fast-forwarding.  Breaching these rules would involve administrator
intervention; as the administrator I would expect to do that only in
very rare situations, where we need to rewrite history eg to remove
legally dangerous objects.

(Because of the archive's policy about new packages, which permits
reuse of version numbers, there is a corresponding exception for
packages which are still entirely in NEW.)

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Git Packaging: Native source formats

[Ian Jackson]

Simon McVittie writes ("Re: Git Packaging: Native source formats"):
> On Thu, 29 Aug 2019 at 11:56:55 +0100, Ian Jackson wrote:
> > If you already don't care about bit-identical upstream tarballs, then
> > dealing with these tarballs is a reasonably well-solved problem.
> > git-deborig etc. FTW.
> 
> I think it's important to distinguish between the two things that you
> might mean by "bit-identical upstream tarballs": you might not care
> whether the orig tarball is bit-identical to upstream's official release
> artifact (assuming they have one), but you still have to care about
> the orig tarball being bit-identical to any orig tarball of the same
> name that might previously have been uploaded to the Debian archive,
> because otherwise the archive will reject your uploads.

Good point.  I meant "bit-identical to upstream".

> Does/can git-deborig guarantee that a given git commit will always
> produce the same tarball?

No, it doesn't.  But it uses git-archive and in practice that nearly
always gives you the same tarball - even though that's not guaranteed.
I haven't seen any trouble (but I normally use `dgit clone' or `dgit
fetch' and that arranges for any existing origs in the archive to be
in the right place on my laptop, so perhaps that means I avoid trouble
that others see).

> I think that last point is a significant reason to want the nativeness
> of the source package to match the nativeness of the version number: for
> version 1.0 source packages, detecting a non-native version with a native
> source format is the only way to generate any sort of warning about this.

This is a good point.  Of course it's only a warning and warnings are
often ignored when you're in a hurry.

> Unlike 1.0 (non-native) vs. 3.0 (quilt), where some people prefer one and
> some people prefer the other, I am not aware of any advantages of 1.0
> (native) over 3.0 (native). If 3.0 (native) is indeed strictly better
> than 1.0 (native), perhaps it would be reasonable to say that packages
> that intentionally have a non-native version number but a native
> source format should declare this explicitly, by using "3.0 (native)"
> in d/source/format? That way, if a version 1.0 source package has a
> non-native version number, tools can assume that it was meant to have
> a .orig, and issue warnings; conversely, if a source package with a
> non-native version number explicitly has "3.0 (native)" format, tools
> could assume that the maintainer wants what they asked for.

Perhaps.  I have a vague feeling that there might be (or have once
been?) some reason to prefer 1.0 (native) to 3.0 (native) but I can't
bring it to mind, and now that I try to think about it it's all just
fog.  Maybe I am remembering some years-old abundance of caution.

Sean and I faced an aspect of this problem with tag2upload and 1.0
source format.  (We didn't want to just say 1.0 wasn't supported.)
The information in git needs to specify whether the output is supposed
to be a native package.  Obviously git-debpush shouldn't look for
tarballs in your `..', like dpkg-source does.  So we ask the users of
format 1.0 source packages to add an appropriate setting to
debian/source/options.

Anyway, it seems to me that if debian/source/format says `3.0
(native)' then the warning is inappropriate.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Git Packaging: Native source formats

[Milan Kupcevic]

On 8/28/19 4:00 PM, Sam Hartman wrote:
> 
> Back in the day, one of the big reasons for separating .orig.tar.gz from
> .diff.gz was to reuse upstream tarballs for space reasons, both in terms
> of space on mirrors when the pool had two Debian revisions with the same
> upstream, as well as to reduce upload time.
> 

Besides the clear separation of upstream content and downstream tweaks
an important quality of this concept is immutability of the upstream
code that has already been obtained, inspected, signed by a DD/DM and
subsequently deposited into Debian archive.

I've seen cases where obtaining an upstream tarball from an official
upstream website at two different points in time results in slightly
different content. I've also seen developers deleting a git tag and then
creating a new git tag using exactly the same name/release number
pointing to different commit. And finally downloading a dynamically
created tarball based on a git tag does not mean you'll get exactly the
same content every time either because the currently running
archiving/compressing tools have changed or because of already mentioned
human factors.

Git commit hash is somewhat more reliable than a git tag but reliance on
SHA-1 hash has also been mentioned as a step backward as Debian has
moved on to a more advanced hash algorithm. These and other possibly
unforeseen problems make the concept of immutable upstream code
deposited in the Debian archive very appealing and surely more reliable
compared to the Git tag method of sourcing.

Milan



signature.asc
Description: OpenPGP digital signature

Re: Git Packaging: Native source formats

[Simon McVittie]

On Thu, 29 Aug 2019 at 11:56:55 +0100, Ian Jackson wrote:
> If you already don't care about bit-identical upstream tarballs, then
> dealing with these tarballs is a reasonably well-solved problem.
> git-deborig etc. FTW.

I think it's important to distinguish between the two things that you
might mean by "bit-identical upstream tarballs": you might not care
whether the orig tarball is bit-identical to upstream's official release
artifact (assuming they have one), but you still have to care about
the orig tarball being bit-identical to any orig tarball of the same
name that might previously have been uploaded to the Debian archive,
because otherwise the archive will reject your uploads.

Does/can git-deborig guarantee that a given git commit will always
produce the same tarball?

> One thing I would like to see changed in this area: some of our tools
> issue warnings about packages with a non-native version, but a native
> source format.  I think there is nothing wrong with this and it should
> not be discouraged and doesn't merit a warning.  (There is an
> awkardness here in that you can sometimes unintentionally generate a
> native format source package if your origs are missing...)

I think that last point is a significant reason to want the nativeness
of the source package to match the nativeness of the version number: for
version 1.0 source packages, detecting a non-native version with a native
source format is the only way to generate any sort of warning about this.

Unlike 1.0 (non-native) vs. 3.0 (quilt), where some people prefer one and
some people prefer the other, I am not aware of any advantages of 1.0
(native) over 3.0 (native). If 3.0 (native) is indeed strictly better
than 1.0 (native), perhaps it would be reasonable to say that packages
that intentionally have a non-native version number but a native
source format should declare this explicitly, by using "3.0 (native)"
in d/source/format? That way, if a version 1.0 source package has a
non-native version number, tools can assume that it was meant to have
a .orig, and issue warnings; conversely, if a source package with a
non-native version number explicitly has "3.0 (native)" format, tools
could assume that the maintainer wants what they asked for.

smcv

Re: Git Packaging: Native source formats

[Simon Richter]

Hi,

On Wed, Aug 28, 2019 at 04:00:10PM -0400, Sam Hartman wrote:

> Back in the day, one of the big reasons for separating .orig.tar.gz from
> .diff.gz was to reuse upstream tarballs for space reasons, both in terms
> of space on mirrors when the pool had two Debian revisions with the same
> upstream, as well as to reduce upload time.

Saving space and bandwidth when only changing the packaging was a nice
bonus, but the main reason we use upstream tarballs verbatim is
verifiability (which you also mention later on).

> Using native source formats (1.0 and 3.0 (native)) is attractive for
> some git workflows.  It means you can just export the git repository and
> don't need to make sure that you use the same upstream tarball when
> upstream  versions  are the same.
> You don't need to synthesize a separate upstream tarball.

FWIW, I don't think that this is much of an impediment. "git archive"
exists and is deterministic, so if we can also compress the archive in a
reproducible way, then the "upstream" tarball will be identical.

> It seems particularly attractive when upstream doesn't produce tarballs
> and instead does their development in git.

Lots of upstream projects use git and still produce tarballs for releases.

What we don't have a good workflow for is upstream software that does not
have distinct releases, and frankly, the technical aspect of that is the
smallest part of the problem.

We already have lots of software in the archive that upstream will not
support, essentially telling users to update to the most recent version.

Unless the package maintainer is willing to support the users themselves in
that case, I would say that including that software in a stable release is
not beneficial for users, and making it easier for Debian contributors to
package this software is not going to improve things.

Not everything needs to be packaged, and even less needs to be included in
releases. As a maintainer, it is a completely valid technical decision to
file an RC bug on your own package "this should not go into the release",
and the stable release managers will not ask you to close the bug so the
package can be included.

We need to work together with upstream to identify a state of the software
that we feel comfortable supporting (also together with upstream) for the
lifetime of a Debian release.

No change in the packaging workflow will make this part any easier.

>   It's clear that we value integrity of upstream source.  That is we
>   want to make it easy for people to start from some upstream source
>   that is trusted because upstream has produced it and audit just our
>   changes.
>   One way to do this is with an upstream tarball and a diff (or set of
>   diffs or a debian directory).

>   But if we're thinking that people will be working in Git, another way
>   to do this is to merge in a signed upstream git tag.  Then you can
>   perform a diff against that git tag.

That requires speaking a complex protocol in which a program that is not
expecting malicious input repeatedly parses and decompresses untrusted data
before it is able to verify any signatures. It also assumes availability of
unfiltered Internet services over which cryptographic signatures can be
exchanged.

   Simon

Re: Git Packaging: Native source formats

[Ian Jackson]

Sam Hartman writes ("Git Packaging: Native source formats"):
> Internet is faster and disks are cheaper.
> I assert this is much less of a concern than it used to be.

We have some extremely large packages.  Also we have users in places
where internet is slow and/or expensive.

Even for medium-sized packages, it is not desirable to constantly ship
copies of the source code about.  I want to be able to upload from a
train wifi, or when roaming abroad on hideously expensive mobile data.

Having said that, I do have some packages which have an upstream where
I don't bother with orig tarballs and just use a native format.

> Using native source formats (1.0 and 3.0 (native)) is attractive for
> some git workflows.  It means you can just export the git repository and
> don't need to make sure that you use the same upstream tarball when
> upstream  versions  are the same.
> You don't need to synthesize a separate upstream tarball.

If you already don't care about bit-identical upstream tarballs, then
dealing with these tarballs is a reasonably well-solved problem.
git-deborig etc. FTW.

The real cost of non-native packages is not the few commands you have
to type for your own packages.  It's the cognitive and codebase cost
of knowing how to deal with them, particularly when you're not the
maintainer.  Especially, `3.0 (quilt)' is very complex.  So we would
gain something if we could get rid of non-native packages *entirely*.
But we can't.

Note that many maintainers use a bare debian ("packaging only")
packaging workflow which doesn't work (at least with current tooling)
without orig tarballs.  You'll have seen messages where some of those
people say they think that is the only tolerable approach.

> I'd like to understand the downsides to doing this:

You have been a bit vague about what exactly you are proposing.  If
you are just proposing that native packages should be a strong
recommendation then the benefits are IMO too weak.

OTOH if people feel they "shouldn't" use a native source format for a
small package for some reason, I think that would be worth fixing.

One thing I would like to see changed in this area: some of our tools
issue warnings about packages with a non-native version, but a native
source format.  I think there is nothing wrong with this and it should
not be discouraged and doesn't merit a warning.  (There is an
awkardness here in that you can sometimes unintentionally generate a
native format source package if your origs are missing...)

HTH.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Git Packaging: Native source formats

[Ian Jackson]

Theodore Y. Ts'o writes ("Re: Git Packaging: Native source formats"):
> Or if we end up moving to dgit for everything, and we don't want to
> use pristine-tar (which I like, but I realize that's not an opinion
> shared by everyone; some people seem to hate it), and upstream uses a
> non-git repo (say, bzr, or hg) and still uses signed tar.gz files, I'd
> argue we need to have a good way to reserve the cryptographic
> signature of upstream's foo.tar.gz and foo.tar.gz.asc in a dgit-only
> world.

I think dgit ought to be compatible with the idea of shipping
upstream's .asc's about, but maybe there are bugs.  I don't ever do
this so I don't know if it works and I doubt there are tests for it.

So, if you have a package where you want to use dgit push and you find
the upstream .asc is not being included, please file bug(s).

Thanks,
Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Git Packaging: Native source formats

[Raphael Hertzog]

Hi,

On Wed, 28 Aug 2019, Sam Hartman wrote:
> Internet is faster and disks are cheaper.

I'm not sure that DSA (which is maintaining snapshot.debian.org) will
be happy with this assertion.

> Using native source formats (1.0 and 3.0 (native)) is attractive for
> some git workflows.  It means you can just export the git repository and
> don't need to make sure that you use the same upstream tarball when
> upstream  versions  are the same.
> You don't need to synthesize a separate upstream tarball.

Do we really care about this if we assume that this work can be
transparent to the maintainer and entirely handled by tag2upload
(or some other equivalent service)?

It has all the information required to be able to build something
reusing an upstream tarball. It can:
- pick a pre-existing tarball if already in the archive
- pick the upstream tarball via the watch file
- pick the output of "git archive"

If there are still differences due to the use of an external tarball, it
can generate a supplementary patch to bring the content in line with the
content in the git repository.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Re: Git Packaging: Native source formats

[Sam Hartman]

> "Paul" == Paul Wise  writes:

Paul> On Thu, Aug 29, 2019 at 4:00 AM Sam Hartman wrote:
>> Using native source formats (1.0 and 3.0 (native)) is attractive
>> for some git workflows.  It means you can just export the git
>> repository and don't need to make sure that you use the same
>> upstream tarball when upstream versions are the same.

Paul> Why use the native format instead of the git format (which
Paul> uses git-bundle)?

I like being able to upload my packages.  As I understand it dak and a
lot of other tools don't support 3.0 (git).  So if I use that format, my
uploads will get rejected.
I find that experience unsatisfying when I want to get a package into
Debian.

Re: Git Packaging: Native source formats

[Paul Wise]

On Thu, Aug 29, 2019 at 4:00 AM Sam Hartman wrote:

> Using native source formats (1.0 and 3.0 (native)) is attractive for
> some git workflows.  It means you can just export the git repository and
> don't need to make sure that you use the same upstream tarball when
> upstream  versions  are the same.

Why use the native format instead of the git format (which uses git-bundle)?

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Git Packaging: Native source formats

[Theodore Y. Ts'o]

On Wed, Aug 28, 2019 at 04:00:10PM -0400, Sam Hartman wrote:
> 
>   But if we're thinking that people will be working in Git, another way
>   to do this is to merge in a signed upstream git tag.  Then you can
>   perform a diff against that git tag.

One of the things to consider is how we should handle cases where
upstream does not sign the git tag, but *do* sign the tar.gz files.

Or if we end up moving to dgit for everything, and we don't want to
use pristine-tar (which I like, but I realize that's not an opinion
shared by everyone; some people seem to hate it), and upstream uses a
non-git repo (say, bzr, or hg) and still uses signed tar.gz files, I'd
argue we need to have a good way to reserve the cryptographic
signature of upstream's foo.tar.gz and foo.tar.gz.asc in a dgit-only
world.

 - Ted

Re: Git Packaging: Native source formats

[Jeremy Stanley]

On 2019-08-28 16:00:10 -0400 (-0400), Sam Hartman wrote:
[...]
> It seems particularly attractive when upstream doesn't produce
> tarballs and instead does their development in git.
[...]

Not to be a pedant, and it probably wasn't what you meant to imply
either, but I want to be clear that upstreams can produce tarballs
as their official distribution format while also performing their
development in Git repositories. In some cases where I'm directly
involved, information like changelogs, release notes and version
numbers are handled as Git metadata and then "exported" into the
tarball artifact via a sort of `make dist` activity. The signed
tarballs are treated as the actual release artifact, and the Git
repositories are considered an upstream development implementation
detail.

While some package maintainers may see this as a reason to prefer
packaging those projects directly from tagged Git repository states,
that does necessarily imply performing similar steps to generate or
extract this metadata for use in their packages. Others may simply
wish to consume the prepared tarballs where this step has already
been performed for them.
-- 
Jeremy Stanley


signature.asc
Description: PGP signature

Git Packaging: Native source formats

[Sam Hartman]

Hi.

Back in the day, one of the big reasons for separating .orig.tar.gz from
.diff.gz was to reuse upstream tarballs for space reasons, both in terms
of space on mirrors when the pool had two Debian revisions with the same
upstream, as well as to reduce upload time.

Internet is faster and disks are cheaper.
I assert this is much less of a concern than it used to be.
It may be there is some small fraction of packages where this is still
an issue.  I have a question into mirror admins and ftpmaster that they
had not gotten back to in a couple of weeks.


Using native source formats (1.0 and 3.0 (native)) is attractive for
some git workflows.  It means you can just export the git repository and
don't need to make sure that you use the same upstream tarball when
upstream  versions  are the same.
You don't need to synthesize a separate upstream tarball.

It seems particularly attractive when upstream doesn't produce tarballs
and instead does their development in git.


I'd like to understand the downsides to doing this:

* One is that you're not using upstream tarballs.  If upstream has
  tarballs they produce, we're not using them.  I guess we may end up
  having that part of the conversation now rather than later.

  It's clear that we value integrity of upstream source.  That is we
  want to make it easy for people to start from some upstream source
  that is trusted because upstream has produced it and audit just our
  changes.
  One way to do this is with an upstream tarball and a diff (or set of
  diffs or a debian directory).

  But if we're thinking that people will be working in Git, another way
  to do this is to merge in a signed upstream git tag.  Then you can
  perform a diff against that git tag.

* I've heard at least one person claim that native format packages are
  problematic for downstreams.
  I'd like to better understand both the theoretical argument here and
  to understand from downstreams whether it is an issue in practice.

  For downstreams where it is a problem, are you using a git or a
  non-git workflow?

Thanks for comments,

--Sam