Bug#1007717: Native source package format with non-native version

[Lucas Nussbaum]

Hi Ian,

On 15/03/22 at 16:29 +, Ian Jackson wrote:
> Part I - belss continued use of 1.0 native format, for now at least:
> 
>  1. Declare explicitly that there is nothing wrong with a package with
> a native format, but a non-native version number.
> 
>  2. Request that the dpkg maintainer relax the restriction which
> prevents the use of 3.0 native with Debian revision.
> 
>  3. Consequently, declare that the recent MBF on this topic ought not
> to have been filed against packages where simply changing the
> source format does not currently work.  That would include at
> least 1.0 native packages with Debian revisions.
> 
> Part II - bless continued use of 1.0-with-diff, for now at least:
> 
>  4. Declare that sometimes the use of 1.0-with-diff can be the best
> tradeoff between different considerations.  In particular,
> because 1.0 is the only format which botH:
>  (a) Optimises bandwidth and storage by reusing the upstream 
>  data when it hasn't changed.
>  (b) Avoids polluting the working tree (package source code)
>  with [patches], which cause trouble especially with
>git-based workflows.
> 
>  5. Consequently, declare that the recent MBF on this topic ought not
> to have been filed against 1.0 with diff packages, at least
> without some further filter.

I did the MBF mentioned in (5) (about suggesting the move from the 1.0
format to one of the 3.0 formats), and agreed to pause those efforts in
.

However, it might be worth clarifying if the MBF in (3) is mine, or
Guillem's one (with usertag dpkg-mismatch-source-vs-version-format and
user debian-d...@lists.debian.org) about "Mismatched source format vs
source version" (an example bug is #1007088). I think it's mine, but I'm
not sure. It might also be both.

Lucas

Bug#1007717: Native source package format with non-native version

[Felix Lechner]

Hi,

On Thu, Mar 17, 2022 at 11:57 AM Russ Allbery  wrote:
>
> source package format

While everyone is receptive to new labels, I prefer "upload format" or
"archive format". Either one helps us to distinguish the intermediate
product from any workflow objects a maintainer may have.

> single tarball as a source package format

It is also not necessary to "define" unpatched sources in the archive
by how many tarballs they have, or any tarballs at all.

In fact, I wrote a manifest utility that could one day replace tarball
signatures with per-file hashes. The latter remain useful even when
sources are repackaged, because manifests can be cryptographically
daisy-chained in a "chain of custody." Of course, they would be more
often used for patched upload formats.

Kind regards
Felix Lechner

Bug#1007717: Native source package format with non-native version

[Russ Allbery]

Sam Hartman  writes:
>> "Russ" == Russ Allbery  writes:

> Russ> Switching terminology to completely leave behind the terms
> Russ> with ambiguous meanings isn't a bad idea, but if so we really
> Russ> need a term that captures "is a packaging of an upstream
> Russ> software package with a separate existence" versus "exists
> Russ> solely as a Debian package."  "with-revision" or
> Russ> "without-revision" doesn't feel to me like it does this.
> Russ> Native and non-native do, which is why I was sticking with
> Russ> them, but maybe we can come up with some other equally-good
> Russ> terminology.

> Why do we need that distinction?

I think we need some way to talk about the various things that change
about a package if it is a packaging of an upstream package.  Examples
include a debian/watch file, an upstream metadata file, an upstream
signing key (still useful with single-tarball formats if upstream uses
signed tags, etc.), and provenance information in debian/copyright.

Those things are not tied to the source package format.  We want
provenance information for the upstream source even if you're using a
single tarball as a source package format, but we don't need it if there
is no upstream.

> Looking at current policy, 5.6.12 talks about having a debian revision
> or not having a debian revision.

It seems less useful to me to frame this solely in terms of the version
number format, rather than having the version number format indicate
whether this is a packaging of an upstream source with an independent
existence.  That would imply that if you just omit the debian revision,
you don't need to record the provenance of the upstream source, which
doesn't seem correct.

-- 
Russ Allbery (r...@debian.org)  

Bug#1007717: Native source package format with non-native version

[Sam Hartman]

> "Russ" == Russ Allbery  writes:

Russ> Switching terminology to completely leave behind the terms
Russ> with ambiguous meanings isn't a bad idea, but if so we really
Russ> need a term that captures "is a packaging of an upstream
Russ> software package with a separate existence" versus "exists
Russ> solely as a Debian package."  "with-revision" or
Russ> "without-revision" doesn't feel to me like it does this.
Russ> Native and non-native do, which is why I was sticking with
Russ> them, but maybe we can come up with some other equally-good
Russ> terminology.

Why do we need that distinction?

Looking at current policy, 5.6.12 talks about having a debian revision
or not having a debian revision.

Other parts of policy talk about  what parts of the source package there
are.

Why do we need more than these two distinctions.

I think that current policy has mostly left behind the work native
(although their are a few uses still).
My suspicion is that avoiding native allowed us to get a broader
consensus in the policy process.

Why isn't what we have good enough?

Bug#1007717: Native source package format with non-native version

[Russ Allbery]

Helmut Grohne  writes:

> Do you think it would be impossible to move forward on this matter in a
> consensus-based way?

I don't know.  I have some reasons to be dubious, but it's possible that
I'm being excessively pessimistic.

> Yes, please. Though as is evidenced in the replies to your mail, I would
> try to avoid "native" and "non-native" as much as possible given the
> existing confusion. I suggest using something like with-revision vs
> without-revision and single-tarball (from your mail) vs
> patches-separated to transport the concepts.

Switching terminology to completely leave behind the terms with ambiguous
meanings isn't a bad idea, but if so we really need a term that captures
"is a packaging of an upstream software package with a separate existence"
versus "exists solely as a Debian package."  "with-revision" or
"without-revision" doesn't feel to me like it does this.  Native and
non-native do, which is why I was sticking with them, but maybe we can
come up with some other equally-good terminology.

> More and more, it seems to me that we are looking into design work as
> opposed to picking an existing option.

*I* was doing design work, for sure.  But I'm not a member of the TC.  :)
The point was to offer you a design to consider as part of submitting the
request to the TC.

> In the spirit of consensus: Do you agree that retrying this in a
> consensus-based way is still possible?

If the relevant people required to implement a decision are willing to
tackle it that way, I am certainly willing to participate from the Policy
side.

-- 
Russ Allbery (r...@debian.org)  

Bug#1007717: Native source package format with non-native version

[Sam Hartman]

> "Helmut" == Helmut Grohne  writes:

Helmut> Hi Russ,
Helmut> On Tue, Mar 15, 2022 at 09:22:09PM -0700, Russ Allbery wrote:
>> > Specifically, I'd like to ask the TC to come up with policy on
>> native > packages and debian revisions using its power under
>> 6.1.1.
>> 
>> As a Policy Editor, I support this request.

Helmut> As a TC member I admit disliking this. While there is
Helmut> disagreement on how things are supposed to work, the views
Helmut> don't seem to be that far apart. Urgency is effectively
Helmut> removed by Lucas agreeing to pause further deprecation
Helmut> work. Do you think it would be impossible to move forward on
Helmut> this matter in a consensus-based way?

I think this is like usrmerge.
I actually think there is a rough consensus, but there are one or two
parties who find that consensus unacceptable and who are in a position
to block the consensus.

As an example, the text in policy 5.6.12 has been significantly revised
since Russ and I started discussing this issue in 2014.
It's possible that policy could be improved, but honestly I think that
policy is already consistent with the hypothesis I presented to the TC.

I also already tried to seek consensus where this is possible as DPL,
and included a pointer to my consensus call on the issues where there
was consensus in the material I already presented to the TC.

This is a case where people have slowly been building consensus since
2014.  One of the key stakeholders has not effectively participated
constructively, and so some things haven't been possible.

We're asking the TC to do the last mile work--review the existing
consensus-forming discussions, figure out where we are, and wrap it up
just like you did with usrmerge.

I've even tried to give you the key discussions that happened on
debian-devel.
I suspect one of the policy editors could go pull that together for
discussions on debian-policy, but if not, I'd be happy to do that work
if requested.

So, no, I do think this is ready for the TC to finish things up now.

--Sam


signature.asc
Description: PGP signature

Bug#1007717: Native source package format with non-native version

[Lucas Nussbaum]

On 16/03/22 at 23:54 +, Wookey wrote:
> On 2022-03-16 15:29 +, Ian Jackson wrote:
> > In practice, the vast majority of packages are maintained in git on
> > salsa.  The maintainers use those git repositories as the PFM.
> 
> > but almost everyone is already treating git as primary.
> 
> Is this definitely true? For example: I know I'm not doing this. I did
> try, and I do have some git repos on salsa, but I've mostly given up
> with it all and stuck with uscan and tarballs and quilt (and my trusty
> 'packages' directory). It's much easier for me. (The salsa repos that
> exist for my packages are not canonical and often stale).
> 
> I'm sure Ian is right that there is a trend towards git from tarballs
> and dscs, but I just question whether we know it is 'the vast
> majority'? Are there really now very few maintainers using the
> 'classic tooling'? How do we know?

Hi,

If you look at https://trends.debian.net/#version-control-system, I
think it's fair to say that the vast majority of packages are now
maintained in git.

And if you look at the status of those repositories according to
vswatch, most of them are actively used:

udd=> select status, count(distinct source) from vcswatch group by status order 
by 2;
 status  | count 
-+---
 UNREL   |   119
 OLD |  1218
 ERROR   |  1328
 COMMITS |  3620
 NEW |  7127
 OK  | 17873

(the last three lines being 'normal' states.)

Lucas

Bug#1007717: Native source package format with non-native version

[Helmut Grohne]

Hi Russ,

On Tue, Mar 15, 2022 at 09:22:09PM -0700, Russ Allbery wrote:
> > Specifically, I'd like to ask the TC to come up with policy on native
> > packages and debian revisions using its power under 6.1.1.
> 
> As a Policy Editor, I support this request.

As a TC member I admit disliking this. While there is disagreement on
how things are supposed to work, the views don't seem to be that far
apart. Urgency is effectively removed by Lucas agreeing to pause further
deprecation work. Do you think it would be impossible to move forward on
this matter in a consensus-based way?

> I've been involved in a lot of these discussions over the years, and the
> tentative conclusion that I've come to is that we have unfortunately mixed
> several different concepts in how we talk about source packages.  This has
> not *technically* reduced the flexibility of the packaging format because
> there are various workarounds, but it's *conceptually* reduced the
> flexibility in a way that makes those workarounds look conceptually
> incorrect.  As a result, we have constant repetitions of similar arguments
> stemming from well-meaning attempts to clean up the apparent
> inconsistencies (including some I have started).

While I did point out the conflation of matters in my initial reply to
Ian, I didn't put it as clearly as you did. Thank you. In the later
replies to your mail, we see that even trying to disentangle this is
difficult terminology-wise.

> There are really two separate concepts in play here:

I am wondering whether maybe one of you could work on a consensus
seeking process about these matters. I believe that resolving this using
consensus is far better than having 8 semi-random people decide upon a
policy. The consensus-based approach does not seem impossible to me at
this stage.

> What I would propose is to separate these concepts cleanly so that we can
> talk about them in a clearer way.  We should define native and non-native
> packages in terms of version numbers, and allow both native and non-native
> packages to use single-tarball source package formats.  We may want to
> steer people away from using the single-tarball source package format for
> the *normal* case of packaging upstream software, but I think it's well
> within the sorts of trade-off decisions that packagers already make and
> there are cases where a single tarball source format makes sense.

Yes, please. Though as is evidenced in the replies to your mail, I would
try to avoid "native" and "non-native" as much as possible given the
existing confusion. I suggest using something like with-revision vs
without-revision and single-tarball (from your mail) vs
patches-separated to transport the concepts. Of course "3.0 (native)" as
the content of debian/source/format cannot be avoided, but we can
describe its current and possibly desired properties in a less confusing
way.

> The name of the 3.0 (native) source package format is unfortunate in this
> context since it strongly implies that only native packages will use that
> source format.  The most formally correct thing to do would probably be to
> introduce a new 3.0 (single-tarball) source format and use 1.0 until then,
> but of course introducing a new source package format is not a fast
> process.  A more expedient thing to do may be to allow use of 3.0 (native)
> with non-native packages, since it's identical to 3.0 (single-tarball)
> except for the name, but it does leave a long-term confusing naming scheme
> in place.  Guillem may well have other and better suggestions from the
> dpkg perspective; I haven't thought all that much about a transition plan.

More and more, it seems to me that we are looking into design work as
opposed to picking an existing option. I think it is preferable to do
that with "normal" Debian processes without involving the TC, because
the latter always comes with constitutional powers attached (used or
unused). If history has taught us anything, involving the TC always
comes with escalation. We may still reach a point where it is clear that
consensus is not possible, but I don't think we're there yet.

In the spirit of consensus: Do you agree that retrying this in a
consensus-based way is still possible?

Helmut