Re: proposed MBF: packages still using source format 1.0

[Sean Whitton]

Hello,

On Tue 29 Mar 2022 at 08:50AM +02, Lucas Nussbaum wrote:

> On 28/03/22 at 16:03 -0700, Sean Whitton wrote:
>> Hello,
>>
>> On Tue 15 Mar 2022 at 06:26PM +01, Lucas Nussbaum wrote:
>>
>> > On 15/03/22 at 15:36 +, Ian Jackson wrote:
>> >> At least the following packages of which I am the maintainer or
>> >> sponsor were includined in the MBF, despite the fact that they are 1.0
>> >> native packages with Debian revision:
>> >>
>> >>its-playback-time
>> >>spigot
>> >>vm
>> >>vtwm
>> >>chroma
>> >>
>> >> Clearly the it makes no sense to have filed bugs saying "please switch
>> >> to this other source format" when the other source format cannot
>> >> represent the package.
>> >
>> > Those five packages:
>> > - are indeed native packages with Debian revisions
>> > - are not maintained in a VCS (or the VCS is not advertized using
>> >   Vcs-*).
>> >
>> > So there's no easy way to understand how the package differs from
>> > upstream (no patch serie, no VCS history). I don't think that it's
>> > something desirable.
>> > (if the packages had declared a VCS, they would have joined cachefilesd,
>> > userv-utils, and vde2 in the "native package with a Debian revision
>> > maintained in a VCS" category.)
>>
>> They have detailed history on dgit-repos.
>> E.g. .
>
> Yes, my point is that those packages don't have Vcs-* headers, so it's
> impossible to discover the above URL.

Right, sorry.

They should have such Vcs-* headers added.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: proposed MBF: packages still using source format 1.0

[Lucas Nussbaum]

On 28/03/22 at 16:03 -0700, Sean Whitton wrote:
> Hello,
> 
> On Tue 15 Mar 2022 at 06:26PM +01, Lucas Nussbaum wrote:
> 
> > On 15/03/22 at 15:36 +, Ian Jackson wrote:
> >> At least the following packages of which I am the maintainer or
> >> sponsor were includined in the MBF, despite the fact that they are 1.0
> >> native packages with Debian revision:
> >>
> >>its-playback-time
> >>spigot
> >>vm
> >>vtwm
> >>chroma
> >>
> >> Clearly the it makes no sense to have filed bugs saying "please switch
> >> to this other source format" when the other source format cannot
> >> represent the package.
> >
> > Those five packages:
> > - are indeed native packages with Debian revisions
> > - are not maintained in a VCS (or the VCS is not advertized using
> >   Vcs-*).
> >
> > So there's no easy way to understand how the package differs from
> > upstream (no patch serie, no VCS history). I don't think that it's
> > something desirable.
> > (if the packages had declared a VCS, they would have joined cachefilesd,
> > userv-utils, and vde2 in the "native package with a Debian revision
> > maintained in a VCS" category.)
> 
> They have detailed history on dgit-repos.
> E.g. .

Yes, my point is that those packages don't have Vcs-* headers, so it's
impossible to discover the above URL.

Lucas


signature.asc
Description: PGP signature

Re: proposed MBF: packages still using source format 1.0

[Sean Whitton]

Hello,

On Tue 15 Mar 2022 at 06:26PM +01, Lucas Nussbaum wrote:

> On 15/03/22 at 15:36 +, Ian Jackson wrote:
>> At least the following packages of which I am the maintainer or
>> sponsor were includined in the MBF, despite the fact that they are 1.0
>> native packages with Debian revision:
>>
>>its-playback-time
>>spigot
>>vm
>>vtwm
>>chroma
>>
>> Clearly the it makes no sense to have filed bugs saying "please switch
>> to this other source format" when the other source format cannot
>> represent the package.
>
> Those five packages:
> - are indeed native packages with Debian revisions
> - are not maintained in a VCS (or the VCS is not advertized using
>   Vcs-*).
>
> So there's no easy way to understand how the package differs from
> upstream (no patch serie, no VCS history). I don't think that it's
> something desirable.
> (if the packages had declared a VCS, they would have joined cachefilesd,
> userv-utils, and vde2 in the "native package with a Debian revision
> maintained in a VCS" category.)

They have detailed history on dgit-repos.
E.g. .

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: proposed MBF: packages still using source format 1.0

[Sean Whitton]

Hello,

On Tue 15 Mar 2022 at 11:46AM +01, Julien Cristau wrote:

> On Sun, Mar 13, 2022 at 02:58:31PM -0700, Sean Whitton wrote:
>> Hello,
>>
>> On Wed 09 Mar 2022 at 05:15PM +01, Lucas Nussbaum wrote:
>>
>> > On 09/03/22 at 08:52 -0700, Sean Whitton wrote:
>> >> On Wed 09 Mar 2022 at 01:08pm +01, Lucas Nussbaum wrote:
>> >> > Also, how would that work with packages that combine direct changes to
>> >> > upstream, and quilt for Debian-created patches?
>> >>
>> >> Could you expand?  I didn't think this category was one of the ones Russ
>> >> and I were talking about.
>> >
>> > My limited understanding of the landscape of git workflows is that a
>> > workflow that is quite popular among packages still using the 1.0 format
>> > is the one used by the Debian X strike force. Julien Cristau described
>> > it as follows when I asked about it on IRC:
>> >
>> > < jcristau> [...]  basically, for upstream patches we cherry-pick
>> > commits directly, and we use quilt for changes that aren't upstream
>>
>> Ah right, thank you.  I wasn't really thinking of this case as being
>> about git workflows.  Are the repos patches-applied or
>> patches-unapplied?
>>
> The patches that we keep in quilt are not applied in the repo.
> (Obviously the cherry-picked patches are.)

Thanks.  Is the rationale for this written down anywhere, may I ask?
I'd like to read about the workflow.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: proposed MBF: packages still using source format 1.0

[Matthew Vernon]

Bastian Blank  writes:

> On Wed, Mar 09, 2022 at 01:08:59PM +0100, Lucas Nussbaum wrote:
>> can we find a middleground where the git workflows don't require staying
>> with 1.0? Even if that means switching to 3.0 (quilt) using the
>> single-debian-patch approach?
>
> Well.  There is a specific source format now for full git workflows:
> 3.0 (gitarchive).
>
> It is implemented outside of dpkg in it's own package
> dpkg-source-gitarchive.

OOI, why is this not part of dpkg proper?

Matthew

-- 
"At least you know where you are with Microsoft."
"True. I just wish I'd brought a paddle."
http://www.debian.org

Re: proposed MBF: packages still using source format 1.0

[Tollef Fog Heen]

]] Matthew Vernon 

> Andrey Rahmatullin  writes:
> 
> > On Tue, Mar 15, 2022 at 08:54:50AM +, Matthew Vernon wrote:
> >> It's probably unfashionable, but I think debian/patches is not a great
> >> way to manage changes, particularly if you're using a VCS for
> >> maintaining your packages. As others have pointed out in this thread,
> >> doing this means you end up essentially trying to version-control your
> >> patches twice - once in the source package, and once in the VCS.
> > That's just a consequence of using two different storage formats for your
> > packages: a Debian source package and a VCS. As long as both of them are 
> > widely
> > used and incompatible, problems will exist in some form when using both.
> > By e.g. merging all patches in the Debian source package into one big diff
> > you are just breaking one of these two storage formats for that package,
> > essentially mandating the usage of the other one (the VCS) for most of the
> > developer operations with it.
> 
> I'm not sure that's entirely true; but even if it were, is that an
> entirely unreasonable position for a package maintainer (or team
> thereof) to take?

My problem with it is that we're then saying that we're not shipping the
source (as in, preferred form for modification).  This might be ok, but
then we should make sure that the «actual» source is available
elsewhere, which means it needs to be somewhere we manage, and treat
source packages as generated artifacts that can't be turned back into
the actual source.

-- 
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are

Re: proposed MBF: packages still using source format 1.0 [revised proposal]

[Thorsten Glaser]

Lucas Nussbaum dixit:

>column on https://udd.debian.org/cgi-bin/format10.cgi )

I’m apparently affected at least for cvs, but that package has
another very interesting use case for format 1.0:

Its .diff.gz file can *directly* be used as patch file in no less
than *two* other packaging systems (BSD ports and OpenSuSE build‐
service RPM), and I *do* use it there. It’s possible that other
downstream consumers exist (I was talking a bit with someone from
Gentoo but don’t recall whether anything came out of that).

So, no, the cvs package will not be switching to 3.0 formats, in
order to not break things for downstream users. (Similar cases
exist where compression formats for the .deb binaries are set to
specific values when they are reused; cvs, again, does that so
dowstreams can take the Tₑχ/LᴬTᴇΧ-generated texinfo PDFs from
that and skip the chore of porting texlive to the OS themselves.)

bye,
//mirabilos

PS: Can that CGI output a dd-list? I’m unsure I found all…
-- 
Yes, I hate users and I want them to suffer.
-- Marco d'Itri on gmane.linux.debian.devel.general

Re: proposed MBF: packages still using source format 1.0

[Stephen Kitt]

On Mon, 14 Mar 2022 13:52:14 +, Holger Levsen 
wrote:
> On Mon, Mar 14, 2022 at 01:10:19PM +, Wookey wrote:
> > > You're trying to produce packages from CI builds or other automation
> > > where you sometimes have native Debian revisions.
> > > 
> > > * you are producing a package where you have distinct upstream and
> > >   debian branches, and you cannot control  the upstream version number.
> > >  
> > 
> > Doesn't this make it 'not a native debian package'?  
> 
> yes, exactly, that's the problem.
> 
> > I thought the whole point of debian native packages was that there was
> > no 'upstream' and it was only for debian so you _are_ in control of
> > the source, the versioning and the releases?   
> 
> yes, that was the idea when native packages were introduced over
> 20 ago, when there were hardly any Debian forks, and certainly no
> CI systems and other automated systems which 'constantly fork'.
> 
> > As soon as that stops
> > being true then should one not shift to making it a standard
> > 'upstream+debian revision' non-native package?  
> 
> yes, we should convert all native packages in our archive,
> the idea of a native package has been obsoleted for long.

I think there are still some cases where it makes sense, e.g. for
“meta-source” packages where the upstream is really another package (e.g.
binutils-mingw-w64 and friends). Suffixes for downstreams work fine, see
https://launchpad.net/ubuntu/+source/gdb-mingw-w64 for one instance.

Regards,

Stephen


pgpf7WU3HcrNr.pgp
Description: OpenPGP digital signature

Re: proposed MBF: packages still using source format 1.0

[Ian Jackson]

Guillem Jover writes ("Re: proposed MBF: packages still using source format 
1.0"):
> Something I might want to see though (although I hold not much hope
> for) is a possible move away from the default behavior when no
> debian/source/format is present, as I think that gives bad defaults
> for newcomers or inexperienced users, and even there just emitting
> warnings tend to be ignored. Possible alternatives could be, either
> erroring out, or changing the default format depending on say a
> dpkg-compat level, or similar, I don't know, have not thought this
> through though. But explicitly marking sources as 1.0 (as has been
> warned for a long time now) would of course keep working as of right
> now.

Thanks for the reassurances.  (I have snipped much I didn't feel the
need to comment on but, it was all welcome.)

What you say above makes sense to me.  I'm not sure what an
appropriate timescale would be.

Would you welcome implementation of a "3.0 (diff)" format which
contained a tarball plus a single diff, arranged to be capable of
representing every git tree object ?  There would have to be some
massaging, I guess, mostly because of symlinks.  That would be
pareto-better than 1.0-with-diff.

Ian.

-- 
Ian JacksonThese opinions are my own.  

Pronouns: they/he.  If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.

Re: proposed MBF: packages still using source format 1.0

[Guillem Jover]

Hi!

On Tue, 2022-03-15 at 15:36:48 +, Ian Jackson wrote:
> However, given that I perceive that:
>   - there is a campaign to abolish 1.0
>   - there are important use cases where 1.0 is needed
>   - the campaign to abolish 1.0 is being prosecuted anyway
> I have deliberately chosen to continue to upload even pure-native
> packages as 1.0, to try to slow this process down in the hope that the
> situation improves and/or people stop trying to forbid useful things.

Ian, just to try to give some kind of reassurance. I'm not involved
with this MBF in any way (neither suggested nor prompted it, etc.).
While I'd like to see the practice of mismatched format vs version
disappear, I also neither did nor have immediate plans to make that
an error for 1.0 formats, as long as there are users at least in
Debian, or there are no viable alternatives, because I consider 1.0
confusing enough and not salvageable, that it makes less of a
difference there. If there ever comes the time where it's feasible to
error out on 1.0 on those conditions, I'd even be fine with a force
option but only for 1.0 formats, because I really want to preserve
2.0+ formats as clear as possible, and improve on those. I also have
no plans to abolish building source format 1.0 in dpkg.

Something I might want to see though (although I hold not much hope
for) is a possible move away from the default behavior when no
debian/source/format is present, as I think that gives bad defaults
for newcomers or inexperienced users, and even there just emitting
warnings tend to be ignored. Possible alternatives could be, either
erroring out, or changing the default format depending on say a
dpkg-compat level, or similar, I don't know, have not thought this
through though. But explicitly marking sources as 1.0 (as has been
warned for a long time now) would of course keep working as of right
now.

HTH,
Guillem

Re: proposed MBF: packages still using source format 1.0

[Lucas Nussbaum]

Hi,

On 15/03/22 at 09:29 -0700, Sean Whitton wrote:
> > What the are the packages for which you are surprised that bugs were
> > filed? I wonder which part of the criteria was too loose.
> 
> It looks like the query didn't do quite what was intended, indeed:
> src:userv-utils is maintained in git but a bug was filed.  Before I go
> ahead and close the bug, would you mind confirming this was an error
> rather than a disagreement about what counts as VCS-maintained?

Taking src:userv-utils:
- it is not maintained by Debian X
- it is maintained in a VCS
- the VCS is OK
- there are no direct changes in diff.gz, because there's no diff.gz,
  because the package is a native package. => the filter evaluates to
  true

So the limit of the query above is that it does not indeed account for
So the query works as intended, but indeed does not properly take into
account native packages, since direct_changes is always false.

There are 16 packages in that case:
- with a Debian revision:
  cachefilesd
  userv-utils
  vde2

- without a Debian revision:
  daptup
  dgit
  games-thumbnails
  gimp-plugin-registry
  gitpkg
  ifupdown-extra
  lpr
  postal
  svn-buildpackage
  uphpmvault
  vim-scripts
  whalebuilder
  xmorph

Indeed, it would have been better to look at whether those packages
include a Debian revision, to deal separately with those three special
cases.

Lucas

Re: proposed MBF: packages still using source format 1.0

[Lucas Nussbaum]

Hi,

On 15/03/22 at 15:36 +, Ian Jackson wrote:
> At least the following packages of which I am the maintainer or
> sponsor were includined in the MBF, despite the fact that they are 1.0
> native packages with Debian revision:
> 
>its-playback-time
>spigot
>vm
>vtwm
>chroma
> 
> Clearly the it makes no sense to have filed bugs saying "please switch
> to this other source format" when the other source format cannot
> represent the package.

Those five packages:
- are indeed native packages with Debian revisions
- are not maintained in a VCS (or the VCS is not advertized using
  Vcs-*).

So there's no easy way to understand how the package differs from
upstream (no patch serie, no VCS history). I don't think that it's
something desirable.
(if the packages had declared a VCS, they would have joined cachefilesd,
userv-utils, and vde2 in the "native package with a Debian revision
maintained in a VCS" category.)

Lucas

Re: proposed MBF: packages still using source format 1.0

[Sean Whitton]

Hello,

On Tue 15 Mar 2022 at 04:16pm +01, Lucas Nussbaum wrote:

> On 15/03/22 at 10:36 +, Ian Jackson wrote:
>> Answers were given, including by a former DPL (whom you may observe
>> is not someone I am on speaking terms with).
>>
>> But I see now that the MBF has gone ahead anyway.
>>
>> I spent some time trying to help by setting out the factual
>> background, but it seems that Debian is not interested in facts.  I
>> don't know why I bother.
>
> Hi Ian,
>
> As explained in https://lists.debian.org/debian-devel/2022/03/msg00165.html
> I proceeded with the MBF for packages that match
> not (debian_x or (vcs and vcs_status != 'ERROR' and direct_changes))
> or, maybe easier to read:
> (not debian_x) and ((not vcs) or vcs_status == 'ERROR' or (not 
> direct_changes))
>
> I did not file bugs for packages that are likely to use a VCS-based
> workflow (category (2) in the mail pointed above, or in
> https://udd.debian.org/cgi-bin/format10.cgi)
>
> What the are the packages for which you are surprised that bugs were
> filed? I wonder which part of the criteria was too loose.

It looks like the query didn't do quite what was intended, indeed:
src:userv-utils is maintained in git but a bug was filed.  Before I go
ahead and close the bug, would you mind confirming this was an error
rather than a disagreement about what counts as VCS-maintained?

Thanks.

-- 
Sean Whitton

Re: proposed MBF: packages still using source format 1.0

[Richard Laager]

On 3/15/22 10:36, Ian Jackson wrote:

Lucas Nussbaum writes ("Re: proposed MBF: packages still using source format 
1.0"):

As explained in https://lists.debian.org/debian-devel/2022/03/msg00165.html
I proceeded with the MBF for packages that match
not (debian_x or (vcs and vcs_status != 'ERROR' and direct_changes))
or, maybe easier to read:
(not debian_x) and ((not vcs) or vcs_status == 'ERROR' or (not direct_changes))

I did not file bugs for packages that are likely to use a VCS-based
workflow (category (2) in the mail pointed above, or in
https://udd.debian.org/cgi-bin/format10.cgi)


At least the following packages of which I am the maintainer or
sponsor were includined in the MBF, despite the fact that they are 1.0
native packages with Debian revision:

its-playback-time
spigot
vm
vtwm
chroma


I picked one of these, spigot, at random.


Clearly the it aakes no sense to have filed bugs saying "please switch
to this other source format" when the other source format cannot
represent the package.


I don't see any reason that 3.0 can't represent the spigot package. It 
looks like a straightforward package. It seems to have an upstream:


The top changelog entry says "Merge from upstream".

debian/control points me to:
http://www.chiark.greenend.org.uk/~sgtatham/spigot/

That site has a tarball available. The Debian package is NOT using that 
as its .orig.tar.gz. It doesn't have a .orig.tar.gz, presumably 
indicating it was built built as a native package (even though it has a 
Debian revision), which is the issue under discussion. To be honest, had 
I stumbled across this package outside of this conversation, I would 
have been confused by this.


My understanding is that Debian values the idea of using the pristine 
upstream tarball. Granted, sometimes that goal has to yield to higher 
priority things, like DFSG compliance.


How do you feel about the pristine tarball concept?

--
Richard


OpenPGP_signature
Description: OpenPGP digital signature

Re: proposed MBF: packages still using source format 1.0

[Ian Jackson]

Lucas Nussbaum writes ("Re: proposed MBF: packages still using source format 
1.0"):
> As explained in https://lists.debian.org/debian-devel/2022/03/msg00165.html
> I proceeded with the MBF for packages that match
> not (debian_x or (vcs and vcs_status != 'ERROR' and direct_changes))
> or, maybe easier to read:
> (not debian_x) and ((not vcs) or vcs_status == 'ERROR' or (not 
> direct_changes))
> 
> I did not file bugs for packages that are likely to use a VCS-based
> workflow (category (2) in the mail pointed above, or in
> https://udd.debian.org/cgi-bin/format10.cgi)

At least the following packages of which I am the maintainer or
sponsor were includined in the MBF, despite the fact that they are 1.0
native packages with Debian revision:

   its-playback-time
   spigot
   vm
   vtwm
   chroma

Clearly the it aakes no sense to have filed bugs saying "please switch
to this other source format" when the other source format cannot
represent the package.

Additionally, there were a further 9 packages where I am the
maintainer or uploader where the current version does not have a
Debian revision.  They could to be switched to "3.0 (native)" for the
better compression support.

However, given that I perceive that:
  - there is a campaign to abolish 1.0
  - there are important use cases where 1.0 is needed
  - the campaign to abolish 1.0 is being prosecuted anyway
I have deliberately chosen to continue to upload even pure-native
packages as 1.0, to try to slow this process down in the hope that the
situation improves and/or people stop trying to forbid useful things.

I maintain all my packages with dgit.  I presume that the vcs status
you are referring to is just that the Vcs-Git header is out of date
since the closure of alioth, or perhaps that there isn't one.

I hope thius clarifies.

Ian.

-- 
Ian JacksonThese opinions are my own.  

Pronouns: they/he.  If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.

Re: proposed MBF: packages still using source format 1.0

[Lucas Nussbaum]

On 15/03/22 at 10:36 +, Ian Jackson wrote:
> Answers were given, including by a former DPL (whom you may observe
> is not someone I am on speaking terms with).
> 
> But I see now that the MBF has gone ahead anyway.
> 
> I spent some time trying to help by setting out the factual
> background, but it seems that Debian is not interested in facts.  I
> don't know why I bother.

Hi Ian,

As explained in https://lists.debian.org/debian-devel/2022/03/msg00165.html
I proceeded with the MBF for packages that match
not (debian_x or (vcs and vcs_status != 'ERROR' and direct_changes))
or, maybe easier to read:
(not debian_x) and ((not vcs) or vcs_status == 'ERROR' or (not direct_changes))

I did not file bugs for packages that are likely to use a VCS-based
workflow (category (2) in the mail pointed above, or in
https://udd.debian.org/cgi-bin/format10.cgi)

What the are the packages for which you are surprised that bugs were
filed? I wonder which part of the criteria was too loose.

Also, feel free to close those bugs with a short explaining message.
I'll try to summarize the reasons for not migrating packages in a couple
of months.

Lucas

Re: proposed MBF: packages still using source format 1.0

[Adam Borowski]

On Tue, Mar 15, 2022 at 10:46:10AM +, Ian Jackson wrote:
> Ian Jackson writes ("Re: proposed MBF: packages still using source format 
> 1.0"):
> > But I see now that the MBF has gone ahead anyway.

> For example, consider a package maintained by a sponsee of mine:
> 
> Debian is not upstream, so it has a Debian revision.  The package is
> maintained in git, and the source package is very small and it is not
> uploaded frequently.  So we use a native source format.  This means
> that we must use format 1.0 because dpkg hates 3.0 native with debian
> revision.

So the package is really non-native; your beef here is with requiring a
tarball.  Your workaround is to [mis]use the native format.

But even legitimely native packages do want a Debian revision sometimes.
Eg. the natural versioning for valgrind-if-available would track
corresponding valgrind versions.  The 3.0 format restriction forbids
that.

So the bad thing is tying the internal format with version numbers.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁'Russkiy voyennyi korabl, idi nakhuy'
⢿⡄⠘⠷⠚⠋⠀
⠈⠳⣄

Re: proposed MBF: packages still using source format 1.0

[Felix Lechner]

Hi,

On Tue, Mar 15, 2022 at 3:46 AM Ian Jackson
 wrote:
>
> Debian is not upstream, so it has a Debian revision.  The package is
> maintained in git, and the source package is very small and it is not
> uploaded frequently.  So we use a native source format.  This means
> that we must use format 1.0 because dpkg hates 3.0 native with debian
> revision.

I do not understand the word "must" in that sentence, nor do I agree
with the word "hate".

I'll reiterate my call for a compromise: Let's deprecate source format
1.0 in exchange for relaxed version strings. It will streamline our
tooling and reduce the cognitive load on new maintainers.

> Yes. People complain about the Debian packaging toolchain being too
> complex or too confusing. This is one of such cases. As has been stated
> countless times, this subverts the semantics of both the source and
> version formats. At least we only have one case remaining, the even
> more senseless 1.0 non-native source with native version was turned
> into an error with dpkg 1.20.1. Recall that dpkg-source currently needs
> to use heuristics to decide whether to use an orig tarball + diff or not
> for format 1.0. :(

I'll buy both you, Ian and Guillem, a beer next time I see you. The
dispute has been going on for too long. Or, is it a battle over the
soul of Dpkg between the current maintainer and its inventor? For good
measure, Lucas is invited, too.

I think a negotiated peace is superior to yet another unhappy
interaction with the Technical Committee. Why do we have to put them
into such a difficult position every time?

Kind regards,
Felix Lechner

Re: proposed MBF: packages still using source format 1.0

[Bastian Blank]

On Wed, Mar 09, 2022 at 01:08:59PM +0100, Lucas Nussbaum wrote:
> can we find a middleground where the git workflows don't require staying
> with 1.0? Even if that means switching to 3.0 (quilt) using the
> single-debian-patch approach?

Well.  There is a specific source format now for full git workflows:
3.0 (gitarchive).

It is implemented outside of dpkg in it's own package
dpkg-source-gitarchive.

Bastian

-- 
Those who hate and fight must stop themselves -- otherwise it is not stopped.
-- Spock, "Day of the Dove", stardate unknown

Re: proposed MBF: packages still using source format 1.0

[Matthew Vernon]

Andrey Rahmatullin  writes:

> On Tue, Mar 15, 2022 at 10:49:17AM +, Matthew Vernon wrote:

>> but even if it were, is that an entirely unreasonable position for a
>> package maintainer (or team thereof) to take?
> Probably not? Just yet another case where you need to learn a specific
> workflow to modify or study a certain package, cannot use established
> documentation for that, and are required to use specific non-default tools
> for that.

In practice, for NMU purposes, one can essentially follow
dgit-nmu-simple(7) regardless of what workflow the package maintainer is
using (and that is one of its great benefits). In a large and diverse
project such as Debian there are always going to be multiple approaches
- and that's a strength in general, though there are costs and
trade-offs (such as the barrier for entry to new maintainers).

[but I fear we are drifting off topic]

Matthew

-- 
"At least you know where you are with Microsoft."
"True. I just wish I'd brought a paddle."
http://www.debian.org

Re: proposed MBF: packages still using source format 1.0

[Andrey Rahmatullin]

On Tue, Mar 15, 2022 at 10:49:17AM +, Matthew Vernon wrote:
> >> It's probably unfashionable, but I think debian/patches is not a great
> >> way to manage changes, particularly if you're using a VCS for
> >> maintaining your packages. As others have pointed out in this thread,
> >> doing this means you end up essentially trying to version-control your
> >> patches twice - once in the source package, and once in the VCS.
> > That's just a consequence of using two different storage formats for your
> > packages: a Debian source package and a VCS. As long as both of them are 
> > widely
> > used and incompatible, problems will exist in some form when using both.
> > By e.g. merging all patches in the Debian source package into one big diff
> > you are just breaking one of these two storage formats for that package,
> > essentially mandating the usage of the other one (the VCS) for most of the
> > developer operations with it.
> 
> I'm not sure that's entirely true; 
Which of my statements?

> but even if it were, is that an entirely unreasonable position for a
> package maintainer (or team thereof) to take?
Probably not? Just yet another case where you need to learn a specific
workflow to modify or study a certain package, cannot use established
documentation for that, and are required to use specific non-default tools
for that.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: proposed MBF: packages still using source format 1.0

[Julien Cristau]

On Sun, Mar 13, 2022 at 02:58:31PM -0700, Sean Whitton wrote:
> Hello,
> 
> On Wed 09 Mar 2022 at 05:15PM +01, Lucas Nussbaum wrote:
> 
> > On 09/03/22 at 08:52 -0700, Sean Whitton wrote:
> >> On Wed 09 Mar 2022 at 01:08pm +01, Lucas Nussbaum wrote:
> >> > Also, how would that work with packages that combine direct changes to
> >> > upstream, and quilt for Debian-created patches?
> >>
> >> Could you expand?  I didn't think this category was one of the ones Russ
> >> and I were talking about.
> >
> > My limited understanding of the landscape of git workflows is that a
> > workflow that is quite popular among packages still using the 1.0 format
> > is the one used by the Debian X strike force. Julien Cristau described
> > it as follows when I asked about it on IRC:
> >
> > < jcristau> [...]  basically, for upstream patches we cherry-pick
> > commits directly, and we use quilt for changes that aren't upstream
> 
> Ah right, thank you.  I wasn't really thinking of this case as being
> about git workflows.  Are the repos patches-applied or
> patches-unapplied?
> 
The patches that we keep in quilt are not applied in the repo.
(Obviously the cherry-picked patches are.)

Cheers,
Julien

Re: proposed MBF: packages still using source format 1.0

[Matthew Vernon]

Andrey Rahmatullin  writes:

> On Tue, Mar 15, 2022 at 08:54:50AM +, Matthew Vernon wrote:
>> It's probably unfashionable, but I think debian/patches is not a great
>> way to manage changes, particularly if you're using a VCS for
>> maintaining your packages. As others have pointed out in this thread,
>> doing this means you end up essentially trying to version-control your
>> patches twice - once in the source package, and once in the VCS.
> That's just a consequence of using two different storage formats for your
> packages: a Debian source package and a VCS. As long as both of them are 
> widely
> used and incompatible, problems will exist in some form when using both.
> By e.g. merging all patches in the Debian source package into one big diff
> you are just breaking one of these two storage formats for that package,
> essentially mandating the usage of the other one (the VCS) for most of the
> developer operations with it.

I'm not sure that's entirely true; but even if it were, is that an
entirely unreasonable position for a package maintainer (or team
thereof) to take?

Matthew

-- 
"At least you know where you are with Microsoft."
"True. I just wish I'd brought a paddle."
http://www.debian.org

Re: proposed MBF: packages still using source format 1.0

[Ian Jackson]

Ian Jackson writes ("Re: proposed MBF: packages still using source format 1.0"):
> But I see now that the MBF has gone ahead anyway.
> 
> I spent some time trying to help by setting out the factual
> background, but it seems that Debian is not interested in facts.  I
> don't know why I bother.

For example, consider a package maintained by a sponsee of mine:

Debian is not upstream, so it has a Debian revision.  The package is
maintained in git, and the source package is very small and it is not
uploaded frequently.  So we use a native source format.  This means
that we must use format 1.0 because dpkg hates 3.0 native with debian
revision.

My sponsee has responded to this but report by adding a
debian/source/format to their package and asking me to upload.  Of
course, I cannot do so.  I must instead ask my sponsee t revert the
change.

Ian.

-- 
Ian JacksonThese opinions are my own.  

Pronouns: they/he.  If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.

Re: proposed MBF: packages still using source format 1.0

[Ian Jackson]

Ian Jackson writes ("Re: proposed MBF: packages still using source format 1.0"):
> Sean Whitton writes ("Re: proposed MBF: packages still using source format 
> 1.0"):
> > [questions]
...
> 
> The situation here is complicated.
> 
> 
> The tl;dr is that
> 
>  * there are several situations where 1.0-native is the best answer,
>  * there are several situations where 1.0-with-diff is the best answer,
> 
> The root cause of both of these situations is that 3.0, sadly,
> is not always better in every respect than 1.0.

After I wrote that, there was a further excvhange where multiple
people replied to me "why are you doing that" - some in very
unpleasant tones of voice.

Answers were given, including by a former DPL (whom you may observe
is not someone I am on speaking terms with).

But I see now that the MBF has gone ahead anyway.

I spent some time trying to help by setting out the factual
background, but it seems that Debian is not interested in facts.  I
don't know why I bother.

Ian.

Re: proposed MBF: packages still using source format 1.0

[Andrey Rahmatullin]

On Tue, Mar 15, 2022 at 08:54:50AM +, Matthew Vernon wrote:
> It's probably unfashionable, but I think debian/patches is not a great
> way to manage changes, particularly if you're using a VCS for
> maintaining your packages. As others have pointed out in this thread,
> doing this means you end up essentially trying to version-control your
> patches twice - once in the source package, and once in the VCS.
That's just a consequence of using two different storage formats for your
packages: a Debian source package and a VCS. As long as both of them are widely
used and incompatible, problems will exist in some form when using both.
By e.g. merging all patches in the Debian source package into one big diff
you are just breaking one of these two storage formats for that package,
essentially mandating the usage of the other one (the VCS) for most of the
developer operations with it.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: proposed MBF: packages still using source format 1.0

[Matthew Vernon]

Hi,

Lucas Nussbaum  writes:

[bit late to this thread; came here when I got some MBF bugs and saw
"make them Severity: serious..." in the linked mail. I think in this
case use of source format 1.0 isn't against policy, _shouldn't_ be
against policy (or at least, not in all cases), and that de facto trying
to change policy by filing serious bugs isn't quite the done thing.]

> 1/ the arguments about using patches to track changes to upstream code.
> Among the ~600 packages in that potential MBF, there are still many that
> make changes to upstream code, which are not properly documented. I
> believe that it is widely accepted that seperate patches in 
> debian/patches/ are the recommended way to manage changes to upstream code 

It's probably unfashionable, but I think debian/patches is not a great
way to manage changes, particularly if you're using a VCS for
maintaining your packages. As others have pointed out in this thread,
doing this means you end up essentially trying to version-control your
patches twice - once in the source package, and once in the VCS.

For the avoidance of doubt, I'm not trying to tell anyone else how they
should manage their packages, but (particularly when I've helped
non-Debian folk needing to handle Debian packages) debian/patches is a
source of confusion in packaging.

Regards,

Matthew

-- 
"At least you know where you are with Microsoft."
"True. I just wish I'd brought a paddle."
http://www.debian.org

Re: proposed MBF: packages still using source format 1.0

[Felix Lechner]

Hi,

> I think we can all agree upon bumping the lintian severity to
> warning.

I am not sure there is unanimous support. Instead, I would like to
propose the following compromise (as I have before).

> 1.0 native is sometimes better than 3.0 (native) because dpkg-source
> refuses to build a 3.0 native package with a Debian revision in its
> version number.

> 1.0-with-diff has the following advantage over 3.0 (quilt):

> The extracted source tree does not contain a diff.  The inclusion of
> the diff *inside* the source tree (which happens with "3.0 (quilt)"
> whether or not single-debian-patch is specified) causes all manner of
> problems: it means that only certain states of the extracted tree are
> valid.

Ian and Guillem: How about we deprecate source format 1.0 in exchange
for relaxing the version strings? In the new source format, nativeness
is explicit [1] so the version acrobatics are no longer needed.

> yes, we should convert all native packages in our archive,
> the idea of a native package has been obsoleted for long.

As someone who co-maintains several Debian-internal tools, I am not
quite sure yet. Maybe let's take one step at a time? Thanks!

Kind regards
Felix Lechner

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=953554#25

Re: proposed MBF: packages still using source format 1.0

[Holger Levsen]

On Mon, Mar 14, 2022 at 01:10:19PM +, Wookey wrote:
> > You're trying to produce packages from CI builds or other automation
> > where you sometimes have native Debian revisions.
> > 
> > * you are producing a package where you have distinct upstream and
> >   debian branches, and you cannot control  the upstream version number.
> 
> Doesn't this make it 'not a native debian package'?

yes, exactly, that's the problem.

> I thought the whole point of debian native packages was that there was
> no 'upstream' and it was only for debian so you _are_ in control of
> the source, the versioning and the releases? 

yes, that was the idea when native packages were introduced over
20 ago, when there were hardly any Debian forks, and certainly no
CI systems and other automated systems which 'constantly fork'.

> As soon as that stops
> being true then should one not shift to making it a standard
> 'upstream+debian revision' non-native package?

yes, we should convert all native packages in our archive,
the idea of a native package has been obsoleted for long.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

Humans despise their genitals so much they often use them as metaphors for
humans they dislike.


signature.asc
Description: PGP signature

Re: proposed MBF: packages still using source format 1.0

[Wookey]

On 2022-03-10 12:09 -0700, Sam Hartman wrote:
> > "Steve" == Steve McIntyre  writes:
> 
> Steve> Why on earth *would* you mess around using Debian revisions
> Steve> on a native-format package, though?
> 
> You're trying to produce packages from CI builds or other automation
> where you sometimes have native Debian revisions.
> 
> * you are producing a package where you have distinct upstream and
>   debian branches, and you cannot control  the upstream version number.

Doesn't this make it 'not a native debian package'?

I thought the whole point of debian native packages was that there was
no 'upstream' and it was only for debian so you _are_ in control of
the source, the versioning and the releases? As soon as that stops
being true then should one not shift to making it a standard
'upstream+debian revision' non-native package?

Wookey
-- 
Principal hats:  Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: PGP signature

Re: proposed MBF: packages still using source format 1.0

[Sam Hartman]

> "Guillem" == Guillem Jover  writes:

Guillem> On Thu, 2022-03-10 at 12:09:14 -0700, Sam Hartman wrote:
>> You're trying to produce packages from CI builds or other
>> automation where you sometimes have native Debian revisions.
>> 
>> * you are producing a package where you have distinct upstream
>> and debian branches, and you cannot control the upstream version
>> number.  You want to do everything in git, and not have to deal
>> with quilt patches.  Say you don't even have any patches, but you
>> sometimes do produce new revisions simply for changes to debian
>> control files and the like.

Guillem> As I mentioned last time around, it has never been made
Guillem> clear why a different “revision”-separator character cannot
Guillem> be used here. (Perhaps out of doctrine? :)

Often that works.
It's kind of strange though to have to change revision characters
 depending on the releases.
For example I've been in situations
where  releases were built from tarballs and were not native, but
betas and other upstreams were built directly from git and thus were
native.
So some revisions had + as a revision character and some had dash.

But also, let's imagine it were doctrin.
Policy should decide that kind of doctrin not dpkg.

>> * You are trying to local (or downstream) builds of debian
>> packages that do have debian revision numbers.  You want to do
>> everything in git because honestly dealing with dscs is a PITA
>> and if you can avoid it you want to.  You need to produce dscs to
>> feed to sbuild, or mini-buildd or something.  But you just want
>> to do that easily from your git CI pipelines.
>> 
>> My frustrations with the number of hours I've lost because of
>> this doctrinal issue has caused me to come close to giving up on
>> Debian more than once.  Part of that is frustration around how
>> the change was handled and how concerns and use cases where not
>> considered and dismissed without consideration.  But part of that
>> is how I've had to hack around the isue in every downstream CI
>> environment where I took Debian packages and modified them.

Guillem> In this second scenario, you seem to be using .dsc in
Guillem> anger, when you don't really want to have to be using them
Guillem> at all. And when doing that you seem to have decided that
Guillem> using them in a way that makes our packaging stack more
Guillem> confusing, incoherent and error-prone is better, than say
Guillem> trying to get those other tools to avoid using the .dsc
Guillem> format at all.


And I'd be a lot less frustrated if the work to get these other tools
not to need dscs and been done *before* the workflow that everyone was
using at the time was broken by the dpkg maintainer.

Re: proposed MBF: packages still using source format 1.0

[Sean Whitton]

Hello,

On Wed 09 Mar 2022 at 05:15PM +01, Lucas Nussbaum wrote:

> On 09/03/22 at 08:52 -0700, Sean Whitton wrote:
>> On Wed 09 Mar 2022 at 01:08pm +01, Lucas Nussbaum wrote:
>> > Also, how would that work with packages that combine direct changes to
>> > upstream, and quilt for Debian-created patches?
>>
>> Could you expand?  I didn't think this category was one of the ones Russ
>> and I were talking about.
>
> My limited understanding of the landscape of git workflows is that a
> workflow that is quite popular among packages still using the 1.0 format
> is the one used by the Debian X strike force. Julien Cristau described
> it as follows when I asked about it on IRC:
>
> < jcristau> [...]  basically, for upstream patches we cherry-pick
> commits directly, and we use quilt for changes that aren't upstream

Ah right, thank you.  I wasn't really thinking of this case as being
about git workflows.  Are the repos patches-applied or
patches-unapplied?

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: proposed MBF: packages still using source format 1.0

[Sean Whitton]

Hello Ian,

Thank you for the summary, which helped refresh my memory.

On Wed 09 Mar 2022 at 04:38PM GMT, Ian Jackson wrote:

> 1. Why is 1.0-without-diff not always worse than 3.0 (native) ?
>
> 1.0 native is sometimes better than 3.0 (native) because dpkg-source
> refuses to build a 3.0 native package with a Debian revision in its
> version number.
>
> This prohibition exists solely because of a doctrinal objection to
> native-format packages with Debian revisions.  There is no technical
> reason why this restriction could not be lifted.  I sometimes upload
> this way and I have never had anyone report problems[1] with it.
>
> IMO there is nothing wrong with native format packages with Debian
> revisions.  They work just fine.  For a small paockage, this is often
> a good choice, because it avoids dealing with patches at all.
>
> For anything but a small package, use of diffs is needed as a storage
> and distribution optimisation.

It it perhaps worth noting that this idea that native source package
formats cannot have Debian revisions is an idea found in dpkg, not in
Policy, which latter otherwise has quite a bit to say about native vs.
non-native.

> Changes not representable by diff is what Sean is talking about here:
>
>> Ian has some cases where something that is representable in git is not
>> representable using 3.0 (quilt) but is representable using 1.0.  I don't
>> have those cases to hand; Ian, could you summarise, please?
>
> Currently, I think diff cannot represent changes to symlinks.
> git can store symlinks and represent their targets, and changes to
> their targets.

So in this case 1.0-native is the only option.  And it would be a
downgrade to mess around with what git represents perfectly well just
for the sake of an output format.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: proposed MBF: packages still using source format 1.0

[Holger Levsen]

Hi Lucas,

On Sun, Mar 06, 2022 at 09:25:45PM +0100, Lucas Nussbaum wrote:
> There are 629 packages in bookworm that use source format 1.0. That's 1.9% of
> bookworm packages.

many thanks for filing these bugs and even more thanks for filing them with
severity wishlist! I've just read one bug report of these, on a package I'm 
vaguely interested, and it felt really good to receive it: knowing/learning 
there 
are 1.9% of these packages and being prodded with wishlist, felt very 
reasonable,
a suggestion given with pat(c)hes to learn and grow, yay. Thanks, again. :) !


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

Only change is constant.


signature.asc
Description: PGP signature

Re: proposed MBF: packages still using source format 1.0

[Guillem Jover]

On Thu, 2022-03-10 at 12:09:14 -0700, Sam Hartman wrote:
> You're trying to produce packages from CI builds or other automation
> where you sometimes have native Debian revisions.
> 
> * you are producing a package where you have distinct upstream and
>   debian branches, and you cannot control  the upstream version number.
>   You want to do everything in git, and not have to deal with quilt
>   patches.
>   Say you don't even have any patches, but you sometimes do  produce new
>   revisions simply for changes to debian control files and the like.

As I mentioned last time around, it has never been made clear why a
different “revision”-separator character cannot be used here. (Perhaps
out of doctrine? :)

> * You are trying to local (or downstream) builds of debian packages that
>   do have debian revision numbers.
>   You want to do everything in git because honestly dealing with dscs is
>   a PITA and if you can avoid it you want to.
> You need to produce dscs to feed to sbuild, or mini-buildd or something.
> But you just want to do that easily from your git CI pipelines.
>
> My frustrations with the number of hours I've lost because of this
> doctrinal issue has caused me to come close to giving up on Debian more
> than once.
> Part of that is frustration around how the change was handled and how
> concerns and use cases where not considered and dismissed without
> consideration.
> But part of that is how I've had to hack around the isue in every
> downstream CI environment where I took Debian packages and modified
> them.

In this second scenario, you seem to be using .dsc in anger, when you
don't really want to have to be using them at all. And when doing that
you seem to have decided that using them in a way that makes our packaging
stack more confusing, incoherent and error-prone is better, than say
trying to get those other tools to avoid using the .dsc format at all.


The other thing, is that people keep confusing the transport source
format (.dsc, etc.), with the unpacked source. Of course it is not
helped that dpkg-source seems to be conflating these, but this is not
really the case. I've for example been pondering adding an extract
mode where instead of generating a quilt .pc/ compatible directory it
would instead generate a git repo. There's also the «3.0 (custom)»
format, which exemplifies this detachment.

But in any case, in your second scenario, if you are doing CI builds,
and only want to be working with git, you could simply do something
like this from your git checkout:

  $ dpkg-source --format="3.0 (git)" --build source-dir-4.0/

regardless of the declared source format in the unpacked tree
(including 1.0, 2.0 or any 3.0 format). For binary builds you could
do:

  $ dpkg-buildpackage --source-option=--format="3.0 (git)" -us -uc

Regards,
Guillem

Re: proposed MBF: packages still using source format 1.0

[Guillem Jover]

Hi!

[ But, this one again… ]

On Thu, 2022-03-10 at 18:17:15 +, Steve McIntyre wrote:
> Why on earth *would* you mess around using Debian revisions on a
> native-format package, though? IMHO it's pointless and is just going
> to confuse people. Unless you can explain a good reason to need this,
> I'd argue strongly that the 3.0 native support is DTRT for the
> principle of least surprise if nothing else!

Yes. People complain about the Debian packaging toolchain being too
complex or too confusing. This is one of such cases. As has been stated
countless times, this subverts the semantics of both the source and
version formats. At least we only have one case remaining, the even
more senseless 1.0 non-native source with native version was turned
into an error with dpkg 1.20.1. Recall that dpkg-source currently needs
to use heuristics to decide whether to use an orig tarball + diff or not
for format 1.0. :(

We currently have 21 source packages in the archive with format 1.0
native source and non-native version:

  $ grep-deb-sources -sPackage -FFormat '1.0' -a \
 -FVersion '-' -a --not -FFiles '.diff.'

My impression is that _most_ of those are packaging mistakes. I just
filed two new bugs on packages that have recently been (apparently)
accidentally uploaded with such broken source tarballs (vde2: #1007087;
etbemon: #1007088).

  

  

Then we are left with at most a literal (vocal) handful of maintainers
potentially doing this on purpose. And the main reason we cannot have
coherent, and understandable semantics here.

I've also asked before why this attachment to subvert the source
format, when the version could be modified to use some other character
instead for the “revision”, such as ‘+’, but I don't recall ever
getting any answer (not even a satisfactory one) to that.

Thanks,
Guillem

Re: proposed MBF: packages still using source format 1.0 [revised proposal]

[Adrian Bunk]

On Thu, Mar 10, 2022 at 09:49:50PM +0100, Lucas Nussbaum wrote:
>...
> For packages in (1.1) and (1.2), I propose to file Severity: wishlist
> bugs using the following template:
> 
> -->8
> Subject: please consider upgrading to 3.0 source format
> Severity: wishlist
> Usertags: format1.0
> 
> Dear maintainer,
> 
> This package is among the few (1.9%) that still use source format 1.0 in
> bookworm.  Please upgrade it to source format 3.0, as (1) this format has many
> advantages, as documented in https://wiki.debian.org/Projects/DebSrc3.0 ; (2)
> this contributes to standardization of packaging practices.
> 
> Please note that this is also a sign that the packaging of this software
> could maybe benefit from a refresh. It might be a good opportunity to
> look at other aspects as well.
> 
> This mass bug filing was discussed on debian-devel@:
> https://lists.debian.org/debian-devel/2022/03/msg00074.html
>...

josch already has tested patches for more than half of the packages, 
starting by submitting bugs for these packages with these patches will 
avoid work for maintainers and result in faster fixing of the bugs.

> Lucas

cu
Adrian

Re: proposed MBF: packages still using source format 1.0 [revised proposal]

[Lucas Nussbaum]

On 10/03/22 at 23:23 +0200, Adrian Bunk wrote:
> On Thu, Mar 10, 2022 at 09:49:50PM +0100, Lucas Nussbaum wrote:
> >...
> > For packages in (1.1) and (1.2), I propose to file Severity: wishlist
> > bugs using the following template:
> > 
> > -->8
> > Subject: please consider upgrading to 3.0 source format
> > Severity: wishlist
> > Usertags: format1.0
> > 
> > Dear maintainer,
> > 
> > This package is among the few (1.9%) that still use source format 1.0 in
> > bookworm.  Please upgrade it to source format 3.0, as (1) this format has 
> > many
> > advantages, as documented in https://wiki.debian.org/Projects/DebSrc3.0 ; 
> > (2)
> > this contributes to standardization of packaging practices.
> > 
> > Please note that this is also a sign that the packaging of this software
> > could maybe benefit from a refresh. It might be a good opportunity to
> > look at other aspects as well.
> > 
> > This mass bug filing was discussed on debian-devel@:
> > https://lists.debian.org/debian-devel/2022/03/msg00074.html
> >...
> 
> josch already has tested patches for more than half of the packages, 
> starting by submitting bugs for these packages with these patches will 
> avoid work for maintainers and result in faster fixing of the bugs.

I just sent a followup to the relevant bugs (it's the "trivial fix"
column on https://udd.debian.org/cgi-bin/format10.cgi )

Thanks

Lucas

Re: proposed MBF: packages still using source format 1.0 [revised proposal]

[Lucas Nussbaum]

On 10/03/22 at 21:49 +0100, Lucas Nussbaum wrote:
> https://udd.debian.org/cgi-bin/format10.cgi provides the list of
> packages for each category. The packages count is currently:
> (1.1): 53 packages
> (1.2): 424 packages
> (2): 149 packages

Actually it's:
(1.1): 60 packages
(1.2): 431 packages
(2): 135

(There was a logic error in the queries)

Lucas

Re: proposed MBF: packages still using source format 1.0 [revised proposal]

[Lucas Nussbaum]

Hi,

Based on the discussion, I propose the following:

Let's split the 626 packages in bookworm that use source format 1.0 into
three categories (1.1), (1.2), (2):
(1) packages with are very unlikely to use a VCS-based workflow (not
maintained by Debian X; not using a VCS; or referring to a broken VCS
repository; or using a VCS but not having any direct changes outside
patches)
  (1.1) Those in (1) that are key packages
  (1.2) Those in (1) that are not key packages
(2) packages which might be using a VCS-based workflow

https://udd.debian.org/cgi-bin/format10.cgi provides the list of
packages for each category. The packages count is currently:
(1.1): 53 packages
(1.2): 424 packages
(2): 149 packages

Packages in (2) need a deeper analysis to understand how VCS-based
workflows, or 3.0 (quilt), can be adapted to better support each other.
So let's not do anything about them for now, and focus on (1.1) and
(1.2).


For packages in (1.1) and (1.2), I propose to file Severity: wishlist
bugs using the following template:

-->8
Subject: please consider upgrading to 3.0 source format
Severity: wishlist
Usertags: format1.0

Dear maintainer,

This package is among the few (1.9%) that still use source format 1.0 in
bookworm.  Please upgrade it to source format 3.0, as (1) this format has many
advantages, as documented in https://wiki.debian.org/Projects/DebSrc3.0 ; (2)
this contributes to standardization of packaging practices.

Please note that this is also a sign that the packaging of this software
could maybe benefit from a refresh. It might be a good opportunity to
look at other aspects as well.

This mass bug filing was discussed on debian-devel@:
https://lists.debian.org/debian-devel/2022/03/msg00074.html

Thanks

Lucas
-->8

Then, I propose that we do not discuss upgrading the severity for bugs in
(1.2) before all packages in (1.1) are fixed (or there's a good reason
not to fix the remaining ones). That way,
1/ people motivated to do the work can do it using the normal NMU
procedure (and use the bugs for coordination).
2/ nobody is forced to do work packages until the packages that we
absolutely need to fix are fixed.

I will file bugs against the 53 packages in (1.1) soon as the number is
reasonably low, and I don't think this is controversial (with wishlist
severity). I will wait a few more days before filing the bugs for
packages in (1.2).

My main motivation for filing bugs against packages in (1.2) ASAP is
that I hope that filing the bugs will trigger some maintainers to fix
their packages, if they had not realized that their packages were still
using 1.0.

Lucas

Re: proposed MBF: packages still using source format 1.0

[Sam Hartman]

> "Steve" == Steve McIntyre  writes:

Steve> Ian Jackson wrote:
>> 
>> 1. Why is 1.0-without-diff not always worse than 3.0 (native) ?
>> 
>> 1.0 native is sometimes better than 3.0 (native) because
>> dpkg-source refuses to build a 3.0 native package with a Debian
>> revision in its version number.
>> 
>> This prohibition exists solely because of a doctrinal objection
>> to native-format packages with Debian revisions.  There is no
>> technical reason why this restriction could not be lifted.  I
>> sometimes upload this way and I have never had anyone report
>> problems[1] with it.
>> 
>> IMO there is nothing wrong with native format packages with
>> Debian revisions.  They work just fine.  For a small paockage,
>> this is often a good choice, because it avoids dealing with
>> patches at all.

Steve> Why on earth *would* you mess around using Debian revisions
Steve> on a native-format package, though?

You're trying to produce packages from CI builds or other automation
where you sometimes have native Debian revisions.

* you are producing a package where you have distinct upstream and
  debian branches, and you cannot control  the upstream version number.
  You want to do everything in git, and not have to deal with quilt
  patches.
  Say you don't even have any patches, but you sometimes do  produce new
  revisions simply for changes to debian control files and the like.

* You are trying to local (or downstream) builds of debian packages that
  do have debian revision numbers.
  You want to do everything in git because honestly dealing with dscs is
  a PITA and if you can avoid it you want to.
You need to produce dscs to feed to sbuild, or mini-buildd or something.
But you just want to do that easily from your git CI pipelines.

My frustrations with the number of hours I've lost because of this
doctrinal issue has caused me to come close to giving up on Debian more
than once.
Part of that is frustration around how the change was handled and how
concerns and use cases where not considered and dismissed without
consideration.
But part of that is how I've had to hack around the isue in every
downstream CI environment where I took Debian packages and modified
them.

Re: proposed MBF: packages still using source format 1.0

[Steve McIntyre]

Ian Jackson wrote:
>
>1. Why is 1.0-without-diff not always worse than 3.0 (native) ?
>
>1.0 native is sometimes better than 3.0 (native) because dpkg-source
>refuses to build a 3.0 native package with a Debian revision in its
>version number.
>
>This prohibition exists solely because of a doctrinal objection to
>native-format packages with Debian revisions.  There is no technical
>reason why this restriction could not be lifted.  I sometimes upload
>this way and I have never had anyone report problems[1] with it.
>
>IMO there is nothing wrong with native format packages with Debian
>revisions.  They work just fine.  For a small paockage, this is often
>a good choice, because it avoids dealing with patches at all.

Why on earth *would* you mess around using Debian revisions on a
native-format package, though? IMHO it's pointless and is just going
to confuse people. Unless you can explain a good reason to need this,
I'd argue strongly that the 3.0 native support is DTRT for the
principle of least surprise if nothing else!

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"We're the technical experts.  We were hired so that management could
 ignore our recommendations and tell us how to do our jobs."  -- Mike Andrews

Re: proposed MBF: packages still using source format 1.0

[Russ Allbery]

Richard Laager  writes:

> Could we only have "3.0 (quilt)" then, no "3.0 (native)"? Or, put
> differently, if you had a "native" package that is using a Debian
> revision and we allow that, what difference is left between "3.0
> (native)" and "3.0 (quilt)"?

3.0 (quilt) always has two tarballs, one for the upstream source and one
for the Debian packaging.  3.0 (native) has a single source tarball, which
I think is the desired goal here.

-- 
Russ Allbery (r...@debian.org)  

Re: proposed MBF: packages still using source format 1.0

[Richard Laager]

On 3/9/22 10:38, Ian Jackson wrote:

This prohibition exists solely because of a doctrinal objection to
native-format packages with Debian revisions.


As I understood it, the idea was that you could just increment the 
"actual" version number. I'm failing to see the advantage of 
incrementing "a" vs "z" in something like x.y.z-a.


There's not really a disadvantage either, if it all works (which you 
said it does). But the doctrinal issue, I think, is that definitionally 
a native package has no Debian revision (corollary: a package with a 
Debian revision is non-native).


If we revise that definition, to allow Debian revisions in native 
packages, then what distinction is left between native and non-native 
packages?


Could we only have "3.0 (quilt)" then, no "3.0 (native)"? Or, put 
differently, if you had a "native" package that is using a Debian 
revision and we allow that, what difference is left between "3.0 
(native)" and "3.0 (quilt)"?


To be clear: I genuinely don't know. I'm not implying that there isn't a 
remaining difference.



IMO there is nothing wrong with native format packages with Debian
revisions.  They work just fine.  For a small paockage, this is often
a good choice, because it avoids dealing with patches at all.


I don't follow the "avoid dealing with patches at all" piece here.

--
Richard


OpenPGP_signature
Description: OpenPGP digital signature

Re: proposed MBF: packages still using source format 1.0

[Ian Jackson]

Sean Whitton writes ("Re: proposed MBF: packages still using source format 
1.0"):
> On Wed 09 Mar 2022 at 01:08pm +01, Lucas Nussbaum wrote:
> > On 08/03/22 at 17:33 -0700, Sean Whitton wrote:
> >> Lucas, as I've had a lot to do with these git workflows and have
> >> probably done the most work documenting them, I can help with any
> >> specific follow-up questions you might have.
> >
> > Thanks!
> >
> > So the main question I think I have is:
> >
> > can we find a middleground where the git workflows don't require staying
> > with 1.0? Even if that means switching to 3.0 (quilt) using the
> > single-debian-patch approach?
> 
> dgit-maint-merge(7) works with single-debian-patch and that's what I
> use.  But it is not clear to me that there are any advantages at all to
> that over 3.0 (quilt) with single-debian-patch?

The situation here is complicated.


The tl;dr is that

 * there are several situations where 1.0-native is the best answer,
 * there are several situations where 1.0-with-diff is the best answer,

The root cause of both of these situations is that 3.0, sadly,
is not always better in every respect than 1.0.


1. Why is 1.0-without-diff not always worse than 3.0 (native) ?

1.0 native is sometimes better than 3.0 (native) because dpkg-source
refuses to build a 3.0 native package with a Debian revision in its
version number.

This prohibition exists solely because of a doctrinal objection to
native-format packages with Debian revisions.  There is no technical
reason why this restriction could not be lifted.  I sometimes upload
this way and I have never had anyone report problems[1] with it.

IMO there is nothing wrong with native format packages with Debian
revisions.  They work just fine.  For a small paockage, this is often
a good choice, because it avoids dealing with patches at all.

For anything but a small package, use of diffs is needed as a storage
and distribution optimisation.


2. Why is 1.0-with-diff not always worse than some 3.0 format ?

1.0-with-diff has the following advantage over 3.0 (quilt):

The extracted source tree does not contain a diff.  The inclusion of
the diff *inside* the source tree (which happens with "3.0 (quilt)"
whether or not single-debian-patch is specified) causes all manner of
problems: it means that only certain states of the extracted tree are
valid.

With 3.0 and single-debian-patch, modifying ordinary files can
generate a tree which doesn't round-trip through dpkg-source --build
and dpkg-source --extract: dpkg-source --build needs to "commit" the
changes to the diff, amending the working tree.

In other words, the working tree contains output files.  This causes
trouble if you want to represent the package in git.  This is why dgit
has all this quilt-fixup stuff.  With 1.0-with-diff, no quilt fixup is
needed.


3. Why is 1.0-native not just as good as 3.0 (native) ?

The only significant advantage of 3.0 (native) is that dpkg-source is
willing to create and extract 3.0 packages using newer compression
algorithms such as xz.

There were no good reason why the additional compression algorithms
were not supported in 1.0.  ("3.0 (native)" does offer some minor
metadata format advantages, so its existence is not entirely
pointless.)

If the restriction against native format with Debian revision were
lifted, "3.0 (native)" would be as capable as 1.0-without-diff, and
slightly superior.  So in that case, "3.0 (native)" should probably
replace all uses of 1.0-native.


4. What disadvantages does 1.0-with-diff suffor from, compared to "3.0
(quilt)" with single-debian-patch ?

The main disadvantage is that there are changes to the source tree
which are representable in "3.0 (quilt)" but which 1.0-with-diff does
not support.  (The precise details and the history are too compiex to
go into.)

I haven't checked recently, but this used to be enforced both on
building and on extraction.  So changing this is not so simple, since
we shouldn't start distributing packages in a new source format (or
variant) until the new extraction capability is very widely deployed.

But, ultimately, it would probably be a good idea.  Whether the to
call resulting thing "1.0" or "3.0 (diff)" doesn't really seem very
important.

The important properties are that it should support at least every
change that current diff(1) can represent.


5. Why is native sometimes superior to any quilt or diff format

We rely on diff(1) to represent changes to source trees and patch(1)
to apply them.  Not every change is representable by diff.

diff/patch does improve, of course.  That is not an unalloyed benefit,
though, because it means that when diff/patch improve we can
accidentally generate source packages that earlier versions can't
extract - a compatibility hazard.

Changes not representable by diff i

Re: proposed MBF: packages still using source format 1.0

[Lucas Nussbaum]

On 09/03/22 at 08:52 -0700, Sean Whitton wrote:
> On Wed 09 Mar 2022 at 01:08pm +01, Lucas Nussbaum wrote:
> > Also, how would that work with packages that combine direct changes to
> > upstream, and quilt for Debian-created patches?
> 
> Could you expand?  I didn't think this category was one of the ones Russ
> and I were talking about.

My limited understanding of the landscape of git workflows is that a
workflow that is quite popular among packages still using the 1.0 format
is the one used by the Debian X strike force. Julien Cristau described
it as follows when I asked about it on IRC:

< jcristau> [...]  basically, for upstream patches we cherry-pick
commits directly, and we use quilt for changes that aren't upstream

Lucas

Re: proposed MBF: packages still using source format 1.0

[Sean Whitton]

Hello,

On Wed 09 Mar 2022 at 01:08pm +01, Lucas Nussbaum wrote:

> On 08/03/22 at 17:33 -0700, Sean Whitton wrote:
>> Lucas, as I've had a lot to do with these git workflows and have
>> probably done the most work documenting them, I can help with any
>> specific follow-up questions you might have.
>
> Thanks!
>
> So the main question I think I have is:
>
> can we find a middleground where the git workflows don't require staying
> with 1.0? Even if that means switching to 3.0 (quilt) using the
> single-debian-patch approach?

dgit-maint-merge(7) works with single-debian-patch and that's what I
use.  But it is not clear to me that there are any advantages at all to
that over 3.0 (quilt) with single-debian-patch?  Anyway, the main case
where I myself use 1.0 is the following shell alias:

alias ...="sbuild --dpkg-source-opts='-Zgzip -z1 --format=1.0 -sn'"

This is from dgit-user(7).  It's the only reasonable way to say "just
sbuild HEAD, please".  One reason to continue to use 1.0 in the archive
is to ensure that a way to say "just sbuild HEAD, please" continues to
be supported, as it's very important to have.

Ian has some cases where something that is representable in git is not
representable using 3.0 (quilt) but is representable using 1.0.  I don't
have those cases to hand; Ian, could you summarise, please?

> Also, how would that work with packages that combine direct changes to
> upstream, and quilt for Debian-created patches?

Could you expand?  I didn't think this category was one of the ones Russ
and I were talking about.

-- 
Sean Whitton

Re: proposed MBF: packages still using source format 1.0

[Lucas Nussbaum]

On 08/03/22 at 17:33 -0700, Sean Whitton wrote:
> Lucas, as I've had a lot to do with these git workflows and have
> probably done the most work documenting them, I can help with any
> specific follow-up questions you might have.

Thanks!

So the main question I think I have is:

can we find a middleground where the git workflows don't require staying
with 1.0? Even if that means switching to 3.0 (quilt) using the
single-debian-patch approach?

Also, how would that work with packages that combine direct changes to
upstream, and quilt for Debian-created patches? Examples of such
packages are:

package| quilt patches
---+--
 xdm   | 8
 xorg-server   | 7
 xorg-server   | 7
 libx11| 6
 mesa  | 4
 xorg-docs | 3
 xserver-xorg-video-ivtvdev| 2
 xserver-xorg-input-synaptics  | 2

Lucas

Re: proposed MBF: packages still using source format 1.0

[Lucas Nussbaum]

On 08/03/22 at 17:10 +0100, Johannes Schauer Marin Rodrigues wrote:
> I did exactly that and rebuilt all the packages found by Lucas with the
> following changes:
> 
> $ mkdir -p debian/source
> $ echo '3.0 (quilt)' >debian/source/format
> 
> 141 source packages produce bit-by-bit reproducible binary packages after
> applying this change:
> 
> [..]
> 
> An additional 223 source packages produce bit-by-bit reproducible binary
> packages after applying this change:

Thanks a lot for this analysis.

I went a bit further and investigated a few packages that are not
in your list, just to provide examples of where issues could arise.

There are 266 packages not in your lists.

101 packages are native packages, and thus should probably be converted
to 3.0 (native). However some use a patch system (examples: mgetty,
libapache2-mod-auth-plain, vde2) or have a debian revision (example:
vanessa-socket).

Out of the 165 other packages, some fail to build (before the change),
such as gsfonts. Others are known to be unreproducible (example:
sofia-sip) and thus cannot be tested that way.

Finally, some result in different binary packages due to a GCC issue
that embeds the build path. If they are built in the same directory,
they result in the same binary packages after migration to 3.0.
(examples: mpclib3, libcompface, netselect, opus-tools)

Other that the known conflict with git workflows, I did not find any case
where migration would result in significant problems.

Lucas


signature.asc
Description: PGP signature

Re: proposed MBF: packages still using source format 1.0

[Sean Whitton]

Hello,

On Tue 08 Mar 2022 at 04:45pm +01, Lucas Nussbaum wrote:

> 1/ the arguments about using patches to track changes to upstream code.
> Among the ~600 packages in that potential MBF, there are still many that
> make changes to upstream code, which are not properly documented. I
> believe that it is widely accepted that seperate patches in
> debian/patches/ are the recommended way to manage changes to upstream code
> (good way to help with those changes getting reviewed, getting merged
> upstream, etc.)

... or a git workflow which has the changes represented as git commits,
rather than files under d/patches.

-- 
Sean Whitton

Re: proposed MBF: packages still using source format 1.0

[Sean Whitton]

Hello,

On Sun 06 Mar 2022 at 01:28pm -08, Russ Allbery wrote:

> If you're going to omit the ones in the last category, I think you should
> also omit the ones in the none/no/yes category, since they may be packages
> that intermittantly have changes and are similarly using a VCS-based
> workflow that doesn't want to use the 3.0 format.

Yes, indeed.

Lucas, as I've had a lot to do with these git workflows and have
probably done the most work documenting them, I can help with any
specific follow-up questions you might have.

> A mass bug filing for the first three categories seems like the change
> with the biggest potential to benefit Debian, since it's a direct
> simplification of the number of ways packages are maintained in the
> archive.  The packages without any patch system feel a bit less
> interesting.

Very much agreed.

-- 
Sean Whitton

Re: proposed MBF: packages still using source format 1.0

[Adrian Bunk]

On Tue, Mar 08, 2022 at 04:45:48PM +0100, Lucas Nussbaum wrote:
>...
> 1/ the arguments about using patches to track changes to upstream code.
> Among the ~600 packages in that potential MBF, there are still many that
> make changes to upstream code, which are not properly documented. I
> believe that it is widely accepted that seperate patches in 
> debian/patches/ are the recommended way to manage changes to upstream code 
> (good way to help with those changes getting reviewed, getting merged 
> upstream, etc.)

This is a reason *against* using RC bugs for forcing people to change it 
this year:

The sane way to minimize the regression risk when NMUing such an RC bug 
would be to dump the diff into a patch without touching it.

>...
> 3/ the arguments about standardization/simplication of packaging 
> practices, that make it easier (1) for contributors to contribute to any
> package (think security support, NMUers, but also derivatives);
>...

Actual work and actual breakage for the benefits of hypothetical 
contributors, that does not sound very convincing to me.

I am doing QA/NMU/*stable uploads for a three digit number of packages
I do not maintain every year, and this is not something I recall being 
a real problem.

> You argue that it's fine to wait 10 years for a transition such as the 
> switch to 3.0 (quilt). Actually, it has already been 11 years, since 
> 3.0 (quilt) was introduced around 2011
>...

Time before an issue is a lintian warning doesn't count.

If it isn't a problem for anyone and lintian does not warn,
how would anyone even notice?

> What we are talking about here is the "end game": there are less than 2%
> of packages in testing that are still using 1.0,
>...

There are lies, damn lies, and statistics.
600 (?) packages is a more realistic depiction than 2%.

If done carefully how many hours of work do you estimate it would take 
for all 600 packages, including ones that might for some reason be hard
to convert?

If you want to force work upon many people in the project, the burden of
proof is on you to show that the time is better spent on this than on
other bookworm work that could be done instead.

> Lucas

cu
Adrian

Re: proposed MBF: packages still using source format 1.0

[Adrian Bunk]

On Tue, Mar 08, 2022 at 05:10:44PM +0100, Johannes Schauer Marin Rodrigues 
wrote:
>...
> So now we have 364 source packages for which we have a patch and for which we
> can show that this patch does not change the build output. Do you agree that
> with those two properties, the advantages of the 3.0 (quilt) format are
> sufficient such that the change shall be implemented at least for those 364?

It sounds like a good idea to submit patches.

Some might be tagged wontfix, e.g. the Debian X Strike Force has a
workflow that would not work the same way with 3.0.

> Thanks!
> 
> cheers, josch

cu
Adrian

Re: proposed MBF: packages still using source format 1.0

[Andreas Tille]

Hi Adrian,

Am Tue, Mar 08, 2022 at 04:11:02PM +0200 schrieb Adrian Bunk:
> > 
> > I agree that there is no real urgency for immediate action - but this
> > seemed to be the case for other bugs on the packages I've touched the
> > case as well.
> 
> what time frame do you have in mind when you write "no real urgency"
> and "did not seem very probable"?

I try hard that all packages of the teams I'm involved strongly (Debian
Med, Debian Blends, R pkg team and may be Debian Science but with a
weaker focus from my side) get updated once per release cycle (just to
get them build with latest toolchain and look at least once whether the
watch file is reporting new versions correctly).  I'm aware that this
measure is not applied by all maintainers.
 
> For me a reasonable time frame for changes that are neither urgent nor
> supposed to create user-visible changes in the binary packages would be
> to ensure it is a lintian warning now, and then wait 10 years.

Given that one release cycle might be a bit dense in all cases I'd
consider two cycles (about 4 years) a sensible goal.  I would call
the term "real urgency" maximum half a year - thus I was choosing
"no real urgency".

> Many maintainers touch their packages at least once per release cycle 
> and also check lintian warnings, so many packages will get fixed within 
> the next 1-2 years.

So we seem to share the same measure.  I think the packages Lucas was
pointing to are in most cases not maintained by these "many maintainers"
(wild guessing from what I was looking at).
 
> Most packages will get a new maintainer or a new team member in an 
> existing maintainance team within the next 10 years, and with the
> help of a lintian warning this is the natural time for doing such
> changes.

I think we can all agree upon bumping the lintian severity to
warning.
 
Kind regards

   Andreas.

-- 
http://fam-tille.de

Re: proposed MBF: packages still using source format 1.0

[Johannes Schauer Marin Rodrigues]

Hi Adrian,

Quoting Adrian Bunk (2022-03-07 22:42:42)
> On Sun, Mar 06, 2022 at 09:25:45PM +0100, Lucas Nussbaum wrote:
> >...
> > I think that we should reduce the number of packages using the 1.0 format, 
> > as
> > (1) format 3.0 has many advantages, as documented in
> > https://wiki.debian.org/Projects/DebSrc3.0 ; (2) this contributes to
> > standardization of packaging practices, lowering the bar for contributors to
> > contribute to those packages.
> >...
> 
> You are not making a compelling case that these benefits clearly 
> outweight the substantial costs.
> 
> Such a MBF also:
> (1) causes a lot of extra work, and
> (2) causes a lot of breakage because such larger packaging changes
> are rarely done as careful as would be necessary
> 
> When people are making invasive packaging changes like a dh compat bump 
> or change the packaging due to such a MBF we often end up with bug 
> reports like #1000229 where something broke due to that (empty binary 
> packages are among the more typical breakages).
> 
> Unless a compelling case is made that the benefits of a MBF clearly 
> outweight these drawbacks, such MBFs usually have a negative benefit.

you are absolutely right! So what happens if we address both of these issues
by:

 (1) providing a patch and thus remove "a lot of extra work" being necessary
 (2) show that the package produces bit-by-bit identical binary packages after
 the change compared to before by  using reproducible builds and thus
 prevent "a lot of breakage"

I did exactly that and rebuilt all the packages found by Lucas with the
following changes:

$ mkdir -p debian/source
$ echo '3.0 (quilt)' >debian/source/format

141 source packages produce bit-by-bit reproducible binary packages after
applying this change:

 abootimg, acpi, amideco, asmix, aspell-el, avfs, aview, avrp, aylet,
 bfbtester, bindfs, cconv, cdde, cereal, citadel-client, cl-log, coco-cpp,
 cycfx2prog, daemontools, dbix-easy-perl, debaux, dictconv, ding-libs, dirdiff,
 dns323-firmware-tools, dv4l, famfamfam-flag, festlex-poslex,
 festvox-kallpc16k, festvox-kallpc8k, festvox-kdlpc16k, festvox-kdlpc8k,
 flvstreamer, fortunes-bofh-excuses, gav-themes, gcc-3.3, glbsp, glktermw,
 glslang, glurp, gnomediaicons, gtkterm, imaprowl, imgvtopgm, jdresolve,
 jtex-base, judy, kawari8, lakai, libalgorithm-dependency-perl,
 libanyevent-serialize-perl, libapache2-mod-log-slow, libaudio-scrobbler-perl,
 libbiblio-isis-perl, libcitadel, libclass-csv-perl, libclass-pluggable-perl,
 libcrypt-smbhash-perl, libdansguardian-perl, libdata-javascript-anon-perl,
 libdatapager-perl, libdata-validate-domain-perl, libdbix-dr-perl,
 libebook-tools-perl, libemail-foldertype-perl, libfile-searchpath-perl,
 libhtml-element-extended-perl, libhtml-popuptreeselect-perl,
 libimage-metadata-jpeg-perl, libjcode-pm-perl, libjlayer-java, libjmac-java,
 liblog-dispatch-filerotate-perl, libmodem-vgetty-perl, libmp4-info-perl,
 libnbcompat, libnet-finger-perl, libnet-proxy-perl, libropkg-perl,
 libtemplate-plugin-cycle-perl, libtemplate-plugin-utf8decode-perl,
 libtext-aligner-perl, libtext-table-perl, libtie-shadowhash-perl,
 libtimezonemap, libtree-multinode-perl, libunibreak, libuser-perl,
 libyaml-shell-perl, m16c-flash, manpages-tr, mapivi, midge, moblin-gtk-engine,
 mpclib3, ng-utils, nomarch, ocamlcreal, ocaml-getopt, ocaml-magic,
 ocaml-shout, osspsa, pandora-build, pcaputils, pdf2svg, pfqueue, phnxdeco,
 pidgin-awayonlock, pngmeta, pngnq, powerman, pscan, qprint, randtype, redet,
 ruby-bsearch, sbox-dtc, scriptaculous, sipcalc, speex, spirv-headers, src2tex,
 ssh-askpass-fullscreen, stx2any, sylseg-sk, tcsh, tor, ufiformat,
 uzbek-wordlist, vanessa-adt, vanessa-logger, watchdog, wayland, weston, xinit,
 xserver-xorg-video-nouveau, xserver-xorg-video-qxl, xsettings-kde,
 xtide-coastline, yaret, zmakebas

Then for the remaining packages we apply these changes and rebuild again:

$ mkdir -p debian/source
$ echo '3.0 (quilt)' >debian/source/format
$ echo single-debian-patch >debian/source/options

An additional 223 source packages produce bit-by-bit reproducible binary
packages after applying this change:

 4g8, aconnectgui, aft, alsamixergui, appconfig, ascdc, asmail, awardeco,
 baycomusb, beav, binstats, blop, catdvi, cbmplugs, cd-circleprint, chase,
 chrpath, ciphersaber, cl-regex, coco-cs, coco-java, code2html, codegroup,
 compartment, console-cyrillic, coolmail, cpipe, crack-attack, crypt++el, cvs,
 dbus-sharp-glib, debfoster, dictem, dist, dmitry, dnsmasq, docbook-website,
 dutch, dvdtape, edict-el, eldav, fake, fortunes-it, freebirth, freetable,
 freetds, funnelweb-doc, fuse-umfuse-ext2, fwanalog, g2p-sk, gkrellm-reminder,
 gkrellm-thinkbat, gkrellm-xkb, glw, gmemusage, gplaycli, gss-ntlmssp,
 gwaterfall, gworldclock, hsqldb1.8.0, ifrench-gut, imgsizer, impose+, inn,
 intel2gas, ipv6calc, its-playback-time, jargon, jgraph, kelbt, knews,
 libapache-gallery-perl, libdbd-sybase-perl,
 

Re: proposed MBF: packages still using source format 1.0

[Lucas Nussbaum]

On 08/03/22 at 16:11 +0200, Adrian Bunk wrote:
> On Tue, Mar 08, 2022 at 11:39:04AM +0100, Andreas Tille wrote:
> > Hi Adrian,
> 
> Hi Andreas,
> 
> > Am Mon, Mar 07, 2022 at 11:42:42PM +0200 schrieb Adrian Bunk:
> >...
> > > lintian already warns or has info tags that should be upgraded to warning,
> > 
> > I absolutely agree here.
> > 
> > > and then there will be slow migrations usually happening when someone
> > > anyway does (and tests!) larger packaging changes.
> > 
> > This "someone anyway does larger packaging changes" did not seem very
> > probable for the packages I've touched (see my other mail in this
> > thread).
> > 
> > > Ensuring that all relevant lintian tags are warnings would be the 
> > > appropriate action (which is not yet true[1]), but there is no urgency 
> > > on getting everything "fixed" immediately.
> > 
> > I agree that there is no real urgency for immediate action - but this
> > seemed to be the case for other bugs on the packages I've touched the
> > case as well.
> 
> what time frame do you have in mind when you write "no real urgency"
> and "did not seem very probable"?
> 
> For me a reasonable time frame for changes that are neither urgent nor
> supposed to create user-visible changes in the binary packages would be
> to ensure it is a lintian warning now, and then wait 10 years.
> 
> Many maintainers touch their packages at least once per release cycle 
> and also check lintian warnings, so many packages will get fixed within 
> the next 1-2 years.
> 
> Most packages will get a new maintainer or a new team member in an 
> existing maintainance team within the next 10 years, and with the
> help of a lintian warning this is the natural time for doing such
> changes.

Hi,

I believe that the arguments in favor of this MBF fall in three
categories:

1/ the arguments about using patches to track changes to upstream code.
Among the ~600 packages in that potential MBF, there are still many that
make changes to upstream code, which are not properly documented. I
believe that it is widely accepted that seperate patches in 
debian/patches/ are the recommended way to manage changes to upstream code 
(good way to help with those changes getting reviewed, getting merged 
upstream, etc.)

2/ the arguments about other advantages of the 3.0 (quilt) format: newer
compression algorithms, multi-tarball support, debian/ dir in upstream 
code, etc. (but I agree that those arguments don't work well here, 
because the packages in question don't seem to need those improvements)

3/ the arguments about standardization/simplication of packaging 
practices, that make it easier (1) for contributors to contribute to any
package (think security support, NMUers, but also derivatives); (2) to 
develop tools that process all packages.


You argue that it's fine to wait 10 years for a transition such as the
switch to 3.0 (quilt). Actually, it has already been 11 years, since 3.0
(quilt) was introduced around 2011 (see
https://trends.debian.net/#source-formats ).
What we are talking about here is the "end game": there are less than 2%
of packages in testing that are still using 1.0, and many of those look
abandonned or of relatively low interest (see
https://people.debian.org/~lucas/format1.0/packages.txt ).

I'm pushing for it now because I believe that it is a good timeframe to
finish that transition before the release of bookworm. Yes, it requires
some effort, and some maintainers might make mistakes and introduce bugs
in the process. However (1) if we start that work now, we still have
plenty of time to uncover issues before the bookworm freeze; (2) if the
maintainers use this opportunity to modernize the packaging, switch to
the best current practices, and for example add tests to their packages,
it also means less bugs in the future.

Lucas

Re: proposed MBF: packages still using source format 1.0

[Adrian Bunk]

On Tue, Mar 08, 2022 at 11:39:04AM +0100, Andreas Tille wrote:
> Hi Adrian,

Hi Andreas,

> Am Mon, Mar 07, 2022 at 11:42:42PM +0200 schrieb Adrian Bunk:
>...
> > lintian already warns or has info tags that should be upgraded to warning,
> 
> I absolutely agree here.
> 
> > and then there will be slow migrations usually happening when someone
> > anyway does (and tests!) larger packaging changes.
> 
> This "someone anyway does larger packaging changes" did not seem very
> probable for the packages I've touched (see my other mail in this
> thread).
> 
> > Ensuring that all relevant lintian tags are warnings would be the 
> > appropriate action (which is not yet true[1]), but there is no urgency 
> > on getting everything "fixed" immediately.
> 
> I agree that there is no real urgency for immediate action - but this
> seemed to be the case for other bugs on the packages I've touched the
> case as well.

what time frame do you have in mind when you write "no real urgency"
and "did not seem very probable"?

For me a reasonable time frame for changes that are neither urgent nor
supposed to create user-visible changes in the binary packages would be
to ensure it is a lintian warning now, and then wait 10 years.

Many maintainers touch their packages at least once per release cycle 
and also check lintian warnings, so many packages will get fixed within 
the next 1-2 years.

Most packages will get a new maintainer or a new team member in an 
existing maintainance team within the next 10 years, and with the
help of a lintian warning this is the natural time for doing such
changes.

> Kind regards
> 
>  Andreas.
>...

cu
Adrian

Re: proposed MBF: packages still using source format 1.0

[Andreas Tille]

Hi Adrian,

Am Mon, Mar 07, 2022 at 11:42:42PM +0200 schrieb Adrian Bunk:
> On Sun, Mar 06, 2022 at 09:25:45PM +0100, Lucas Nussbaum wrote:
> > I think that we should reduce the number of packages using the 1.0 format, 
> > as
> > (1) format 3.0 has many advantages, as documented in
> > https://wiki.debian.org/Projects/DebSrc3.0 ; (2) this contributes to
> > standardization of packaging practices, lowering the bar for contributors to
> > contribute to those packages.
> >...
> 
> You are not making a compelling case that these benefits clearly 
> outweight the substantial costs.

I inspected / fixed four packages where I might be slightly interested.
I agree that the benefits of only moving from source format 1.0 to 3.0
(quilt) might be limited.  However, those packages usually had other
issues (bugs, lagging behind upstream, missing watch files, etc.) which
were worth fixing.  So raising some awareness about packages that are
potentially not in good shape makes sense, IMHO.
 
> Such a MBF also:
> (1) causes a lot of extra work, and
> (2) causes a lot of breakage because such larger packaging changes
> are rarely done as careful as would be necessary
> 
> When people are making invasive packaging changes like a dh compat bump 
> or change the packaging due to such a MBF we often end up with bug 
> reports like #1000229 where something broke due to that (empty binary 
> packages are among the more typical breakages).

Running debdiff after switching to dh / bumping compat should definitely
be part of the procedure and should at least avoid this kind of bugs.
 
> Unless a compelling case is made that the benefits of a MBF clearly 
> outweight these drawbacks, such MBFs usually have a negative benefit.

Well, filing a bug is one thing.  Discussing inside the bug report why
changes are not wanted (may be even by tagging the bug wontfix) is
another thing.  Than we at least have some documentation about the
issue.  From my experience with the packages I've touched the only
reason for source format 1.0 was that these are not actively maintained.
Fixing the (not yet reported issue) was not a big deal (in contrast to
other changes I considered in the interest of our users).
 
> lintian already warns or has info tags that should be upgraded to warning,

I absolutely agree here.

> and then there will be slow migrations usually happening when someone
> anyway does (and tests!) larger packaging changes.

This "someone anyway does larger packaging changes" did not seem very
probable for the packages I've touched (see my other mail in this
thread).

> Ensuring that all relevant lintian tags are warnings would be the 
> appropriate action (which is not yet true[1]), but there is no urgency 
> on getting everything "fixed" immediately.

I agree that there is no real urgency for immediate action - but this
seemed to be the case for other bugs on the packages I've touched the
case as well.

Kind regards

 Andreas.

> [1] https://lintian.debian.org/tags/older-source-format

-- 
http://fam-tille.de

Re: proposed MBF: packages still using source format 1.0

[Andreas Tille]

Hi,

no need to file the suggested bug reports against

Am Sun, Mar 06, 2022 at 09:25:45PM +0100 schrieb Lucas Nussbaum:
>pngmeta

Fixed and adopted by Debian Phototools team.

>pngnq

Fixed and adopted by Debian Phototools team.

>libimage-metadata-jpeg-perl

Fixed and adopted by Debian Perl team.

>pixelize

Fixed and adopted by Debian Phototools team.

All smell removed from those packages (hopefully).

>src2tex

I consider to move this to Debian TeX team and fix issues
similarly.

Kind regards

 Andreas.

-- 
http://fam-tille.de

Re: proposed MBF: packages still using source format 1.0

[Adrian Bunk]

On Sun, Mar 06, 2022 at 09:25:45PM +0100, Lucas Nussbaum wrote:
>...
> I think that we should reduce the number of packages using the 1.0 format, as
> (1) format 3.0 has many advantages, as documented in
> https://wiki.debian.org/Projects/DebSrc3.0 ; (2) this contributes to
> standardization of packaging practices, lowering the bar for contributors to
> contribute to those packages.
>...

You are not making a compelling case that these benefits clearly 
outweight the substantial costs.

Such a MBF also:
(1) causes a lot of extra work, and
(2) causes a lot of breakage because such larger packaging changes
are rarely done as careful as would be necessary

When people are making invasive packaging changes like a dh compat bump 
or change the packaging due to such a MBF we often end up with bug 
reports like #1000229 where something broke due to that (empty binary 
packages are among the more typical breakages).

Unless a compelling case is made that the benefits of a MBF clearly 
outweight these drawbacks, such MBFs usually have a negative benefit.

lintian already warns or has info tags that should be upgraded to warning,
and then there will be slow migrations usually happening when someone
anyway does (and tests!) larger packaging changes.

Ensuring that all relevant lintian tags are warnings would be the 
appropriate action (which is not yet true[1]), but there is no urgency 
on getting everything "fixed" immediately.

cu
Adrian

[1] https://lintian.debian.org/tags/older-source-format

Re: proposed MBF: packages still using source format 1.0

[Adam Borowski]

On Mon, Mar 07, 2022 at 05:35:43PM +0200, Wouter Verhelst wrote:
> On Sun, Mar 06, 2022 at 09:25:45PM +0100, Lucas Nussbaum wrote:
> > Wouter Verhelst 
> >aspic
> >logtool
> 
> Yeah, no. These will be reduced to "wishlist" and probably tagged
> "wontfix".

Both of these packages have no patch system, so the whole work is:

mkdir -p debian/source
echo '3.0 (quilt)' >debian/source/format
echo single-debian-patch >debian/source/options

Less effort than typing an answer to the bug report.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁'Russkiy voyennyi korabl, idi nakhuy'
⢿⡄⠘⠷⠚⠋⠀
⠈⠳⣄

Re: proposed MBF: packages still using source format 1.0

[Jeremy Bicha]

On Mon, Mar 7, 2022 at 10:36 AM Wouter Verhelst  wrote:
> On Sun, Mar 06, 2022 at 09:25:45PM +0100, Lucas Nussbaum wrote:
> > Wouter Verhelst 
> >aspic
> >logtool
>
> Yeah, no. These will be reduced to "wishlist" and probably tagged
> "wontfix".
>
> The packages work just fine, the source format is still supported, I
> have better things to do with my time?

I guess you'd be ok with orphaning aspic to allow others to more
easily modernize the packaging?
https://bugs.debian.org/657083

Thank you,
Jeremy Bicha

Re: proposed MBF: packages still using source format 1.0

[Wouter Verhelst]

On Sun, Mar 06, 2022 at 09:25:45PM +0100, Lucas Nussbaum wrote:
> Wouter Verhelst 
>aspic
>logtool

Yeah, no. These will be reduced to "wishlist" and probably tagged
"wontfix".

The packages work just fine, the source format is still supported, I
have better things to do with my time?

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

Re: proposed MBF: packages still using source format 1.0

[Lucas Nussbaum]

Hi,

On 06/03/22 at 22:24 +, Holger Levsen wrote:
> So I'd rather propose to file these bugs with severity 'normal' and then wait
> and then get policy updated, and then raise the severity further.

For reference, there's a debian-policy bug about deprecating 1.0 +
dpatch/quilt: #850157 (no activity since 2018)

Lucas

Re: proposed MBF: packages still using source format 1.0

[Lucas Nussbaum]

On 06/03/22 at 22:25 +0100, Marco d'Itri wrote:
> On Mar 06, Lucas Nussbaum  wrote:
> 
> > I think that we should reduce the number of packages using the 1.0 format, 
> > as
> > (1) format 3.0 has many advantages, as documented in
> > https://wiki.debian.org/Projects/DebSrc3.0 ; (2) this contributes to
> > standardization of packaging practices, lowering the bar for contributors to
> > contribute to those packages.
> inn is a bit peculiar. It uses a patch system, has no direct changes and 
> is maintained in a VCS. But the build process is from a different age 
> and quite arcane, and I remember that switching to 3.0 would have 
> required significant work, so I see no compelling reason to do it.

So I looked into inn and it made me realize that there was a bug in my
analysis of the current status. We have packages using format 1.0 with
dpatch or quilt, but also with direct changes to files outside debian/
not tracked in the patch system.

So the correct breakdown is:
 patch_system | direct_changes | direct_changes_and_patch_system | vcs | count 
--++-+-+---
 dpatch   | N/A| no  | no  | 2
 dpatch   | N/A| yes | no  | 1
 quilt| N/A| no  | no  |17
 quilt| N/A| no  | yes |34
 quilt| N/A| yes | no  | 9
 quilt| N/A| yes | yes |62
 none | no | N/A | no  |   185
 none | no | N/A | yes |78
 none | yes| N/A | no  |   166
 none | yes| N/A | yes |74

I also updated https://people.debian.org/~lucas/format1.0/packages.txt

And inn is in quilt / N/A / yes / yes: there are files added in extra/
that are not tracked in a patch.

I tried to port inn to 3.0 (quilt), and after adding a patch for those
files using dpkg-source --commit, I could successfully build it. Do you
remember more details about the problems you ran into?

Lucas

Re: proposed MBF: packages still using source format 1.0

[Holger Levsen]

Hi Lucas,

thanks for doing this MBF!

I agree with the other two replies and have another thing to add:

On Sun, Mar 06, 2022 at 09:25:45PM +0100, Lucas Nussbaum wrote:
> I propose to file bugs using the following template, and make them Severity:
> serious after a month (minimum).
> 
> -->8
> Subject: upgrade to 3.0 source format
> Severity: important

I think those severities are too high and that they will cause unneeded friction
and frustration, also because (I think) they are not meeting the BTS criteria: 

serious:
is a severe violation of Debian policy (roughly, it violates a "must" or 
"required" directive), or, in the package maintainer's or release manager's
opinion, makes the package unsuitable for release.

important:
a bug which has a major effect on the usability of a package,
without rendering it completely unusable to everyone.


So I'd rather propose to file these bugs with severity 'normal' and then wait
and then get policy updated, and then raise the severity further.

To be frank: right now (or in a month) I'd just shake my head at severity
'serious', downgrade the bug and do something else. I totally agree it's
a smell (for some packages, see other replies) but IMO definitly not seriously
smelly :)

IOW: It doesn't stink, it's just a tiny bit awkward and less than ideal.
And that's neither important nor serious.

Finally and again: thank you for doing this work!


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

"Climate change" is an euphenism. "Global warming" as well.


signature.asc
Description: PGP signature

Re: proposed MBF: packages still using source format 1.0

[Russ Allbery]

Lucas Nussbaum  writes:

> The breakdown in terms of packages count is:

>  patch_system | direct_changes | vcs | count 
> --++-+---
>  dpatch   | no | no  | 3
>  quilt| no | no  |26
>  quilt| no | yes |96
>  none | no | no  |   185
>  none | no | yes |78
>  none | yes| no  |   166
>  none | yes| yes |74

> I propose to file bugs for packages in all categories above, except for
> packages in the last category that are maintained in an active VCS
> repository, because those are the most likely to be be using a git
> workflow that makes it hard to use the 3.0 format (even if I don't fully
> understand the arguments against using single-debian-patch in that
> case).

If you're going to omit the ones in the last category, I think you should
also omit the ones in the none/no/yes category, since they may be packages
that intermittantly have changes and are similarly using a VCS-based
workflow that doesn't want to use the 3.0 format.

A mass bug filing for the first three categories seems like the change
with the biggest potential to benefit Debian, since it's a direct
simplification of the number of ways packages are maintained in the
archive.  The packages without any patch system feel a bit less
interesting.

-- 
Russ Allbery (r...@debian.org)  

Re: proposed MBF: packages still using source format 1.0

[Marco d'Itri]

On Mar 06, Lucas Nussbaum  wrote:

> I think that we should reduce the number of packages using the 1.0 format, as
> (1) format 3.0 has many advantages, as documented in
> https://wiki.debian.org/Projects/DebSrc3.0 ; (2) this contributes to
> standardization of packaging practices, lowering the bar for contributors to
> contribute to those packages.
inn is a bit peculiar. It uses a patch system, has no direct changes and 
is maintained in a VCS. But the build process is from a different age 
and quite arcane, and I remember that switching to 3.0 would have 
required significant work, so I see no compelling reason to do it.

-- 
ciao,
Marco


signature.asc
Description: PGP signature