Re: Enabling some -Werror=format* by default?

[Guillem Jover]

Hi!

On Mon, 2024-06-10 at 18:01:58 +0200, Helmut Grohne wrote:
> Ideally, we'd not just do a rebuild with the flags, but also do a
> rebuild without and then compare the binary .debs. In the event that we
> misguide configure, we expect the .debs to differ and otherwise to equal
> due to the work of the reproducible builds folks. That equality has a
> really annoying difference in practice though: Build ids are dependent
> on the compiler flags, so the comparison would have to magically ignore
> changes in build id and this is where things become quite difficult.

A quick check seems to indicate warning flags do not get recorded in
the object files, which make sense as these are not supposed to change
the emitted objects.

(This can be seen with «dwarfdump .../*.debug | grep DW_AT_producer»
on the debug files from the -dbgsym packages.)

Thanks,
Guillem

Re: Enabling some -Werror=format* by default?

[Guillem Jover]

Hi!

On Mon, 2024-06-10 at 20:09:21 +0500, Andrey Rakhmatullin wrote:
> On Mon, Jun 10, 2024 at 04:24:54PM +0200, Guillem Jover wrote:
> > > Do you think it makes sense to add this a flag that enables -Werror=format
> > > to dpkg-buildflags(1), before, or after a test rebuild, before, or after
> > > the MBF if we do one?
> > 
> > If by adding you mean adding a new feature flag that is disabled by
> > default, then depending on the feature area, that might unfortunately
> > be equivalent to enable it by default (due to the «all» keyword).
> 
> Another related question: if not via dpkg-buildflags, how do we do
> rebuilds with changed default flags?

Ah, you can still use its mechanism (just not its policy) where a
rebuild should be able to use DEB__APPEND for that, or set
them in one of its config files.

If having a modified dpkg-dev emitting those by default would be less
cumbersome for the rebuilder, I'm happy to provide an out of archive
package for that for example, which I think we have done for some
rebuilds.

Thanks,
Guillem

Re: Enabling some -Werror=format* by default?

[Guillem Jover]

Hi!

On Mon, 2024-06-10 at 16:06:13 +0500, Andrey Rakhmatullin wrote:
> On Mon, Jun 10, 2024 at 07:48:59AM +0200, Helmut Grohne wrote:
> > > We recently increased the time_t size on certain architectures and some
> > > packages started failing to build because they were using a format
> > > specifier too narrow for the wider time_t, e.g. #1067829.
> > > But the only reason those package FTBFS is they enable either -Werror or
> > > some more specific non-default switch like -Werror=format=2, so I suspect
> > > much more packages contain similar code but gained only a warning. Isn't
> > > this a bad thing? Should we enable at least some combination of -Wformat*
> > > switches by default? Should we at least add a new flag to dpkg-buildflags
> > > and do some test rebuilds with it enabled?
> > 
> > It wasn't quite clear to me what -Werror=format=2 actually means.
> > According to the gcc documentation[1], -Wformat=2 currently means:
> > 
> > -Wformat -Wformat-nonliteral -Wformat-security -Wformat-y2k.

Those are in addition to the ones enabled by -Wformat=1, which are:

  -Wnonnull -Wno-format-contains-nul -Wno-format-extra-args
  -Wno-format-zero-length

So all the above would supposedly turn into -Werror too.

> But I don't know if even -Werror=format is too much to enable by default
> globally (I assume it's a good thing to enable it, though).

Hard to say w/o a rebuild I guess. (And take into account this can
also affect configure-style checks, which might become silent errors
but end up disabling features or enabling unexpected code paths or
similar.)

In general I think it's good (in principle) to enable -Werror flags
that detect actual bugs in code, the problem is always going to be
the amount of fallout and work that creates, and whether there's
consensus about that work commitment being acceptable to maintainers
in Debian, or whether that can interfere with transitions going on,
etc.

> > It also is unclear how this impacts the archive and yes, I'd recommend a
> > rebuild. Note though that we likely need this rebuild both on a 64bit
> > architecture and a 32bit architecture that is not i386 (due to how t64
> > works). A partial archive rebuild may work to gauge the size of the
> > problem.
> > 
> > I note that this kind of change affects cross builds, so performing
> > cross builds for armhf on amd64 will likely show many of these failures
> > (in addition to all the regular cross build failures).
> > 
> > I recommend doing more research before moving forward with this. In
> > particular a MBF about introduced problems would be prudent before doing
> > the switch and even if we don't switch, such a MBF bears value on its
> > own.

> Do you think it makes sense to add this a flag that enables -Werror=format
> to dpkg-buildflags(1), before, or after a test rebuild, before, or after
> the MBF if we do one?

If by adding you mean adding a new feature flag that is disabled by
default, then depending on the feature area, that might unfortunately
be equivalent to enable it by default (due to the «all» keyword).

(I started a design to version the build flags interface, but I got
stuck, so I let it brew as a background process, have pending to finish
that up and propose it on d-dpkg.)

Thanks,
Guillem

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Guillem Jover]

Hi!

On Thu, 2024-06-06 at 15:31:55 +0100, Simon McVittie wrote:
> On Thu, 06 Jun 2024 at 13:32:27 +0300, Hakan Bayındır wrote:
> > C, or C.UTF-8 is not a universal locale which works
> > for all.
> 
> Sure, and I don't think anyone is arguing that you or anyone else should
> set the locale for your interactive terminal session, your GUI desktop
> environment, or even your servers to C.UTF-8.
> 
> But, this thread is about build environments for our packages, not about
> runtime environments. We have two-and-a-half possible policies:

> 1. Status quo, in theory:
> 
>Packages cannot make any assumptions about build-time locales.
> 
>The benefits are:
> 
>- Diagnostic messages are in the maintainer's local language, and
>  potentially easier to understand.

I think this is way more important than the relative space used to
mention it though. :) I'm a non-native speaker, who has been involved
in l10n for a long time, while at the same time I've pretty much
always run my systems with either LANG=C.UTF-8 or before that LANG=C,
LC_CTYPE=ca_ES.UTF-8 and LC_COLLATE=ca_ES.UTF-8.

And I think forcing a locale on buildds makes perfect sense, because
we want easy access to build logs. But forcing LC_ALL from the build
tools implies that no tool invoked will get translated messages at
all, and means that users (not just maintainers) might have a harder
time understanding what's going on, we make lots of l10n work rather
pointless, and if no one is running with different locales then l10n
bugs might easily creep in.

>- If a mass-QA effort wants to assess whether the program is broken by
>  a particular locale, they can easily try running its build-time tests
>  in that locale, **if** the tests do not already force a different
>  locale. (But this comes with some serious limitations: it's likely
>  to have a significant number of false-positive situations where the
>  program is actually working perfectly but the **tests** make assumptions
>  that are not true in all locales, and as a result many upstream
>  projects set their build-time tests to force specific locales
>  anyway - often C, en_US.UTF-8 or C.UTF-8 - regardless of what we
>  might prefer in Debian.)

I consider locale sensitive misbehavior as a category of "upstream"
bugs (be that in the package upstream or the native Debian tools), that
deserve to be spotted and fixed. I can understand though the sentiment
of wanting to shrug this problem category off and wanting instead to
sweep it under the carpet, but that has accessibility consequences.

>   The costs are:

>   - […] but if I'm expected to diagnose the
> problem by reading Chinese error messages, as a non-Chinese-speaker I
> am not going to get far.)

Just as an aside, but while getting non-English messages makes for
harder to diagnose bugs, I've never found it a big deal to deal with
that kind of bug reports, as you can grep for (parts of) the
translated message, and then get the original English string from the
.po for example, or can translate the text back to know what it is
talking about, or ask the reported to translate it for you.

> 2½. Unwelcome compromise (increasingly the status quo):
> 
>Whenever a package is non-reproducible, fails to build or fails tests
>in certain locales (for example legacy non-UTF-8 locales like C or
>en_GB.ISO-8859-15), we add `export LC_ALL=C.UTF-8` to debian/rules and
>move on.
> 
>This is just (2.) with extra steps, and has the same benefit and cost
>for the affected packages as (2.) plus an additional cost (someone must
>identify that the package is in this category and copy/paste the extra
>line), and the same benefit and costs for unmodified packages as (1.).

I agree though, that if we end up with every debian/rules
unconditionally exporting LC_ALL, then there's not much point in not
making the build driver do it instead.


Related to this, dpkg-buildpackage 1.20.0 gained a --sanitize-env,
which for now on Debian and derivatives sets LC_COLLATE=C.UTF-8 and
umask=0022.

But _iff_ we end up with dpkg-buildpackage being declared the only
supported entry point, _and_ there is consensus that we'd want to set
some kind of locale variable from the build driver, then I guess this
could be done as a Debian vendor-specific thing, or via the
dpkg-build-api(7) interface.

Thanks,
Guillem

Re: MBF: drop dependencies on system-log-daemon

[Guillem Jover]

Hi!

On Tue, 2024-05-28 at 10:57:13 +0900, Simon Richter wrote:
> On 5/27/24 22:18, Simon McVittie wrote:
> > So I think your syslogd-is-journald could not be a Provides on the
> > existing systemd-sysv package, and would have to be a separate package.
> > I'm not sure that the benefit is worth it (and I see that Luca is sure
> > that the benefit *isn't* worth it).
> 
> I agree -- that's why I suggested changing the dependency to
> 
> "systemd-sysv | system-log-daemon"
> 
> This does not require an extra package, leaves out the system-log-daemon on
> most systems, still leaves the option of co-installing a flat logging daemon
> parallel to journald, and the packages work out-of-the-box on derived
> non-systemd distributions, so we don't waste developer time on maintaining a
> fork just for this issue.

I also care about portability and non-default alternatives, so I
assume for packages I maintain I'll be going instead with:

  " | system-log-daemon | systemd-sysv"

I don't think the original proposal is technically sound to represent
what is really going on with logging, but given its tone and how it
is being rushed (not even a day for discussion), it seems to me like
spending time thinking or proposing alternatives would be a waste of
time and energy.

Regards,
Guillem

Re: New supply-chain security tool: backseat-signed

[Guillem Jover]

Hi!

On Sat, 2024-04-06 at 19:13:22 +0800, Sean Whitton wrote:
> On Fri 05 Apr 2024 at 01:31am +03, Adrian Bunk wrote:
> > Right now the preferred form of source in Debian is an upstream-signed
> > release tarball, NOT anything from git.
> 
> The preferred form of modification is not simply up for proclamation.
> Our practices, which are focused around git, make it the case that
> salsa & dgit in some combination are the preferred form for modification
> for most packages.

People keep bringing this up, and it keeps making no sense. I've
covered this over the years in:

  https://lists.debian.org/debian-devel/2014/03/msg00330.html
  https://lists.debian.org/debian-project/2019/07/msg00180.html

(There's in addition the part that Adrian covers in another reply.)

Thanks,
Guillem

Re: Upstream dist tarball transparency (was Re: Validating tarballs against git repositories)

[Guillem Jover]

Hi!

On Wed, 2024-04-03 at 23:53:56 +0100, James Addison wrote:
> On Wed, 3 Apr 2024 19:36:33 +0200, Guillem wrote:
> > On Fri, 2024-03-29 at 23:29:01 -0700, Russ Allbery wrote:
> > > On 2024-03-29 22:41, Guillem Jover wrote:
> > > I think with my upstream hat on I'd rather ship a clear manifest (checked
> > > into Git) that tells distributions which files in the distribution tarball
> > > are build artifacts, and guarantee that if you delete all of those files,
> > > the remaining tree should be byte-for-byte identical with the
> > > corresponding signed Git tag.  (In other words, Guillem's suggestion.)
> > > Then I can continue to ship only one release artifact.
> >
> > I've been pondering about this and I think I might have come up with a
> > protocol that to me (!) seems safe, even against a malicious upstream. And
> > does not require two tarballs which as you say seems cumbersome, and makes
> > it harder to explain to users. But I'd like to run this through the list
> > in case I've missed something obvious.
> 
> Does this cater for situations where part of the preparation of a source
> tarball involves populating a directory with a list of filenames that
> correspond to hostnames known to the source preparer?
> 
> If that set of hostnames changes, then regardless of the same source
> VCS checkout being used, the resulting distribution source tarball could
> differ.

> Yes, it's a hypothetical example; but given time and attacker patience,
> someone is motivated to attempt any workaround.  In practice the
> difference could be a directory of hostnames or it could be a bitflag
> that is part of a macro that is only evaluated under various nested
> conditions.

I'm not sure whether I've perhaps misunderstood your scenario, but if
the distributed tarball contains things not present in the VCS, then
with this proposal those can then be easily removed, which means it
does not matter much if they differ between same tarball generation
(I mean it matters in the sense that it's an alarm sign, but it does
not matter in the sense that you can get at the same state as with a
clean VCS checkout).

The other part then is whether the remaining contents differ from what
is in the VCS.

If any of these trigger a difference, then that would require manual
review. That of course does not exempt one from reviewing the VCS, it
just potentially removes one avenue for smuggling artifacts.

> To take a leaf from the Reproducible Builds[1] project: to achieve a
> one-to-one mapping between a set of inputs and an output, you need to
> record all of the inputs; not only the source code, but also the build
> environment.
> 
> I'm not yet convinced that source-as-was-written to distributed-source-tarball
> is a problem that is any different to that of distributed-source-tarball to
> built-package.  Changes to tooling do, in reality, affect the output of
> build processes -- and that's usually good, because it allows for
> performance optimizations.  But it also necessitates the inclusion of the
> toolchain and environment to produce repeatable results.

In this case, the property you'd gain is that you do not need to trust
the system of the person preparing the distribution tarball, and can
then regenerate those outputs from (supposedly) good inputs from both
the distribution tarball, and _your_ (or the distribution) system
toolchain.

The distinction I see from the reproducible build effort, is that in
this case we can just discard some of the inputs and outputs and go
from original sources.

(Not sure whether that clarifies or I've talked past you now. :)

Thanks,
Guillem

Upstream dist tarball transparency (was Re: Validating tarballs against git repositories)

[Guillem Jover]

Hi!

On Fri, 2024-03-29 at 23:29:01 -0700, Russ Allbery wrote:
> On 2024-03-29 22:41, Guillem Jover wrote:
> > (For dpkg at least I'm pondering whether to play with switching to
> > doing something equivalent to «git archive» though, but see above, or
> > maybe generate two tarballs, a plain «git archive» and a portable one.)
> 
> Yeah, with my upstream hat on, I'm considering something similar, but I
> still believe I have users who want to compile from source on systems
> without current autotools, so I still need separate release tarballs.
> Having to generate multiple release artifacts (and document them, and
> explain to people which ones they want, etc.) is certainly doable, but I
> can't say that I'm all that thrilled about it.
> 
> I think with my upstream hat on I'd rather ship a clear manifest (checked
> into Git) that tells distributions which files in the distribution tarball
> are build artifacts, and guarantee that if you delete all of those files,
> the remaining tree should be byte-for-byte identical with the
> corresponding signed Git tag.  (In other words, Guillem's suggestion.)
> Then I can continue to ship only one release artifact.

I've been pondering about this and I think I might have come up with a
protocol that to me (!) seems safe, even against a malicious upstream. And
does not require two tarballs which as you say seems cumbersome, and makes
it harder to explain to users. But I'd like to run this through the list
in case I've missed something obvious.

I've implemented a prototype for dpkg, in the branch:

  
https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=next/dist-transparency

For context, for a long time dpkg dist tarballs have already shipped a
«.dist-version», I think some GNU projects started to do something
similar but with a different name.

The relevant commits:

  
https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/commit/?id=54a6ad9db3da335a40fed9020195864c4a87bdc1
(Add .dist-vcs-id, in git main already)

At least for dpkg, if «make dist» is run from outside a tag, then
the version will include the commit and whether the working dir
was dirty, but from a tag, only the version is included and there's
no link to what commit that was pointing to at that time. This file
adds that link, regardless of the current commit. And prints it as
part of the configure summary.

  
https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/commit/?id=1944a90d13c7c63592c438e550a212ab9e3aad76
(Remove VCS specific files from dist)

Simplifies the comparisons.

  
https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/commit/?id=39d181e60b3413c58a72056beec0a5a6f584cd92
(Add .dist-vcs-url)

This adds a new file to track the upstream VCS URL, so that it can
used from a deterministic place, for verification purposes.

  
https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/commit/?id=b3d7e0f195bd69b4622121e78fce751ea76dc0bc
(Add .dist-vcs-files)

This adds a new file with the list of files *in* the VCS, so that
we can get back to that clean state, even from a distributed
tarball, or from a extracted directory with built artifacts.

I also thought about listing the autogenerated files as Russ
mentions, but that seems error-prone and non-exhaustive, because
those might depend on the version of the autotools (or other build
system) used, and does not include artifacts part of the build phase,
which could be used to smuggle things in.

This last commit lists the three operations that all this makes
possible:

  * list difference in file lists (should be none)
  * list difference in file contents (should be none)
  * resetting the directory into a state like the VCS (except
for the VCS tracking/supporting files)

These operations are fairly generic, the one thing I could see
being "configurable" is the VCS files to exclude, maybe via
another file, but I've not thought about the consequences here.


I think this is safe (in the sense of detecting smuggled artifacts or
modifications in the dist tarball not present in the VCS, but certainly
not against modifications or artifacts smuggled in the VCS), because a
user that wants to verify any of this can make sure the URL is the
expected one, and everything else seems to fall from here, otherwise
you should get differences. (Thinking now, perhaps one of the checks
should be whether the expected tag or branch matches the commit id?)

This is currently catered for a Debian native package or just handling
the upstream part with no packaging, but I don't think it would be much
work to integrate this into packaged upstreams (mostly excluding whatever
is in the debian.tar parts?), or even to use something like this from
an upstream that does not provide these files by adding equivalent files
or metadata in the packaging.

The only things that one would need to trust

autoreconf --force not forcing (was Re: Validating tarballs against git repositories)

[Guillem Jover]

Hi!

On Sat, 2024-03-30 at 14:16:21 +0100, Guillem Jover wrote:
> Let's try to go in detail on how this was done on the build system
> side (I'm doing this right now, as previously only had skimmed over
> the process).
> 
> The build system hook was planted in the tarball by adding a modified
> m4/build-to-host.m4 file. This file is originally from gnulib (but
> gettext would usually embed it if it required it). The macros contained
> within are used by m4/gettext.m4 coming from gettext.
> 
> So to start with, this dependency (the AM_GNU_GETTEXT macro uses
> gl_BUILD_TO_HOST) is only present with newer gettext versions. The
> tarball was autoreconf'ed with gettext 0.22.4, Debian has gettext 0.21,
> which does not pull that dependency in. In that case if gettext.m4
> would get modified in this build now, then the hook would be inert,
> but once we update to a newer gettext then it would get activated
> again.
> 
> The m4/build-to-host.m4 file in addition to hooking the payload into
> the build system, also got its serial number bumped from 3 to 30.
> 
> And the bigger issue is that «autoreconf -f -i» does not even refresh
> the files (as you'd expect from the --force), if the .m4 serial is higher.
> So in Debian currently, the gettext.m4 in the tarball does not get
> refreshed (still pulling in the malicious build-to-host.m4, which
> would not happen with the gettext version from Debian), and if we
> updated to a newer gettext then it would not update build-to-host.m4
> anyway due to its bumped serial.
> 
> This seems like a serious bug in autoreconf, but I've not checked if
> this has been brought up upstream, and whether they consider it's
> working as intended. I expect the serial to be used only when not
> in --force mode though. :/

I filed a report to autoconf upstream at:

  https://lists.gnu.org/archive/html/bug-autoconf/2024-03/threads.html

the discussion now continues on the next month archive at:

  https://lists.gnu.org/archive/html/bug-autoconf/2024-04/msg3.html

We might have to perform a mass rebuild to check if there could be
fallout out of a true --force behavior change I guess.

Thanks,
Guillem

Re: Some t64 libraries already in testing; I'm confused

[Guillem Jover]

Hi!

On Sun, 2024-03-31 at 06:54:10 +0200, Andreas Metzler wrote:
> On 2024-03-30 Julian Gilbey  wrote:
> > My very limited understanding of this major transition was that the
> > t64 libraries are being held in unstable until (almost) everything is
> > ready, at which point there will be a coordinated migration into
> > testing.  But I've now been asked to upgrade something on my testing
> > machine which pulls in a t64 library.  Has something slipped through
> > the net, or is this intentional?
> 
> > Looking through testing, I see the following t64 libraries present:
> 
> > libaio1t64
> 
> I also stumbled over libaio1t64 and looked at the changelog. It is not
> part of the transition in that it was simply rebuilt with different
> compiler flags and therefore broke the ABI. Instead source code changes
> were made to extend the ABI to support usage both from code built with
> t64 flags and without.

Right, for libaio this is intentional. The changes were made to avoid
breaking the ABI (so as you say, implement dual ABI) because the
library uses direct syscalls so simply rebuilding with new flags would
have broken the ABI, although the actual SONAME change did then break
the ABI :), and it was done to avoid stomping over upstream interfaces.
Once I've looked into fixing Linux, I'll be submitting this upstream
with the hope that I can revert the SONAME change with backwards
compatibility symlink and package rename.


This also brings something I mentioned to Steve, but which I'm not
sure has been acted on. We need to check for all affected libraries
and whether they are using direct syscalls, because then if they do
not have explicit support for the equivalent 64-bit variants of those
syscalls, a simple rebuild of those libraries might actually have
broken the ABI between userland and kernel for those transitioned
packages. I had a list of the potentially affected syscalls, but need
to run out now, will try to post something better later today.

Thanks,
Guillem

Re: Validating tarballs against git repositories

[Guillem Jover]

Hi!

On Fri, 2024-03-29 at 23:53:20 -0600, Antonio Russo wrote:
> On 2024-03-29 22:41, Guillem Jover wrote:
> > On Fri, 2024-03-29 at 18:21:27 -0600, Antonio Russo wrote:
> >> Had tooling existed in Debian to automatically validate this faithful
> >> reproduction, we might not have been exposed to this issue.
> > 
> > Given that the autogenerated stuff is not present in the git tree,
> > a diff between tarball and git would always generate tons of delta,
> > so this would not have helped.
> 
> I may not have been clear, but I'm suggesting scrubbing all the
> autogenerated stuff, and comparing that against a similarly scrubbed
> git tag contents.  (But you explain that this is problematic.)

Yes, the point here is how we determine what is autogenerated stuff
when confronted with a malicious upstream, so the problem again is
that if you need to verify everything then you might easily get
overwhelmed by sheer amount of autogenerated output. But see below.

> >> Having done this myself, it has been my experience that many partial
> >> build artifacts are captured in source tarballs that are not otherwise
> >> maintained in the git repository.  For instance, in zfs (which I have
> >> contributed to in the past), many automake files are regenerated.
> >> (I do not believe that specific package is vulnerable to an attack
> >> on the autoconf/automake files, since the debian package calls the
> >> upstream tooling to regenerate those files.)
> 
> (Hopefully the above clears up that I at least have some superficial
> awareness of the build artifacts showing up in the release tarball!)

(Sorry, I guess my reply might have sounded patronizing? I noticed later
on that you explicitly mentioned this, but thought that would be clear
then when reading the whole mail, thought about adding a note to the
earlier text, but considered it unnecessary. Should have probably added
it anyway. :)

> >> 1. Move towards allowing, and then favoring, git-tags over source tarballs
> > 
> > I assume you mean git archives out of git tags? Otherwise how do you
> > go from git-tag to a source package in your mind?
> 
> I'm not wed to any specific mechanism, but I'd be content with that.  I'd
> be most happy DD-signed tags that were certified dfsg, policy compliant
> (i.e., lacking build artifacts), and equivalent to scrubbed upstream source.
> (and more on that later, building on what you say).
> 
> Many repositories today already do things close to this with pristine-tar,
> so this seems to me a direction where the tooling already exists.
> 
> I'll add that, if we drop the desire for a signed archive, and instead
> require a signed git-tag (from which we can generate a source tar on
> demand, as you suggest), we can drop the pristine-tar requirement.  If we
> are less progressive, but move to exclusively with Debian-regenerated
> .tar files, we can probably avoid many of the frustrating edge cases that
> pristine-tar still struggles with.

I'm personally not a fan of pristine-tar, and my impression is that it
is falling out of favor in various corners and big teams within the
project. And then I'm also not a fan either for mixing packaging with
upstream git history. The non-native packages I maintain only contain
debian/ directories, which to me have the superior properties (but not
tooling), including in a situation like this. I'll expand on this later.

I've been thinking and, perhaps the only thing we'd need, is to include
either a file or a field in some file that refers to the upstream commit
we think the tarball is derived from. We also have fields that contain
the upstream VCS repo. Then we could also have tooling that could perform
such checks, independently from how we transport and pack our sources.

> >> 2. Require upstream-built artifacts be removed (instead, generate these
> >>ab-initio during build)
> > 
> > The problem here is that the .m4 file to hook into the build system was
> > named like one shipped by gnulib (so less suspicious), but xz-utils does
> > not use gnulib, and thus the autotools machinery does not know anything
> > about it, so even the «autoreconf -f -i» done by debhelper via
> > dh-autoreconf, would not regenerate it.
> 
> The way I see it, there are two options in handling a buildable package:
> 
> 1. That file would have been considered a build artifact, consequently
> removed and then regenerated.  No backdoor.
>
> 2. The file would not have been scrubbed, and a difference between the
> git version and the released tar version would have been noticed.
> Backdoor found.
> 
> Either of these is, in my mind, dramatically better than what happened.

Sure, but that relies on knowing for certain what is and

Re: Validating tarballs against git repositories

[Guillem Jover]

Hi!

On Fri, 2024-03-29 at 18:21:27 -0600, Antonio Russo wrote:
> This is a vector I've been somewhat paranoid about myself, and I
> typically check the difference between git archive $TAG and the downloaded
> tar, whenever I package things.  Obviously a backdoor could have been
> inserted into the git repository directly, but there is a culture
> surrounding good hygiene in commits: they ought to be small, focused,
> and well described.

But the backdoor was in fact included in a git commit (it's hidden
inside a test compressed file).

The part that was only present in the tarball was the code to extract
and hook the inclusion of the backdoor via the build system.

> People are comfortable discussing and challenging
> a commit that looks fishy, even if that commit is by the main developer
> of a package.  I have been assuming tooling existed in package
> maintainers' toolkits to verify the faithful reproduction of the
> published git tag in the downloaded source tarball, beyond a signature
> check by the upstream developer.  Apparently, this is not universal.
> 
> Had tooling existed in Debian to automatically validate this faithful
> reproduction, we might not have been exposed to this issue.

Given that the autogenerated stuff is not present in the git tree,
a diff between tarball and git would always generate tons of delta,
so this would not have helped.

> Having done this myself, it has been my experience that many partial
> build artifacts are captured in source tarballs that are not otherwise
> maintained in the git repository.  For instance, in zfs (which I have
> contributed to in the past), many automake files are regenerated.
> (I do not believe that specific package is vulnerable to an attack
> on the autoconf/automake files, since the debian package calls the
> upstream tooling to regenerate those files.)
> 
> We already have a policy of not shipping upstream-built artifacts, so
> I am making a proposal that I believe simply takes that one step further:
> 
> 1. Move towards allowing, and then favoring, git-tags over source tarballs

I assume you mean git archives out of git tags? Otherwise how do you
go from git-tag to a source package in your mind?

> 2. Require upstream-built artifacts be removed (instead, generate these
>ab-initio during build)

The problem here is that the .m4 file to hook into the build system was
named like one shipped by gnulib (so less suspicious), but xz-utils does
not use gnulib, and thus the autotools machinery does not know anything
about it, so even the «autoreconf -f -i» done by debhelper via
dh-autoreconf, would not regenerate it.

Removing these might be cumbersome after the fact if upstream includes
for example their own maintained .m4 files. See dpkg's m4 dir for an
example of this (although there it's easy as all are namespaced but…).

Not using an upstream provided tarball, might also mean we stop being
able to use upstream signatures, which seems worse. The alternative
might be promoting for upstreams to just do the equivalent of
«git archive», but that might defeat the portability and dependency
reduction properties that were designed into the autotools build
system, or increase the bootstrap set (see for example the
pkg.dpkg.author-release build profile used by dpkg).

(For dpkg at least I'm pondering whether to play with switching to
doing something equivalent to «git archive» though, but see above, or
maybe generate two tarballs, a plain «git archive» and a portable one.)

> 3. Have tooling that automatically checks the sanitized sources against
>the development RCSs.

Perhaps we could have a declarative way to state all the autogenerated
artifacts included in a tarball that need to be cleaned up
automatically after unpack, in a similar way as how we have a way to
automatically exclude stuff when repackaging tarballs via uscan?

(.gitignore, if upstream properly maintains those might be a good
starting point, but that will tend to include more than necessary.)

> 4. Look unfavorably on upstreams without RCS.

Some upstreams have a VCS, but still do massive code drops, or include
autogenerated stuff in the VCS, or do not do atomic commits, or in
addition their commit message are of the style "fix stuff", "." or
alike. So while this is something we should encourage, it's not
sufficient. I think part of this might already be present in our
Upstream Guidelines in the wiki.

> In the present case, the triggering modification was in a modified .m4 file
> that injected a snippet into the configure script.  That modification
> could have been flagged using this kind of process.

I don't think this modification would have been spotted, because it
was not modifying a file it would usually get autogenerated by its
build system.

> While this would be a lot of work, I believe doing so would require a
> much larger amount of additional complexity in orchestrating attacks
> against Debian in the future.

It would certainly make it a bit harder, but I'm 

Bug#1067413: RFP: keydb -- persistent key-value database with network interface

[Guillem Jover]

Package: wnpp
Severity: wishlist
X-Debbugs-Cc: Chris Lamb , Sascha Steinbiss 

* Package name: keydb
  Version : 6.3.4
  Upstream Contact: https://github.com/Snapchat/KeyDB
* URL : https://keydb.dev/
* License : BSD-3-clause
  Programming Lang: C, C++
  Description : persistent key-value database with network interface

 keydb is a key-value database in a similar vein to memcache but the dataset
 is non-volatile. keydb additionally provides native support for atomically
 manipulating and querying data structures such as lists and sets.
 .
 The dataset is stored entirely in memory and periodically flushed to disk.



This project started as a fork from redis, for improved performance and
multi-threading support. We switched at work to it some time ago, and
had pending sending an RFP/ITP, but given the just announced license
change for redis to make it non-free [L], this seems more relevant now.
We've got the packaging bits around, which I'll try to send to this bug
report, but there's still some work that might be needed before an
upload, at least:

  - Unvendor more stuff (we have at least unvendored jemalloc and
rocksdb, but most of the rest need to go too).
  - Switch to use _keydb:_keydb as user and group (instead of
keydb:keydb).
  - Review and improve the maintscripts (as I think we initially based
our packaging on the upstream provided templates).

I'll try to do this during this week or next one, but if someone would
like to package this right ahead, I can speed this up.

I'm CCing Chris, who might perhaps be interested in replacing Redis with
KeyDB as its spiritual successor and taking this on? Or if not, at least
to perhaps potentially coordinate some kind of transition, even though
we've had issues migrating persistent DBs from newer Redis to KeyDB, so
that might be tricky or not feasible at all. I'm also CCing Sascha who
might be interested (given the keydb packaging repo in salsa). (I'm not
sure we are up for sole maintainership if no one takes this on, but
we'd be happy to give a hand in a team maintainership setting and that
might be an option for us if someone else is interested in driving this.)

[L] https://lwn.net/Articles/966133/

Thanks,
Guillem

Extended description of filesystem namespace clashes

[Guillem Jover]

Hi!

On Wed, 2024-02-28 at 07:41:50 +0100, Helmut Grohne wrote:
> That said, I appreciate your work on analyzing the situation as it also
> uncovers tangential problems e.g. where different packages put programs
> with different functionality into bin and sbin. It is up to
> interpretation of Debian policy whether that should be considered an
> RC-bug (10.1 "same filenames").

The Debian policy distinguishes between filename and pathname, and
other implementations make little sense TBH, but it could be stated
explicitly to make sure there's no room for misinterpretation.

In any case this seem like a more generic problem with conflicting
interfaces which would be nice to clarify. Some time ago I proposed
#562863, but it got eventually closed as inactive. Although I'd be
happy to work on that again if there is interest, by proposing some
wording.

> In general, I think that having each
> program name on either bin or sbin but not both is a desirable property
> and it should be easier to gain consensus on this.

For the same implementation I think this is fine, if it provides
compatibility or for a transition period. For different implementations
or worse for different interfaces, this would be just wrong, as each one
might shadow the other depending on the system or user PATH order.

Thanks,
Guillem

Re: Package marked for autoremoval due to closed bug? [and 1 more messages]

[Guillem Jover]

Hi!

On Tue, 2024-03-19 at 10:32:04 +, Ian Jackson wrote:
> [2] In my case src:dgit depends on git-buildpackage.  The autoremoval
> robot wants to remove git-buildpackage because of the time_t bugs
> against rpm, xdelta, and pristine-tar.  One root cause is that
> src:dpkg isn't migrating because of #1066952.

Just to clarify, I've now closed that report as I mentioned I'd be
doing, but this will have no effect, because dpkg and gcc-* are not
migrating primarily because they are being used by the release team
to gate the transition, to avoid having them get into testing and
potentially then getting things to build with the wrong ABI there

> The logic of the autoremoval system is that as an affected maintainer
> I ought to go and involve myself with #1066952.

Perhaps the release team blocks should be listed more prominently at
the beginning of the tracker page and perhaps be somewhat linkified
(I'll file a report)? (Or maybe you got this information from somewhere
else, which lacks that information or is also listed at the end?)

Regards,
Guillem

Re: Another take on package relationship substvars

[Guillem Jover]

Hi!

On Fri, 2024-02-23 at 17:59:14 -0800, Steve Langasek wrote:
> One generic case that this doesn't handle is Essential: yes packages.  For
> many of these, the ${shlibs:Depends} gets promoted in debian/control to
> Pre-Depends, not to Depends.

Ah! Good point.

I think the particular case of the Essential: yes and Pre-Depends
should really be handled by dpkg-shlibdeps. So I've now implemented a
new --package option for it, that will grab the package information
from debian/control, and if it is «Essential: yes» it will change the
default from Depends to Pre-Depends (in addition to also setting the
default Package-Type, so no need for explicit -t anymore), and the
package name will be added to -x. If -p is not passed, then it will
behave as now.

This, combined with the upcoming implicit substvar support should then
by default give the expected behavior (which can always be overridden).

Thanks,
Guillem

Re: time_t progress report

[Guillem Jover]

Hi!

On Mon, 2024-02-19 at 19:48:38 -0800, Steve Langasek wrote:
> I have coordinated with the gcc maintainer so that we can have the default
> flags in gcc-13 changed this week.
> 
> We are therefore targeting Friday for the mass NMUs to unstable though there
> is a possibility this won't start until as late as Monday depending on
> capacity.

Yesterday while trying to get a time for today's upload, I discussed
with Matthias, and it looks like today is not great for timing. There
are some things that might need to be hammered out in gcc, and Matthias
was not going to be available today until next week or so. I also just
realized the transition exceptions coverage does not match between
what some porters expect (f.ex. hurd-i386 which I'll discuss with them
later today), what dpkg and debhelper have implemented and what gcc
does, I'll discuss the latter with Matthias separately.

So as mentioned by Steve, this might need to be postponed a tiny bit
more.

Thanks,
Guillem

Re: Another take on package relationship substvars

[Guillem Jover]

Hi!

On Thu, 2024-02-22 at 23:14:13 +0100, gregor herrmann wrote:
> On Thu, 22 Feb 2024 19:32:21 +0100, Niels Thykier wrote:
> > If you forget to add a susbtvars that you should added, it is a latent RC
> > bug with only a warning from dpkg-gencontrol that you might miss if you grab
> > a coffee while waiting for the build to complete.  If you add one that is
> > not provided, you get a warning from dpkg-gencontrol that will nag you when
> > you *don't* go for coffee while waiting for the build.
> 
> I can relate a lot to this paragraph and would append: With or
> without the coffee, each time I see a warning from dpkg-gencontrol I
> have to think for some time about which of the two possible problems
> is meant (substvar defined in d/control but not generated, or
> substvar generated but missing in d/control), and that is really
> annoying.

If the warnings are not clear, I'm happy to further reword them to
try to make them so. I think they were previously even more confusing!
If you or anyone else has wording suggestions (which are ideally still
succinct) that would be great.

Thanks,
Guillem

Re: Another take on package relationship substvars

[Guillem Jover]

Hi!

On Thu, 2024-02-22 at 19:32:21 +0100, Niels Thykier wrote:
> Our current way of dealing with package relationship substvars such as
> ${misc:Depends} has been annoying me for a while. As it is, we are stuck in
> this way setup where the "Depends" field in debian/control is de facto
> mandatory. It is an RC bug (waiting to happen) if you omit ${misc:Depends}
> or, for arch:any packages, ${shlib:Depends} from the dependency field.
> 
> Personally, I feel we have this weird setup where we have tooling that will
> scream at you if you forget. But I lack a compelling reason for why we need
> tooling to remind us when we could just as well have tooling solve this
> problem for us.
>   Additionally, our tooling for reminding us rarely knows about anything but
> the 2-3 standard substvars everyone uses even though we should get reminders
> for other substvars like ${python3:Depends}, ${gir:Depends}, etc.
> Particularly the rules for when a substvars is applicable turns out to be
> quite non-trivial.
> 
> I have given this some thought and I think I have a conceptually idea for
> how to solve this, which I will go over in this email.

Right, this is annoying. This was actually brought up some time
ago (2010) in debian-devel as part of #597340. There was not much
reaction at the time (one good, a couple bad).

> # How do we get here?
> 
> High level, any helper framework above dpkg can provide this feature if it
> wants to. It is basically a question of parsing d/*.substvars to figure out
> which substvars are relevant, compute a new field from substvars file + the
> static provided data from d/control, and then passing -DField=<...> to
> dpkg-gencontrol.
> 
> There are probably a bunch of nitty gritty implementation details in
> practice, but the overall approach stands.
> 
> I think each helper stack (debhelper, cdbs, debputy, etc.) would define
> their own rule set for how to get there. As an example, I could see this be
> the new default in the next debhelper compat level, possibly with a way to
> be early adopter on existing compat levels.

If we agree we want this implicit mechanism of adding some substvars
to fields, then doing it in helpers seems like a suboptimal path, as
each one would need to replicate the logic, and implement substvar
substitution logic. I think if this is to be done, it should be done
at the dpkg-dev level.

Some time ago Dpkg::Substvars got support for optional substvars via
the ?= operator (prompted by you! :). I've started a WIP dpkg branch
and will try to finish implementing something for this tomorrow:

  https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=next/substvars%2b%2b

For now I've added a new (tentative) operator $=, that would declare
the variable is to be implicitly applied. Then the generator can
select whether to use old or new semantics.

> ## Alternative solutions
> 
> We could also make unused substvars a hard failure (FTBFS). Personally, I
> feel auto-managing them will be less painful for users. But if the consensus
> goes down this direction, then I would be behind it as it is still better
> than the status quo in my book.

I also implemented in that branch another operator !=, which marks a
substvar as required, so failure to use it will generate an error. I
might have to move the error emission into some other function that
is not called warn_about_unused() though… :)

> That is my proposal for how Debian contributors can spend less mental effort
> tracking relationship substvars and use their new spare mental capacity
> providing value for our users. Thanks for reviewing the proposal and any
> feedback you might have for making relationship substvars be less annoying
> for users and Debian as a whole.

Personally I think I agree with this being a problem and with a
potential solution for this based on implicit substvars. But I'm not
sure I agree at the layer this needs to be implemented. :) I think
going with explicit operators might be what was missing from the
earlier proposal and calm people's qualms about the too much magic
perception. And if there's no major push back on the overall idea
I'm happy to pursue this directly in dpkg-dev.

Thanks,
Guillem

Re: Bug#1064082: ITP: golang-github-cheggaaa-pb -- Console progress bar for Golang

[Guillem Jover]

Hi!

On Fri, 2024-02-16 at 15:07:55 -0800, Loren M. Lang wrote:
> Package: wnpp
> Severity: wishlist
> Owner: Loren M. Lang 

> * Package name: golang-github-cheggaaa-pb
>   Version : 3.1.5-1
>   Upstream Author : Sergey Cherepanov
> * URL : https://github.com/cheggaaa/pb
> * License : BSD-3-clause
>   Programming Lang: Go
>   Description : Console progress bar for Golang

> This is a dependency for pwru which is in RFP and I plan to complete packaging
> shortly. pwru is an eBPF-based Linux kernel networking debugger.

This is already packaged as:

  golang-github-cheggaaa-pb.v3-dev
  golang-gopkg-cheggaaa-pb.v2-dev
  golang-gopkg-cheggaaa-pb.v1-dev

Thanks,
Guillem

Changes to abi=+time64 behavior (was Re: 64-bit time_t transition in progress)

[Guillem Jover]

Hi!

On Fri, 2024-02-02 at 08:21:57 -0800, Steve Langasek wrote:
> Once all of these packages have built in experimental and we have identified
> and addressed all adverse interactions with the usrmerge transition, the
> plan is:
> 
>  - dpkg uploaded to unstable with abi=time64 enabled by default[5]

> [5] https://bugs.debian.org/1037136

I just realized recently that I think it'd be better to change a bit
the semantics of the abi time64 feature on i386, where by default it
will not include the time64 flags (as before), but if the maintainer
has requested it explicitly via DEB_BUILD_OPTIONS=abi=+time64, then
it should enable it also on i386 (changed behavior).

The reason is that this does not now break ABI for any package (in Debian
or out of Debian) that might have already opted-in to that feature, it
does not require adding all the necessary flags manually if one wants
to opt-in on i386, and makes it possible to selectively enable time64
on packages where the binary backwards compatibility make no sense
(such as dpkg itself, where this has already been requested and where
I mentioned in the libselinux report that I'd like to do that, and
where otherwise the port might be unable to install stuff at all).
Otherwise the majority of packages should have no need to explicitly
declare abi=+time64 as that's going to be the default (except for i386).

I've queued these updated semantics, including manual page updates and
unit tests into the next/time64 branch

(the previous semantics and incomplete commits are still in the
next/time64-default branch), which I'd be merging into main close to
the release, once there's agreement on the dpkg upload date.

If someone has issue with this, let me know and we can discuss the
merits of going some other way.

Thanks,
Guillem

Re: 64-bit time_t: updated archive analysis, proposed transition plan with timeline

[Guillem Jover]

Hi!

[ It seems the original post didn't get through to debian-devel (yet?),
  I found it on debian-release though
  https://lists.debian.org/debian-release/2024/01/msg00033.html,
  you might want to repost it here though, so that it can be commented
  on properly? :) ]

On Fri, 2024-01-05 at 00:23:00 -0800, Steve Langasek wrote:
> On Fri, Jan 05, 2024 at 12:17:04AM -0800, Steve Langasek wrote:
> > == Results ==
> > 
> > The overall findings of this analysis are 1,745 "dev" packages which either
> > are confirmed to have ABI changes or could not be checked; mapping to 2,154
> > runtime libraries (list attached) from 1,195 source packages (list attached)
> > and 5,477 reverse-dependencies requiring no-change rebuilds (list attached).
> > This is within the previously calculated range of "5300 to 5600", but there
> > are a number of newly-identified packages that fail to compile and have a
> > large number of reverse-dependencies.  I will continue to work to identify
> > false-positives here in the hopes of bringing this count down before pulling
> > the trigger on an actual transition.

> Guillem Jover 
>libaio
>libmd
>liburing

I checked these, and it looks like libmd and liburing are
false-positives?

 * libmd uses AC_SYS_LARGEFILE, and on 32-bit arches it is already built
   with LFS, the problem is that the header exposes off_t which means
   the code linking against it needs to match its build flags,
   otherwise they would already be broken now. You might want to look
   into this as a potential pattern for other false-positives probably.
   (I should probably update upstream the .pc file to include the
   -D_FILE_OFFSET_BITS=64 flags if enabled, but I don't think the
   analysis used .pc files anyway.)

 * liburing is marked as LFS-sensitive, but that comes from inline
   functions using off_t which end up casting that into an u32 type,
   so I don't think this should affect the ABI. It is also forcibly
   built with -D_FILE_OFFSET_BITS=64 in the upstream build system
   (and with future=+lfs for good measure in the packaging, which
   I'll switch to abi=+lfs).

Thanks,
Guillem

SOP migration (was Re: Reaction to potential PGP schism)

[Guillem Jover]

Hi!

Daniel thanks for all your work on the OpenPGP working group,
and on SOP! :)

On Wed, 2023-12-20 at 22:16:28 -0500, Daniel Kahn Gillmor wrote:
> # What Can Debian Do About This?
> 
> I've attempted to chart one possible path out of part of this situation
> by proposing a minimized, simplified interface to some common baseline
> OpenPGP semantics -- in particular, the "Stateless OpenPGP" interface,
> or "sop", as documented here:
> 
>https://datatracker.ietf.org/doc/draft-dkg-openpgp-stateless-cli/

> If your part of Debian's infrastructure depends on GnuPG, consider
> making it depend on a sop implementation instead, so we don't end up
> stuck on a single OpenPGP implementation in the future.  If the sop
> semantics are insufficient for your purposes, please report your needs
> at https://gitlab.com/dkg/openpgp-stateless-cli !

I think this is the way to go, and to try to support that goal I
started a wiki page to track what might need to be switched:

https://gitlab.com/dkg/openpgp-stateless-cli/-/wikis/Stateless-OpenPGP-status

I listed there some potential issues I could come up with for such
migrations. Also at the time, something that felt like a soft blocker
was that the schism was not widely known, so having to give that full
context first for every contacted project seemed a bit awkward, which
now should be out of the way, and a reference to some of the published
articles should be enough.

Time and energy permitting, I'd like to start at least filing issues
for these projects, and ideally provide patches. Help with any of that
would be highly appreciated! Including how to best integrate SOP into
a distribution (I'll be updating one of the tickets for a potentially
better «alternatives» usage pattern).

Also if a project uses perl, and using the Dpkg::OpenPGP modules would
make sense there, please reach out so that we can see what might be
missing so that they can be stabilized to make them public interfaces.

Thanks,
Guillem

Re: Signature strength of .dsc

[Guillem Jover]

Hi!

On Fri, 2023-12-01 at 00:20:16 +, Dimitri John Ledkov wrote:
> Currently dak requires signatures on .changes & .dsc uploads. .changes with
> signatures are publicly announced and then .dsc are published in the
> archive with signatures. .changes references .dsc.
>
> All .dsc have Checksums-Sha256 for the files they reference, .dsc itself
> can be verified through strong checksum in Sources metadata, chained via
> InRelease to the strong debian archive key signature.

Yes, the maintainer and uploader signatures (which can be different,
.dsc vs .changes) are most useful during the upload process, and are
part of the trust anchor handover. So that an archive can take over,
and then can implement certificate policies, such as certificate
rotation or handle expirations, revocations, etc. Performing such
actions on all .dsc would be not only infeasible as their checksums
would change, but I guess also rather costly.

> The same is not true for signatures on .dsc themselves. Majority of .dsc
> use at least sha256 and can be successfully verified.

Sure, and that's why for example dpkg-source by default does not
enforce --require-valid-signature by default.

> But some use weak hash:
> 5 dsc signed using Hash: RIPEMD160
> 152 dsc signed using Hash: SHA1

Although dpkg-source considers RIPEMD160 and SHA1 to be weak
algorithms, and when using its GnuPG OpenPGP backend will configure
it explicitly to that effect (given that by default it does not do
so), for other OpenPGP backends (such as SOP or Sequoia) this should
already be the tools builtin default.

> And many of them cannot be verified using debian-keyring:
> 2,455 no public key
> 3 wrong key usage

Did you use all keyrings like dpkg-source or dscverify do? For example
one of the first entries in the list, 7zip has a certificate in the
debian-maintainers.gpg, I assume many others do too (or the nonupload
one).

> Lists of affected .dsc are published at
> https://people.canonical.com/~xnox/dsc-analysis/ due to size.
> 
> This makes me wonder if signatures on uploaded or published .dsc have any
> value at all.
> Ultimately one should use apt secure to retrieve both .deb and .dsc; and
> verify .changes signature if one wants to figure out authorship.

Yes, for the users of an archive, the trust anchor is the archive. And
checking the signatures from the .changes file should in general give
no better results than the ones from the .dsc. But because the
.changes files are not generally published alongside in the archive
they are way more cumbersome to obtain (for example who-uploads needs
to scrape HTML…).

Having signatures in .dsc, even if some might end up expiring, their
certificates revoked, or removed from the keyrings, it seems better
than not having them at all.

> Should we upload sourceful NMU to eliminate SHA1, RIPEMD160,
> wrong-key-usage signatures in .dsc?

In the end in most cases this will probably just be an indicator that
these packages have not been uploaded in a very long time. Having
lintian classify them or error or warn would seem appropriate, I could
even see a release criteria where source packages that are to be part
of a supported release need to have a valid and strong signature, for
example. But just uploading to refresh the sigs right now (w/o such
criteria) seems a bit pointless, perhaps they should be uploaded as
part of a process to get them orphaned instead?

> Should we stop requiring signed .dsc on uploads?

I think they do have value. You can provides signed .dsc to someone
else before an upload (even people that have no certificates in the
archive, knowing they cannot upload these), and then the same files
will appear on the archive. It is an easy way to check for provenance,
and to link the uploader to the archive, even on mirrors, in the same
way as upstreams provide signatures for source tarballs, for example,
which can also suffer from the same fate. I don't see a strong reason
why we'd want to drop them TBH.

Thanks,
Guillem

Re: New Essential package procps-base

[Guillem Jover]

Hi!

On Tue, 2023-11-14 at 17:29:01 +1100, Craig Small wrote:
> What:
> Create a new package procps-base. This uses the existing procps source
> package and just enable building of pidof. procps-base will be an Essential
> package and only contain pidof.
> 
> Why:
> This would bring the pidof variant in line with other distributions.
> sysvinit-utils would no longer need to be Essential (though that's a
> separate issue) and would only have init-d-script, fstab-decode, and
> killall5.

I'm all in for shrinking the essential-set. If there is consensus to
switch pidof implementations that also seems fine to me in the abstract.
But this shuffling around of essential-ness and new tiny packages and
stuff seems a bit unnecessary to me, more so when this increases the
bootstrapping-set. I'd also rather see instead a _proper_ transition to
get sysvinit-utils out of the essential-set, and then at some later
point procps can take over pidof.

Then there's the following, which I guess complicates things:

  $ dpkg -S bin/pidof | cut -d: -f2
   /bin/pidof

Also why is killall5 not a candidate too? In any case the pidof CLI
interface does not seem too big, so this does not feels urgent to me,
given the trade-offs.

> The majority of usage of pidof is in init or pre/post scripts, which really
> should be using the LSB pidofproc function. That function in turn
> optionally uses pidof if the pidfile parameter is not given. That's
> probably a way forward for sometime in the future to not need procps-base
> Essential, but it is a way off.

I think the status_of_proc function could be switched to use
start-stop-daemon (s-s-d) --status instead of pidofproc. To replace
pidof inside pidofproc I guess s-s-d could grow some option to print
the pid, I'd be happy to implement that. After doing a quick scan over
codesearch.debian.org, I notice that it looks like many current uses
of pidofproc should instead probably be using status_of_proc, and others
seem to just be fetching the pid to then perform some action that could
instead all be done directly with s-s-d.

Thanks,
Guillem

Re: Linking coreutils against OpenSSL

[Guillem Jover]

Hi!

On Thu, 2023-11-09 at 17:38:05 -0500, Benjamin Barenblat wrote:
> coreutils can link against OpenSSL, yielding a substantial speed boost
> in sha256sum etc. For many years, this was inadvisable due to license
> conflicts. However, as of bookworm, coreutils requires GPL-3+ and
> OpenSSL is Apache-2.0, so I believe all license compatibility questions
> have been resolved.
> 
> What would you think about having coreutils Depend on libssl3? This
> would make the libssl3 package essential, which is potentially
> undesirable, but it also has the potential for serious user time savings
> (on recent Intel CPUs, OpenSSL’s SHA-256 is over five times faster than
> coreutils’ internal implementation).

This increases the pseudo-essential-set by around 6 MiB, to increase
performance for the digest tools, which do not seem like something
that gets run too often? So the trade-off here does not seem worth it
to me TBH.

The other thing is that the OpenSSL support seems to be coming from
gnulib, which is using the deprecated low-level digest functions from
OpenSSL (supposedly to be removed in the future), and where the
supported high-level ones have the problem (AFAIUI) of being affected
by the OpenSSL policies (such as FIPS or other config or similar) which
could then render those commands unusable once and if they switch API.
This is one of the reasons for why I switched dpkg from its embedded
digest functions to use libmd (currently pseudo-essential) instead one
of the other crypto libs. Of course libmd does not currently contain
optimized functions, but I'd be happy to take patches for that, or might
eventually look into adding support to it to use the Linux Kernel Crypto
API or similar.

Thanks,
Guillem

Re: [idea]: Switch default compression from "xz" to "zstd" for .deb packages

[Guillem Jover]

Hi!

On Sat, 2023-09-16 at 10:31:20 +0530, Hideki Yamane wrote:
> ## More bandwidth
> 
>  According to https://www.speedtest.net/global-index, broadband bandwidth
>  in Nicaragua becomes almost 10x
> 
>  - 2012: 1.7Mbps
>  - 2023: 17.4Mbps

Well that page still does not look too great for many other countries,
including with fixed broadband.

> ## More CPUs
> 
>  2012: ThinkPad L530 has Core i5-3320M (2 cores, 4 threads)
>  2023: ThinkPad L15 has Core i5-1335U (10 cores, 12 threads)
> 
>  
> https://www.cpubenchmark.net/compare/817vs5294/Intel-i5-3320M-vs-Intel-i5-1335U
>   - i5-3320M: single 1614, multicore 2654
>   - i5-1335U: single 3650, multicore 18076 points.
> 
>  And, xz cannot use multi core CPUs for decompression but zstd can.
>  It means that if we use xz, we just get 2x CPU power, but change to zst,
>  we can get more than 10x CPU power than 2012.

That's not correct. dpkg-deb is doing multi-threaded xz decompression
since 1.21.13, and dpkg-source is doing multi-threaded xz compression
and decompression since 1.21.14.

Also the Ubuntu zstd implementation did not have multi-threaded support
at all, the one implemented in dpkg 1.21.18 does have explicit
multi-threaded support for _compression_, but AFIUI (from zstd code and
its API being used in libdpkg) not for decompression.

>  It reduces a lot of time for package installation.

> * So, if we change {data,control}.tar file format in .deb from xz to zst,
>   we can reduce package installation time than ever since less decompression
>   time. It saves our lifetime a bit :)
> 
> * Downside: package file size is a bit larger than now, but enough bandwidth
>   will ease it for download time
>   - Not sure how repository size will be, need an experiment

Thus these seem rather hand-wavy.

And in addition support for zstd at least in Debian seems rather poor:

  https://wiki.debian.org/Teams/Dpkg/DebSupport

(Support in dpkg was mainly added to overcome the schism that Ubuntu
had created in the .deb format. :/)

Thanks,
Guillem

Re: Enabling branch protection on amd64 and arm64

[Guillem Jover]

Hi!

On Sun, 2023-08-27 at 12:51:53 +0200, Guillem Jover wrote:
> On Tue, 2023-06-27 at 16:09:40 +0100, Wookey wrote:
> > OK. We're all agreed on that then. Guillem can stick it in the next
> > dpkg upload.

So this happened, and Johannes reported that this seems to be breaking
cross-building. :(

The problem, which is in fact not new, but is made way more evident
now, is that the flags used are accepted only per arch, so when
passing for example CFLAGS (the host ones) into CC_FOR_BUILD, then
that one will not know about them and fail. (We have had this problem
up to now as we set flags per arch as some are broken in some arches,
but it being a problem depends on the host and build arches involved.)

I'm thinking about uploading later today a workaround to disable these
flags for now when cross-building. And then for the next release after
that support for _FOR_BUILD which can then take into account
these arch differences. I think some upstream code can already make
use of these, but this might need going over packaging or upstream
build systems to adapt/fix stuff. :/

And until that's done I don't think the workaround can be lifted,
and cross-compiling will generate different binaries than
non-cross-compiling. Another option would be to revert this change
until we can add it safely, but that would also be unfortunate.
OTOH, upstream code that uses stuff like CFLAGS with things like
CC_FOR_BUILD are already broken in regards cross-building, so perhaps
this can be an opportunity to flush them out?

Thanks,
Guillem

Re: Enabling -fstack-clash-protection for trixie

[Guillem Jover]

Hi!

On Sun, 2023-08-06 at 23:25:23 +0200, Moritz Mühlenhoff wrote:
> Following the procedure to modify default dpkg-buildflags I propose to
> enable -fstack-clash-protection on amd64. The bug for dpkg tracking this
> is #918914.
> 
> | -fstack-clash-protection
> | Generate code to prevent stack clash style attacks. When this option
> | is enabled, the compiler will only allocate one page of stack space
> | at a time and each page is accessed immediately after allocation.
> | Thus, it prevents allocations from jumping over any stack guard page
> | provided by the operating system.
> 
> This has been enabled on other distros for many years already (e.g.
> Fedora since 27, RHEL since 8, OpenSUSE since 15.1, Ubuntu since 19.10).
> 
> I worked with Lucas a while back and he made an archive rebuild on amd64,
> only a minimal list of packages will need to be adapted:
> http://qa-logs.debian.net/2023/05/24/
> 
> The open question is whether to also enable this for arm64, mips64el,
> ppc64el, riscv and s390x. I'm adding the respective porter lists, if there's
> consensus among porters of a given arch other than amd64 to also add
> the flag, please post a followup to #918914.

Given the results from the rebuilds for amd64 and arm64 with minimal
fallout, and no complaints, I'm going to enable this for amd64 and the
three arm arches (arm64, armhf and armel) with dpkg 1.22.0, to be
uploaded later today. We can later on modify the set of architectures
(by request from porters) or tune them if it ends up causing problems.

Thanks,
Guillem

Re: Enabling branch protection on amd64 and arm64

[Guillem Jover]

Hi!

On Tue, 2023-06-27 at 16:09:40 +0100, Wookey wrote:
> On 2023-06-27 16:58 +0200, Moritz Mühlenhoff wrote:
> > Am Wed, Jun 21, 2023 at 05:41:36PM +0200 schrieb Emanuele Rocca:
> > > On 2022-10-26 08:20, Moritz Mühlenhoff wrote:
> > > > I think this should rather be applied early after the Bookworm
> > > > release (and ideally we can also finish off the necessary testing
> > > > and add -fstack-clash-protection at least for amd64 and other archs
> > > > which are ready for it (#918914)).
> > > 
> > > Can we go ahead with the dpkg patch now, any specific tests you had in
> > > mind before applying it?
> > 
> > Note that I'm not the one driving this change (I'll start a separate
> > thread for -fstack-clash-protection in the next days), but the original
> > request was from Wookey.
> 
> > Personally I think now at the beginning of the new development cycle
> > is the ideal time to start this.
> 
> OK. We're all agreed on that then. Guillem can stick it in the next
> dpkg upload.

Right, I've queued the patch for 1.22.0, which I'm planning to upload
around today/tomorrow.

> I've not yet grokked James' comments above either which maybe imply
> adjustments to the patch? That's x86 stuff which is not my area of
> expertise. 

From a quick skim this seems most relevant for code that controls the
CPU state such as the kernel. I think we can go as is, and can amend
the flags if needed.

Thanks,
Guillem

Re: Issues in the Patch Tagging Guidelines

[Guillem Jover]

Hi!

[ Started this some days ago and only finished it now, I see Jonathan has
  covered some parts of this. ]

On Thu, 2023-08-10 at 15:42:03 +0200, Lucas Nussbaum wrote:
> On 08/08/23 at 01:25 +0200, Guillem Jover wrote:
> > Lately I've been updating metadata in patches in packages I maintain and
> > noticed several issues with the Patch Tagging Guidelines, and after Lucas
> > created the new great patches UDD service [P] and we discussed some
> > other issues there, it looked like the guidelines could do with some
> > fixes and updates.
> > 
> >   [P] <https://udd.debian.org/patches.cgi> also part of DMD.
> > 
> > The following list are some of the issues or things that might deserve
> > to be clarified, fixed or updated (for reference the current guidelines
> > can be found at <https://dep.debian.net/deps/dep3>):
> 
> For context, there are several bugs about UDD's implementation of patch 
> parsing:
> #1028503 UDD: Unknown "yes" value for Forwarded field in patch metadata
> #1031381 UDD/dmd: Incorrectly considers patches needing work
> #1043043 UDD patches: marks Forwarded as invalid if not 'no', 'not-needed', 
> 'yes' or URL
> 
> The UDD parser of DEP3 metadata is
> https://salsa.debian.org/qa/udd/-/blob/master/rimporters/patches.rb#L163
> 
> The issues listed below fall in two categories:
> 1/ general issues about DEP3
> 2/ issues about computation of the "Forwarded" field value

> 1/ General issues about DEP3
> 
> 
> > * dpatch complicates parsing, it is deprecated, probably worth dropping
> >   support for it.
> 
> I believe this is solved: dpatch is no longer available in
> stable/testing/unstable, and we don't have any packages build-depending
> on it. There are still some patches that are dpatch-formatted but do not
> depend on dpatch-specific features, so this is not a big issue.
> 
> See:
> select source, patch from sources_patches inner join patches using (hash, 
> file)
> where release='trixie' and headers ~* 'dpatch' order by 1,2;
> => 96 source packages, 295 patches

Yes, probably unclear, my point was to drop the dpatch support from
the Patch Guidelines, precisely due to this, and because otherwise a
strictly conforming parser would need to support it as it stands.

> > * Although the guidelines seem to intend for git formatted patches to be
> >   supported, the actual specification of the format is not very clear
> >   on its usage and a strict reading does not really allow it.
> > 
> >   - There is a requirement for the first field to appear on the first
> > line, but git formatted patches start with «From ».
> 
> That's OK, since From is an alternative to Author, so it's a valid DEP3
> field.

Notice the space. Here's what a git format-patch patch looks like:

,---
From 9767e87d08803d78208464be3652a9231b6b08e3 Mon Sep 17 00:00:00 2001
From: Guillem Jover 
Date: Thu, 10 Aug 2023 20:31:02 +0200
Subject: [PATCH] Patch description

Long explanation.
---
 test | 1 +
 1 file changed, 1 insertion(+)
 create mode 100644 test

diff --git a/test b/test
new file mode 100644
index 000..58bc469
--- /dev/null
+++ b/test
@@ -0,0 +1 @@
+Something.
-- 
2.40.1
`---

Which is not what the example looks like.

> >   - You cannot easily add custom Patch Guideline patches into the first
> > git stanza, those need to go into the git trailers in the commit
> > message, separated by whatever text is found in the description,
> > which does not follow the deb822 format.
> 
> That's OK. You can have random text before the next DEP3 headers. See
> the first example in the DEP3 specification.

Right. I guess my core issue here is that the specification mixes git
format patches with something that is deb822 formatted, which seems
like a bad idea, because even though they are close they have different
formatting. I think ideally it would state that you have to use either
format, but not mix them?

> >   - Having to store the patch guideline fields in git commit trailer
> > fields might mean these pollute the patch which might need to be
> > removed before submitting upstream.
> 
> In practice, supporting both git-formatted patches and custom-formatted
> patches was a non-issue when writing a parser.

This is an issue for developers not parsers though.

> > * For Applied-Upstream it is not easy to predict what will be the
> >   future upstream version that the patch will be included in. I've opted
> >   for stuff like «3.2.1+, commit:» when 3.2.1 is the last release,
> >   but that does not seem optimal.
> > 
> > * The git Date field could somehow be used in place of Last-Update (even
> >   though the Committer Dat

Re: Potential MBF: packages failing to build twice in a row

[Guillem Jover]

On Wed, 2023-08-09 at 22:10:51 +0200, Johannes Schauer Marin Rodrigues wrote:
> Quoting Guillem Jover (2023-08-09 20:55:17)
> > I think I've mentioned this before, but dpkg-source is supposed to be
> > generating reproducible source packages since around the time dpkg-deb
> > has been generating reproducible binary packages. If that's not the case
> > in some circumstance I'd consider that a bug worth fixing (or at least
> > pondering whether it makes sense to support :).

> I ran diffoscope on the differing debian.tar.xz files and got:
> 
>   --- ../hello_2.10-3.debian.tar.xz.bak
>   +++ ../hello_2.10-3.debian.tar.xz
>   │┄ Format-specific differences are supported for XZ compressed files 
> but no file-specific differences were detected; falling back to a binary 
> diff. file(1) reports: XZ compressed data, checksum CRC64
> 
> I suspect that different versions of xz produce differently compressed
> archives?

As Sven mentions this is probably due to the new parallel xz execution,
but…

> In any case, I have no time for a more thorough analysis right now but this 
> was
> an example that makes my point: passing --source to sbuild might overwrite 
> your
> existing source package with something different and thus it's not trivial
> anymore to assure that what you built really came from those exact same 
> sources
> as the source that the built was done from was not bit-by-bit identical to the
> source produced by sbuild --source. In fact the .buildinfo file produced by
> sbuild will reference the *new* dsc and that was not the dsc that the built 
> was
> done from. So my point stands: avoid running sbuild with --source.

…the same constraints would apply to dpkg-source as they would when
building binary packages, as in, it can produce reproducible source as
long as the environment described in the .buildinfo file is the same
(or equivalent functionality wise).

So this works even ignoring the .buildinfo, as long as the source is
regenerated close enough in time to when it was first created.
Otherwise if it gets regenerated at any other future time then it
might fail to reproduce, yes, and trying to honor the .buildinfo and
use an up-to-date build environment seems would be in conflict. So I
guess your recommendation makes sense if this was intended to be used
unconditionally (more so in circumstances where previously no source
would generally be created, say on binNMUs or similar).

Thanks,
Guillem

Re: Potential MBF: packages failing to build twice in a row

[Guillem Jover]

Hi!

On Wed, 2023-08-09 at 19:55:41 +0200, Johannes Schauer Marin Rodrigues wrote:
> I would only consider switching the default if at the same time, some checks
> were done that made sure that the result is bit-by-bit identical to the
> original.
> 
> The source package is the *input* to sbuild not its output. If sbuild builds
> the source package it can happen that the resulting source package is not what
> was given to sbuild to get built before.
> 
> So if the source package gets rebuilt and checked whether it is bit-by-bit
> identical to what was given to sbuild before, then essentially we would've
> enforced reproducible source packages. If I remember correctly, reproducible
> source packages are something that the reproducible builds team discarded as a
> concept many years ago.

I think I've mentioned this before, but dpkg-source is supposed to be
generating reproducible source packages since around the time dpkg-deb
has been generating reproducible binary packages. If that's not the case
in some circumstance I'd consider that a bug worth fixing (or at least
pondering whether it makes sense to support :).

Thanks,
Guillem

Issues in the Patch Tagging Guidelines

[Guillem Jover]

Hi!

Lately I've been updating metadata in patches in packages I maintain and
noticed several issues with the Patch Tagging Guidelines, and after Lucas
created the new great patches UDD service [P] and we discussed some
other issues there, it looked like the guidelines could do with some
fixes and updates.

  [P]  also part of DMD.

The following list are some of the issues or things that might deserve
to be clarified, fixed or updated (for reference the current guidelines
can be found at ):

* dpatch complicates parsing, it is deprecated, probably worth dropping
  support for it.

* Although the guidelines seem to intend for git formatted patches to be
  supported, the actual specification of the format is not very clear
  on its usage and a strict reading does not really allow it.

  - There is a requirement for the first field to appear on the first
line, but git formatted patches start with «From ».
  - You cannot easily add custom Patch Guideline patches into the first
git stanza, those need to go into the git trailers in the commit
message, separated by whatever text is found in the description,
which does not follow the deb822 format.
  - Having to store the patch guideline fields in git commit trailer
fields might mean these pollute the patch which might need to be
removed before submitting upstream.

* Forwarded does not support recording it being sent to an email address.
  Not all upstreams have a public mailing list or web site to file reports
  or send comments to, and it might be worth allowing a mailto: reference,
  or simply an email address.

* Forwarded lists "yes" as a valid implicit value, but does not make it
  clear whether an explicit one should be supported. If a mailto: or
  email is accepted then this is probably less of an issue. I've also
  used this value when I have sent the patch upstream and it has been
  applied before I have gotten around to updating the patch metadata,
  but I guess at that point not providing the field would also be
  fine.

* Forwarded fuzzy specification means parsing its values is rather hard,
  see UDD .
  It would be better to be strict here. In general choosing to be
  fuzzy about the whole specification with the intent to help humans,
  I think means that programs have a hard time (see UDD above, and
  lintian, where both disagree on semantics) and even humans can get
  more confused when crafting or parsing them.

* It is not clear, but I think «Origin: vendor» should be clarified to
  state that the actual vendor name should be included if there is no
  other reference (such as an URL) as in say «Origin: vendor, Debian»?
  Otherwise an undefined vendor is not really distinguishable from the
  «other» value as it could be any vendor. Also because if a «vendor»
  maintainer has created the patch then there might be no URL to point
  to except for the VCS it is kept in (if any at all).

* For Applied-Upstream it is not easy to predict what will be the
  future upstream version that the patch will be included in. I've opted
  for stuff like «3.2.1+, commit:» when 3.2.1 is the last release,
  but that does not seem optimal.

* The git Date field could somehow be used in place of Last-Update (even
  though the Committer Date instead of the Author Date is what matches
  here more closely, but which is not available from «git format-patch»).

* Add new metadata to track single-debian-patch autogenerated patches,
  perhaps a new «Autogenerated: yes» or perhaps better something like
  «Origin: autogenerated, by:dpkg-source» (or similar descriptive thing)?
  These should in general not be warned as needing to be forwarded
  upstream, at least not as-is (dpkg-source in 1.22.0 will add a
  «Forwarded: not-needed» for these though).

* The language could use some clarification and standardize on the field
  and stanza naming used in other parts of the deb822 ecosystem instead
  of headers and sets of headers and similar.


This is my current list, Lucas (CCed) probably has other stuff.

Thanks,
Guillem

Re: hardening flags

[Guillem Jover]

Hi!

On Sat, 2023-07-29 at 16:59:29 +0200, Martin Uecker wrote:
> are there any plans to add -fstack-clash-protection to
> the hardening flags?

See #918914.

Thanks,
Guillem

Re: systmd-analyze security as a release goal

[Guillem Jover]

Hi!

On Thu, 2023-07-06 at 18:41:38 +1000, Trent W. Buck wrote:
> "Trent W. Buck"  writes:
> > e.g. I expect "SystemCallArchitectures=native" to break for a lot of
> > people (anyone doing dpkg --add-architecture)

Yes, see #982456.

> Short version:
> 
>   • SystemCallArchitectures=native + debianutils:i386 doesn't break 
> dpkg-db-backup.service.
>   • Probably savelog simply never calls an ARCHITECTURE-SPECIFIC syscall.

That's because the only thing used from debianutils:i386 is savelog
which is a shell script, and as long as you have dash:amd64 then that
would work.

The same would apply to any other interpreted program, as long as the
interpreter matches the systemd native architecture.

>   • SystemCallArchitectures=native + nginx:i386 DOES break nginx.service.
>   • Neither journalctl nor coredumpctl makes it obvious this is WHY nginx 
> crashed.

Right.

> So I guess a program like savelog doesn't trigger it, because
> it's so simple it never hits an architecture-specific syscall?

Any binary program, will always at least call 1 syscall (either the
implicit exit() from the libc CRT function calling main(), or an
explicit exit() or raise() or similar to be able to terminate
itself. So there's no escape from this.

So SystemCallArchitectures=* seems fine as a local admin tool when they
control their deployment or for very controlled distribution builders.
Otherwise for a general purpose distribution where you can install
foreign packages, it has always seemed like a bad idea to include them
as part of the .deb, because it's depending on external implementation
details, and this is not helped with the error failure mode.

As I mentioned now on that report above, this could be made safe and
be enabled for I guess all installed services (for free!) by
"something" generating overrides based on the current list of dpkg
architectures (could perhaps be added as a hook on dpkg for the add
and remove architecture actions, or maybe as a generator, dunno).

Thanks,
Guillem

Re: Policy consensus on transition when removing initscripts.

[Guillem Jover]

On Tue, 2023-06-27 at 21:05:57 -0700, Russ Allbery wrote:
> Simon Richter  writes:
> > On 6/28/23 02:31, Russ Allbery wrote:
> >> Normally Conflicts is always added with Replaces because otherwise you can
> >> have the following situation:
> 
> >> * Package A version 1.0-1 is installed providing file F.
> >> * File F is moved to package B as of package A 1.0-3.
> >> * User installs package B, which replaces the file in package A.
> >> * User upgrades package A to version 1.0-2 (*not* 1.0-3). Or, rather,
> >>tries, because this will fail with an error due to the file conflict.
> 
> > No, that is fine. "Replaces: A (<< 1.0-3)" is sufficient here that the
> > file is not unpacked from A 1.0-2.
> 
> Oh!  Of course.  Okay.

Yes, Replaces works regardless of the order. Maybe you had in mind
that this was not always the case, and dpkg could fail to unpack
depending on the order, but that was fixed long long time ago
(dpkg 1.13.x for the general fix and 1.15.x to handle downgrades).

> In that case, I don't actually know why we usually use Conflicts with
> Replaces.  Maybe we don't really need to?

We use Breaks to avoid getting into a situation where neither of the
packages ship the file, which might make things well, break. :)
(And when we do not want to impose a Depends, but instead just force a
versioned constraint on something else.)

Regards,
Guillem

Re: 64-bit time_t transition for 32-bit archs: a proposal

[Guillem Jover]

On Tue, 2023-06-13 at 21:41:50 -0700, Steve Langasek wrote:
> After thinking about it, I'd like to suggest the following approach.
> 
> - dpkg with the new default behavior uploaded to experimental
> - libraries uploaded to experimental with the new package names (so, NEW
>   processing gets done) and with a versioned build-dependency (easy to
>   automate with sed on debian/control)
> - once all the libraries have cleared NEW, copy to unstable without dropping
>   the versioned build-dependency; it will never be wrong, it will always at
>   most be cruft that can be cleaned up lazily
> 
> What do you think?

Yeah, sounds good to me.

> On Thu, Jun 08, 2023 at 04:06:08AM +0200, Guillem Jover wrote:
> > On Mon, 2023-05-22 at 18:17:44 -0700, Steve Langasek wrote:
> > > > Hmm, rechecking the script, I think we'd want to at least
> > > > unconditionally add the Breaks and Replaces (no need for substvars
> > > > then), otherwise we'd get upgrade errors?
> 
> > > > That would leave only the Provides as potentially conditional…
> 
> > > You're absolutely right, thanks for catching this!  Fixed in git.
> 
> > As hinted above, I think the source:Version substvar should be
> > switched to a hardcoded version at the point the migration was done,
> > otherwise it will be floating forward and not catch older affected
> > versions?
> 
> Oh ok, I didn't catch that this is what you meant.  But it's not clear to me
> what you mean by "not catch older affected versions" - why would it be wrong
> to Breaks/Replaces against non-existent, newer versions of an obsolete
> binary package name?

Err, sorry right, that paragraph didn't make much sense, I think I might
have lost context between the first time I checked this and the next. :)

> It's not a difficult change to make, I just don't understand why it's
> important.

Rereading my own comment, I think what I might have been thinking about
was that the version restrictions do not make much sense, and would not
protect against potential reintroduction of those obsolete package
(probably not in Debian but elsewhere). So probably not important in
the Debian context, but it would seem more correct and future-proof to
me to completely drop the version restrictions?

> > Just to try to understand whether we are on the same page. If these
> > libraries have no time_t usage at all, then disabling time64 should
> > then provoke no time_t issue and should stop implicitly enabling LFS.
> > But if the library contains internal time_t usage that is not part of
> > the exposed ABI, but part of its operation, then I'm not sure I see
> > how we can patch them to disable LFS w/o at the same time losing
> > time64 support (as the latter forces/requires the former).
> 
> > I'm not sure whether you are talking about the first or second case?
> > And whether we have no libraries at all falling under the second case?
> 
> I was only thinking about the first case, I had not previously considered
> the second case.  We should be able to determine fairly easily whether there
> are any in the second case; for all ELF binaries which are built from the
> same source package as an LFS-sensitive library, check whether they have
> references to any of the static list of symbols in glibc that are affected
> by time_t, and if they are, add them to the list of libraries to transition.
> 
> I'll add this to the analysis in progress.

Perfect, thanks!

Regards,
Guillem

Re: re-introduction of epoch? #1037190 dhcpcd: version is lower than in wheezy

[Guillem Jover]

Hi!

On Sat, 2023-06-17 at 11:39:51 +0200, Étienne Mollier wrote:
> Martin-Éric Racine, on 2023-06-16:
> > Someone filed a bug asking to re-introduce an epoch.
> > 
> > An older fork of the same package back in Wheezy last featured the epoch.
> > 
> > Personally, I'm fine with either marking the bug as WONTFIX or
> > re-introducing the epoch for one specific binary target whose name
> > matches what was last seen in Wheezy. I simply want to hear what is
> > the mailing list's concensus.
> 
> Hmn, hard to tell, I tend to believe the severity is justified,
> as one could have carried the old dhcpcd package over a number
> of Debian versions since wheezy, and won't get the dhcpcd you
> introduced.

While I guess in general and in theory this would apply, in this
particular case I think the following does make some sense:

> On the other hand, you mention your package is a
> different implementation, so perhaps the version bump from the
> old fork to your package might have unintended effects, for
> instance if configuration file formats and such were to have
> evolved.

By reading #594672 and a quick skim over #551034, these seem to have
been the same project, but the version introduced later as dhcpcd5 was
a new major version with an incompatible redesign, which would break
on upgrade, that's why it was not packaged at the time using the same
source package. So to me it makes sense to avoid adding the epoch to
avoid automatic upgrades like it was avoided in the past, otherwise
people might expect a smooth upgrade path. Also for reference the old
dhcpcd was removed from Debian in 2014:

  https://packages.qa.debian.org/d/dhcpcd.html

Unfortunately, even though this was long ago, there seems to still be
a short tail of such package installed on systems:

  https://qa.debian.org/popcon.php?package=dhcpcd5

> The bug seems to only affect your binary package dhcpcd, so
> maybe a possible option could be to move ressources provided by
> the dhcpcd package to dhcpcd5 and remove the dhcpcd package.  It
> would avoid you the epoch bump and the hassle to handle the
> version bump from the old fork, but it also might confuse people
> using the package.  What do you think?

The only problem I see with leaving things as is, is that some users
might not notice they need to upgrade. It would be nice if we had some
way to notify of these kind of obsolete packages or upgrades.

But if you end up deciding on adding the epoch, then yeah please, just
add it to the affected binary package (even though in this case that
matches the source package, so it's going to be sticking forever I guess
anyway :/).

Thanks,
Guillem

Re: 64-bit time_t transition for 32-bit archs: a proposal

[Guillem Jover]

Hi!

On Mon, 2023-05-22 at 18:17:44 -0700, Steve Langasek wrote:
> On Fri, May 19, 2023 at 05:34:56AM +0200, Guillem Jover wrote:
> > Enabling time64 explicitly is what also had first come to my mind,
> > which does not seem too onerous if all the debhelper override
> > disappears? :) Then NEW processing could be done in one go to let the
> > flood gates open at a specific coordinated date and time range (or
> > staged via NEW into experimental, then mass uploaded to unstable to
> > reduce the pressure on the ftpmaster(s) having to approve those), at
> > which point dpkg could also be uploaded with the default switched.
> 
> The difference in my view is that the changes to handle Provides: are
> something that should persist in the packaging (until the next soname
> change, at which point it's easy to handle as part of the overall renaming),
> whereas explicit changes to set DEB_BUILD_OPTIONS to the future default are
> something that ideally would be dropped immediately after dpkg-buildflags is
> updated, and could (though unlikely) be a source of bugs later.  I'd prefer
> to avoid any transition plan which means I should be NMUing all of the
> affected library packages twice.

I think I understand where you are coming from, and I see how ideally
these would be dropped soon, but I also don't see this as a pressing
matter that would even need NMUs, as the explicit settings should map
to the then current defaults. Of course that precludes those defaults
then changing globally in the future, but if needed (!?) that could be
handled later on. But again, I think the flag day can potentially be
done in multiple other ways that might avoid the breakage w/o needing
these changes, and if so I'm happy with that as well.

> Either way, we will need to come to some sort of decision about what to do
> on i386 before we can move forward.  How should we reach that decision?  Are
> you persuaded by the arguments presented in this thread?

I think it depends a bit on the overall timing and support guarantees.
If say i386 was to be dropped very soon from the archive, then it does
not seem to me that making it an exception might be a good trade-off,
because people could then use an unsupported release for time32 and
a later unsupported one for time64 if they want to use either of those
ABIs. If there is a will to keep it longer (either as a full or partial
arch), then that changes the equation and it depends what people would
value more between ABI compat or time64-ready.

I think I covered part of this in
<https://lists.debian.org/debian-devel/2023/05/msg00228.html>. So if
there's consensus either way, I'm OK with that, but given that this is
a matter of trade-offs and what we'd want to support, Helmut's proposal
for a GR also sounds very enticing TBH, and would be a clear signal
and remove ambiguity over what the project wants to do.

> > Hmm, rechecking the script, I think we'd want to at least
> > unconditionally add the Breaks and Replaces (no need for substvars
> > then), otherwise we'd get upgrade errors?
> 
> > That would leave only the Provides as potentially conditional…
> 
> You're absolutely right, thanks for catching this!  Fixed in git.

As hinted above, I think the source:Version substvar should be
switched to a hardcoded version at the point the migration was done,
otherwise it will be floating forward and not catch older affected
versions?

> > > And btw, given the analysis that there are likely < 100 shared libraries
> > > overall whose ABI changes when enabling LFS, this might be a change we 
> > > want
> > > to consider in the trixie cycle as well - just preferably not bundled with
> > > the time_t transition at the same time, as that would just cause more 
> > > delays
> > > for testing migration vs splitting the two transitions.
> 
> > If the plan is to go with a flag day by switching the default for
> > time64, then I don't see how the LFS change can be decoupled, as that
> > automatically will also forcibly enable LFS globally on glibc arches.
> > I've also in general been worried about automatically enabling LFS w/o
> > someone looking into each package, given the greater potential for
> > data loss. :/
> 
> I think in the case of LFS-sensitive libraries that aren't part of the
> dependency graph of packages affected by the time_t change, it's easy enough
> to patch them to not turn on the LFS flag and avoid a transition.

Just to try to understand whether we are on the same page. If these
libraries have no time_t usage at all, then disabling time64 should
then provoke no time_t issue and should stop implicitly enabling LFS.
But if the library contains internal time_t usage that is not part of
the exposed ABI, but part of its operation, then I'm not sure I see
how 

Re: i386 in the future (was Re: 64-bit time_t transition for 32-bit archs: a proposal)

[Guillem Jover]

Hi!

On Fri, 2023-05-19 at 12:42:32 +0100, Steve McIntyre wrote:
> Guillem Jover wrote:
> >On Thu, 2023-05-18 at 12:01:40 -0700, Steve Langasek wrote:
> >> > […], I'm also dubious about this, and introduces a special case
> >> > and complexity that does not seem warranted TBH. If this was the case it
> >> > would seem to me it would disallow SOVERSION bumps for example, which we
> >> > have never been concerned with.

Err, sorry the SOVERSION example does not make sense here, precisely
because in those cases the dependency system works for the user and
(in general) should let the old versions of those shared libraries
even if now obsolete packages be co-installed along the new versions.

> >The problem with obsolete packages is also shared by all other arches,
> >and for those and for local packages the dependency system works for
> >the user and should let them know whether they can upgrade or they
> >would need to remove such packages. For other local FOSS packages
> >they might just be able to rebuild them.
> >
> >Excluding i386 from this transition seems to me will pretty much
> >sentence it, and would also make it rather hard to perform that
> >transition cleanly going forward if people want to keep it alive. And
> >while Debian might eventually remove it from its official ports, we
> >have multiple old ports that are still maintained and used.
> >
> >If the main reason is to support non-free binaries, at least to me
> >that does not seem like a very compelling reason. And people can
> >always use old chroots or similar I guess?
> 
> i386 is in a really awkward situation here, I think. Nobody is working
> on it explicitly any more (AFAICS?),

I assume because it seems to "work" and people might have not seen the
need to jump in, compared to say if it was in ports.d.o, but I might be
wrong.

> but its history as by far the
> most common architecture means that:
> 
>  * we still have a (very!) long tail of installations using it
>  * there are *massively* more old binaries available for it, free,
>proprietary *and* locally-built
> 
> Moving forwards, we need to make a call on what we want i386 for. I
> was hoping to wait until after bookworm is released to have the meat
> of that discussion, but...

> […]

> As and when we switch i386 to a secondary status like this (however we
> label it!), then I think we should *only* consider it as a
> compatibility layer for older software. People *could* just use old
> chroots or similar, but the need is likely to be around for a
> while.

> There's a tension here: I think it's important to keep the old ABI
> around for those old binaries, and I genuinely don't see a use case
> for a new incompatible ABI on a mostly-dead architecture that won't
> support those binaries. *But* I think we'll also need to keep the port
> going with security fixes - it's still likely to be quite common and
> we need to keep users safe.

> People are even likely to want to keep old software running beyond
> 2038, in which case I envisage clock hacks coming to keep things
> limping on. :-/

So I guess my concern is three fold:

  a) There's the part that I mentioned where excluding it means it is
 destined to die in exchange for that backwards binary compat,
 although it's not clear for how long this all will be supported
 by Debian (and where I consider I've registered my concern here,
 but if people in general think the backwards compat trumps other
 stuff I'm fine with that).
  b) I still keep at least one 32-bit Athlon system around mostly to
 be able to support 3Dfx hardware, which I'll have to stop
 supporting either when the machine, the port or time dies. OTOH
 I've been pondering stopping the support in the future anyway so
 it's not such a great concern for me.
  c) And then there's the port support long term in dpkg, where my
 expectation is that when and if Debian does not officially support
 it, people that might still want to use that port on vintage
 hardware will most probably request to flip the switch, and then
 me and/or the ports.d.o maintainers might need to decide between
 users that have expected such backwards compat and porters that
 might want to be able to continue using the port. :/

(I also keep that Athlon around as I need to recover data from a bunch
of 3.5" floppy disks! But that should stop being relevant once I sit down
through the drudge of changing the piles of disks every few minutes. :)

Thanks,
Guillem

Re: 64-bit time_t transition for 32-bit archs: a proposal

[Guillem Jover]

On Thu, 2023-05-18 at 12:01:40 -0700, Steve Langasek wrote:
> On Thu, May 18, 2023 at 03:04:30AM +0200, Guillem Jover wrote:
> > On Tue, 2023-05-16 at 21:04:10 -0700, Steve Langasek wrote:
> > > === Technical details ===
> 
> > > The proposed implementation of this transition is as follows:
> 
> […] (But I've got locally a few
> > commits fixing and improving the logic for some of the ports which
> > will push out in a bit.)

(I've pushed out those now.)

> > I'm perhaps missing something, but the ordering here seems wrong/odd?
> > Enabling this by default globally before the shared libraries affected
> > by the ABI have been handled seems to me would mean a mess of ABIs
> > in unstable until the entire thing converges?
> 
> The proposal is to treat this as a flag day, where the toolchain (in this
> case, dpkg-buildflags) changes, then we upload the affected libraries in a
> short period of time, then once they're built and through new we rebuild the
> reverse-dependencies.  This is basically the same approach as was used for
> past ABI transitions (ldbl, c102, c2) with the only difference here that the
> change triggering the flag day would be in dpkg rather than gcc(-defaults).
> 
> There could be some misbuilt binaries in unstable if the maintainer uploads
> them after dpkg is uploaded, but before the libraries are built.  If we
> uploaded all of the library packages to set DEB_BUILD_MAINT_OPTIONS =
> feature=+time64 we would avoid this race, but it would mean more
> non-scriptable changes to debian/rules which I would rather avoid, and this
> didn't prove necessary in past transitions; the impact of any misbuilt
> binaries would be short-lived as they would be replaced ASAP with binNMUs.

Given the potential amount of libraries involved that would need an
upload and the chokepoint in NEW, to me this seems like a big window
of opportunity for unstable users that might not be following along
to get a broken system (I mean yes «unstable» and all that, but if it
can be avoided?).

Enabling time64 explicitly is what also had first come to my mind,
which does not seem too onerous if all the debhelper override
disappears? :) Then NEW processing could be done in one go to let the
flood gates open at a specific coordinated date and time range (or
staged via NEW into experimental, then mass uploaded to unstable to
reduce the pressure on the ftpmaster(s) having to approve those), at
which point dpkg could also be uploaded with the default switched.

Another option, perhaps, could be to flip the defaults, but then block
uploads for affected packages using
<https://ftp-master.debian.org/transitions.yaml>, which I think the
release team controls?

But, if people in general are not bothered about this transitory
breakage, and say, an announcement is considered enough, then sure
I guess.

> > > * … but NOT on i386.  Because i386 as an architecture is primarily of
> > >   interest for running legacy binaries which cannot be rebuilt against a 
> > > new
> > >   ABI, changing the ABI on i386 would be counterproductive, as mentioned 
> > > in
> > >   https://wiki.debian.org/ReleaseGoals/64bit-time.
> 
> > Like Russ, I'm also dubious about this, and introduces a special case
> > and complexity that does not seem warranted TBH. If this was the case it
> > would seem to me it would disallow SOVERSION bumps for example, which we
> > have never been concerned with.
> 
> I didn't see anything in Russ's email in this thread that implied he was
> dubious of this approach?  He simply has a library he maintains for which he
> believes binary compatibility is uninteresting.

Ah, indeed, just reread the specific paragraph now, sorry Russ! :)

> FWIW in Ubuntu where we maintain i386 strictly as an architecture for
> compatibility with third-party binaries, we have 1034 source packages that
> build Arch: i386 debs.  Not all of those source packages are shared
> libraries, but enough of them are that manually updating them to maintain
> ABI compatibility on i386 would be a substantial amount of work.  In terms
> of overall complexity, I think the scales tip in favor of the toolchain not
> defaulting to _TIME_BITS=64 on i386.

The problem with obsolete packages is also shared by all other arches,
and for those and for local packages the dependency system works for
the user and should let them know whether they can upgrade or they
would need to remove such packages. For other local FOSS packages
they might just be able to rebuild them.

Excluding i386 from this transition seems to me will pretty much
sentence it, and would also make it rather hard to perform that
transition cleanly going forward if people want to keep it alive. And
while Debian might eventually remove it from its official ports, 

Re: 64-bit time_t transition for 32-bit archs: a proposal

[Guillem Jover]

Hi!

On Tue, 2023-05-16 at 21:04:10 -0700, Steve Langasek wrote:
> === Technical details ===
> 
> The proposed implementation of this transition is as follows:
> 
> * Update dpkg-buildflags to emit -D_FILE_OFFSET_BITS=64 and -D_TIME_BITS=64
>   by default on 32-bit archs.  (Note that this enables LFS support, because
>   glibc’s 64-bit time_t implementation is only available with LFS also
>   turned on, to avoid a combinatorial explosion of entry points.)

As has been mentioned, there's future=+time64 support in dpkg git HEAD
targeted at 1.22.0. This gets automatically disabled on ports where
time_t is known to be 64-bits (even if the port is 32-bits). When time64
is set it does indeed force future=+lfs. (But I've got locally a few
commits fixing and improving the logic for some of the ports which
will push out in a bit.)

I'm perhaps missing something, but the ordering here seems wrong/odd?
Enabling this by default globally before the shared libraries affected
by the ABI have been handled seems to me would mean a mess of ABIs
in unstable until the entire thing converges?

> * … but NOT on i386.  Because i386 as an architecture is primarily of
>   interest for running legacy binaries which cannot be rebuilt against a new
>   ABI, changing the ABI on i386 would be counterproductive, as mentioned in
>   https://wiki.debian.org/ReleaseGoals/64bit-time.

Like Russ, I'm also dubious about this, and introduces a special case
and complexity that does not seem warranted TBH. If this was the case it
would seem to me it would disallow SOVERSION bumps for example, which we
have never been concerned with.

> * For a small number of packages (~80) whose ABI is not sensitive to time_t
>   but IS sensitive to LFS, along with their reverse-dependencies, filter out
>   the above buildflags with DEB_BUILD_MAINT_OPTIONS=future=-lfs[1]. 
>   Maintainers may choose to introduce an ABI transition for LFS, but as this
>   is not required for time_t, we should not force it as part of *this*
>   transition.  If there is a package that depends on both a time_t-sensitive
>   library and an LFS-sensitive but time_t-insensitive library, however, then
>   the LFS library will need to transition.  

Hmm, I'm not sure whether I'm not understanding this point. Perhaps it
is assuming that future=-lfs will strip the LFS options from the
emitted flags? (If so that only disables the feature, but does no
stripping). Also as mentioned above when the +time64 feature is enabled
an -lfs would not be effective at all anyway, and should not! Because if
the time64 flags are emitted glibc will error out if the LFS ones are
not set. (See .)

So perhaps this means to say that in these cases we'd use:

  DEB_BUILD_MAINT_OPTIONS=future=-time64

?

> * In order to not unnecessarily break compatibility with third-party (or
>   obsolete) packages on architectures where the ABI is not actually
>   changing, on 64-bit archs + i386, emit a Provides/Replaces/Breaks of the
>   old library package name.  A sample implementation of the necessary
>   packaging changes is at
>   https://salsa.debian.org/vorlon/armhf-time_t/-/blob/main/time-t-me.

I appreciate the desire to reduce inflicting these effects on arches
that are not affected, but I don't _think_ (or do not recall) we have
ever done this kind of special casing when doing similar ABI changing
transitions in the past? And while ports have clearly arch-specific
divergences when it comes to dependencies, it seems to me this would
create a divide at a very different scale? Say for upgrade paths and
similar testing.

Personally, I'm not sure I'd be thrilled to add that kind of code to
packages I maintain. :)

> [1] Seeing that the future is now, maybe we want to consider changing the
> canonical name for this from future=-lfs to abi=-lfs?  Bikeshed at will!

Well, we are still not done with LFS, and I'm not seeing the end of that,
so it seemed appropriate at the time, and it kind of still does. :) The
«abi» name seems nice in that it conveys more clearly than «future» that
this can break the ABI (although not necessarily so), but in this case
it's probably better to err on the scarier meaning. So I'll check what
it would take to introduce an alias for that and ponder about it as
there's of course the danger of added confusion. But thanks for the
suggestion!

Thanks,
Guillem

Re: RFC: More C errors by default in GCC 14 (no more implicit function declarations etc.)

[Guillem Jover]

Hi!

On Tue, 2023-04-18 at 17:54:20 -0500, G. Branden Robinson wrote:
> At 2023-04-18T16:07:45+0200, Florian Weimer wrote:
> > TL;DR: I want to propose a GCC 14 change which will impact
> > distributions, so I'd like to gather some feedback from Debian.

> > I would appreciate some discussion on the Debian impact.  As Debian
> > generally doesn't do mass rebuilds and we have upstreamed the fixes,
> > maybe the impact is somewhat reduced?  Given that you'll get the fixes
> > as part of the rebases.  Of course, other mass-changes might trigger
> > rebuilds at a larger scale.  I guess it also depends on when you want
> > to update to GCC 14.  The later, the more likely other packages have
> > already imported the upstream fixes.  The alternative would be to
> > apply the fixes in a proactive manner, like we are trying to in
> > Fedora.
> 
> Debian has little fear of sweeping transitions, but it does like them to
> be precisely defined in scope, and easily measured in adoption.  The
> addition of a versioned contour flag to "DEB_BUILD_OPTIONS" or
> similar[3] is straightforward to measure by scanning source packages.

At the time I got aware of this effort, I implemented a dpkg-buildflags
feature (qa=+c99) to enable most of these flags (see [F]), but didn't
mention it because it was not clear whether this was a desired
direction and the current implementation seemed slightly restrictive,
also due to the freeze.

But perhaps switching more gradually does indeed work better, at least
in the Debian context. I think the feature name is not great, because it
is not very descriptive and precisely because once used it could not be
currently extended, and that ties with work I've been doing to version
feature flags (but I guess this one could always be introduced and be
considered v0 or similar, probably with a better name(?)).

Thanks,
Guillem

[F] 
https://git.hadrons.org/git/debian/dpkg/dpkg.git/commit/?h=next/modern-c=3316845bf415436299d61501db655fd2c1813436

Re: Dropping debpkg from devscripts (in trixie)

[Guillem Jover]

Hi!

On Mon, 2023-03-20 at 12:54:18 +, Benjamin Drung wrote:
> README for debpkg in devscripts says: "debpkg: A wrapper for dpkg used
> by debi to allow convenient testing of packages.  For debpkg to work, it
> needs to be made setuid root, and this needs to be performed by the
> sysadmin -- it is not installed as setuid root by default.  (Note that
> being able to run a setuid root debpkg is effectively the same as having
> root access to the system, so this should be done with caution.)  Having
> debpkg as a wrapper for dpkg can be a Good Thing (TM), as it decreases
> the potential for damage by accidental wrong use of commands in
> superuser mode (e.g., an inadvertent rm -rf * in the wrong directory is
> disastrous as many can attest to)."

Ugh, yes, this seems like very bad advice TBH. It also seems a bit
pointless? If you are going to open up such root back-door in your
system why all this complication, you might as well make dpkg itself
set-uid-root or set-gid-root (just to be clear, for unsuspecting
readers, the previous is not a recommendation; do not do that!).

And the wrapper is simply forwarding everything to dpkg itself, so
there is not much of filtering or sanitization going on there.

> The "Wrapper script" section in README from devscripts goes into the
> details and explains that you can invoke the wrappers with "sudo" or
> "super" or, highly dangerous, make debpkg setuid.

> debpkg uses a wrapper script written in C which makes devscripts
> architecture any. If we drop debpkg, we can make devscripts architecture
> all.

> IMO sudo (or equivalent) is superior to make debpkg setuid. Are there
> use cases that cannot be covered by using sudo? If there are no
> objections, my plan will be to remove debpkg from devscripts in trixie
> (i.e. after the bookworm release).

Yes, please, let's remove the wrapper and all the recommendations
about it. If there's a need/demand, I'd be happy to also include
a polkit action for dpkg itself (alongside the existing one for
update-alternatives), which could be an alternative to the
sudo/super usage.

Thanks,
Guillem

Re: Problems verifying signed github releases (Re: Q: uscan with GitHub)

[Guillem Jover]

Hi!

On Sun, 2023-02-19 at 18:34:56 +0100, Jens Reyer wrote:
> [This is a followup to the thread "Q: uscan with GitHub" at
> https://lists.debian.org/debian-devel/2022/10/msg00313.html. I manually set
> in-reply-to, but am not sure if it'll work.]

> My upstream creates a tarball with git-archive, creates a signature and
> uploads it (as described in the wiki[3]).  This used to work to verify
> the github-created tarball, but fails now - while creating my own
> tarball like upstream and verifying it with upstream's signature works.
> 
> The uncompressed .tar files are identical (same hashsum), just the tar.gz
> differ.  Does anyone know why, and how to fix it?  I tried non-default
> compression levels for gzip with git-archive, but that didn't help to get an
> identical tar.gz like the one from github.
> 
> I'd like to avoid having my upstream downloading the github-created
> tarball, verify it and then upload this signature.

I assume you (or whatever service or tool is failing the verification
while creating a local tarball) might be seeing issues with git having
switched implementation for gzip, and a mismatch with the implementation
being used in either side. Perhaps try to set git's
tar.tar.gz.command="gzip -c" (or/and «tgz» instead of «tar.gz») to use
the external command instead of the internal implementation? Or perhaps
you are using an old git that defaults to the external gzip but upstream
uses the internal one?

(There was a recent LWN article covering this, see
https://lwn.net/Articles/921787/.)

Thanks,
Guillem

Re: rust-*/librust-* arch:any vs. arch:all and cross-compilation support

[Guillem Jover]

Hi!

On Wed, Feb 15, 2023, at 7:51 PM, Fabian Grünbichler wrote:
> debcargo, the tool used by the Debian rust team to streamline the work
> of transforming upstream crates into Debian source packages generates
> d/control entries for librust-* binary packages qualified as arch:any
> and M-A:same, in order to support cross compilation. This solution was
> suggested by dpkg developers (Guillem?) according to the following
> comment left in the debcargo source code:

Hmm, I had no recollection of having had such conversation or given
that recommendation. Checking the history for that comment, and the
IRC logs for #debian-dpkg around the same time, seems like that was
discussed there and apparently resolved, and I guess I mostly skimmed
afterwards over the conversation and mentioned I was glad it had been
resolved (but where I don't recall having spent time pondering about
it :).

>// This is the best but not ideal option for us.
>//
>// Currently Debian M-A spec has a deficiency where a package X that
>// build-depends on a (M-A:foreign+arch:all) package that itself
>// depends on an arch:any package Z, will pick up the BUILD_ARCH of
>// package Z instead of the HOST_ARCH. This is because we currently
>// have no way of telling dpkg to use HOST_ARCH when checking that the
>// dependencies of Y are satisfied, which is done at install-time
>// without any knowledge that we're about to do a cross-compile. It
>// is also problematic to tell dpkg to "accept any arch" because of
>// the presence of non-M-A:same packages in the archive, that are not
>// co-installable - different arches of Z might be depended-upon by
>// two conflicting chains. (dpkg has so far chosen not to add an
>// exception for the case where package Z is M-A:same co-installable).
>//
>// The recommended work-around for now from the dpkg developers is to
>// make our packages arch:any M-A:same even though this results in
>// duplicate packages in the Debian archive. For very large crates we
>// will eventually want to make debcargo generate -data packages that
>// are arch:all and have the arch:any -dev packages depend on it.
> 
> https://salsa.debian.org/rust-team/debcargo/-/blob/master/src/debian/control.rs#L342

I'm afraid after a quick read, I'm failing to understand what this
comment is trying to convey. Some of the things it's saying seem like
would apply to frontends driving dpkg for example, others I'm not sure
what they refer to.

From my reread of the IRC logs, it seems like the thing this was trying
to avoid was the "multiarch interpreter problem", but then w/o having
really put much thought at all into this right now (so take this with
a huge grain of salt), this does not feel to me like the same issue
when it comes to arch:all packages shipping sources to be used to
build the actual binaries. But again take this more like me having
currently no opinion on it right now w/o investing some time to think
this thoroughly, than anything else.

I think Helmut is probably at a better position to at least give the
rationale he had at the time for this.

Thanks,
Guillem

Re: -ffile-prefix-map option and reproducibility

[Guillem Jover]

On Tue, 2023-02-07 at 20:00:06 +0100, Sven Joachim wrote:
> On 2023-02-07 17:50 +0100, Guillem Jover wrote:
> > On Tue, 2023-02-07 at 16:41:47 +0100, Stéphane Glondu wrote:
> >> I suspect this was added to improve reproducibility. Ironically, it makes
> >> packages that capture this variable non reproducible, since the build path
> >> seems to be randomized (has it always been the case? since when?). It is 
> >> the
> >> case of OCaml (see #1030785), and seemingly of R as well (found by grepping
> >> in my /etc). I wouldn't be surprised other packages are affected as well.
> >
> > AFAIR this was considered at the time, yes. If the flag is effectively
> > not fixing anything for the set of packages involved, and is in fact
> > actually making them unreproducible when they would not then, you can
> > disable the fixfilepath feature in the reproducible build flags area,
> > via DEB_BUILD_MAINT_OPTIONS.
> 
> This does not help for packages which capture all build flags and store
> them in some file in the package (as is the case here).  With
> DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath, dpkg-buildflags falls
> back to "-fdebug-prefix-map==.", and you have the same
> problem.  If you disable that as well via
> DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath,-fixdebugpath, the
> -dbgsym packages will most likely end up unreproducible.

Ah, you are absolutely right. I don't think the case of these flags
making the build unreproducible and the package not generating any
debug objects are going to be common at all. I considered mentioning
fixing the build to stop capturing, but felt it might be more effort
than requested. :) Should probably have mentioned anyway, as was done
elsewhere in the thread.

In any case I'm thinking to add something like the attached to the man
page to try to clarify this.

Thanks,
Guillem
From 1d94da75b63115485ad1247c44f64973ed74339f Mon Sep 17 00:00:00 2001
From: Guillem Jover 
Date: Wed, 8 Feb 2023 11:26:14 +0100
Subject: [PATCH] man: Add notes about reproducibility properties for path
 fixing build features

Mention that if the build captures the build flags that will make it
unreproducible, the traps with trying to disable these flags to get
back to a reproducible output. And that the ideal fix is to stop
capturing build flags.

Prompted-by: Sven Joachim 
---
 man/dpkg-buildflags.pod | 9 +
 1 file changed, 9 insertions(+)

diff --git a/man/dpkg-buildflags.pod b/man/dpkg-buildflags.pod
index 6067d4923..01c14269e 100644
--- a/man/dpkg-buildflags.pod
+++ b/man/dpkg-buildflags.pod
@@ -622,6 +622,13 @@ This has the effect of removing the build path from any generated file.
 If both B and B are set, this option
 takes precedence, because it is a superset of the former.
 
+B: If the build process captures the build flags into the resulting
+built objects, that will make the package unreproducible.
+And while disabling this option might make some of the objects reproducible
+again this would also require disabling B, which might make
+any generated debug symbols objects unreproducible.
+The ideal fix is to stop capturing build flags.
+
 =item B
 
 This setting (enabled by default) adds
@@ -632,6 +639,8 @@ set to the top-level directory of the package being built.
 This has the effect of removing the build path from any generated debug
 symbols.
 
+B: This feature has similar reproducible properties as B.
+
 =back
 
 =head1 ENVIRONMENT
-- 
2.39.1

Re: -ffile-prefix-map option and reproducibility

[Guillem Jover]

Hi!

On Tue, 2023-02-07 at 16:41:47 +0100, Stéphane Glondu wrote:
> When building packages, a -ffile-prefix-map option is automatically injected
> into CFLAGS. Where does it come from? Since when?

This is coming from dpkg-buildflags (in this case probably indirectly
via debhelper). AFAICS it was added in dpkg 1.19.1 disabled by default,
and then switched to enabled by default in dpkg 1.20.6 (see #974087).

> I suspect this was added to improve reproducibility. Ironically, it makes
> packages that capture this variable non reproducible, since the build path
> seems to be randomized (has it always been the case? since when?). It is the
> case of OCaml (see #1030785), and seemingly of R as well (found by grepping
> in my /etc). I wouldn't be surprised other packages are affected as well.

AFAIR this was considered at the time, yes. If the flag is effectively
not fixing anything for the set of packages involved, and is in fact
actually making them unreproducible when they would not then, you can
disable the fixfilepath feature in the reproducible build flags area,
via DEB_BUILD_MAINT_OPTIONS. Perhaps even "globally" from a language
specific packaging helper or similar?

> Is there a way to not get this option? More elegant than explicitly
> filtering it out of CFLAGS in debian/rules...

See above.


I just noticed that several of these build flag features do not have
information in the man page about when they got first introduced, so
I'll be adding that in dpkg 1.22.x, once development opens up again.

Thanks,
Guillem

Re: Please, minimize your build chroots

[Guillem Jover]

On Sun, 2023-01-29 at 12:28:37 +0200, Adrian Bunk wrote:
> On Sun, Jan 29, 2023 at 05:00:56AM +0100, Guillem Jover wrote:
> > On Sat, 2023-01-28 at 21:35:01 +0200, Adrian Bunk wrote:
> > > Removing tzdata from the essential set made sense, and I do see your 
> > > point that tzdata does not fit the "totally broken" definition of
> > > "required".
> > 
> > > What we need are technical discussions like whether packages like tzdata 
> > > should also be removed from the build essential set, or whether tzdata 
> > > being a part of the build essential set should be expressed by making 
> > > the build-essential package depend on tzdata.
> > 
> > I guess my question is, what makes tzdata build essential, besides
> > that it's currently kind-of implicitly there? To me it does not look
> > like it has the properties to be considered as such, besides that if
> > we lower its priority (as it deserves) it makes a bunch of packages
> > FTBFS.
> 
> It has historically been part of the build essential set.

(So the "that it's currently kind-of implicitly there".)

> It is used in the build of many packages.

Given the number of packages that currently declare a dependency on
tzdata (34~), the ones that seem to have the most potential to pull it
for others are language bindings such as python3-dateutil, python3-tz
ruby-tzinfo, etc, which handle timezone stuff anyway and would keep
pulling it. So I find this assertion hard to believe. And the following
points seem to be based on this fundamental assumption.

I think though, we might be able to easily answer at least how many
builds ended up with tzdata installed (even if it ended up not being
used), if Santiago still has the build logs around, those would be
easily grep-able.

(Whether it was used at all could be checked after each build through
atime I guess. But this seems all like a lot of wasted effort TBH.)

And for reference, Santiago seems (I might be wrong) to have filed
around 46 reports about this, which seems like a very tiny speck in
the grand scheme of things.

> There would be so many invisible undeclared build dependencies of 
> packages using it who have it pulled in by other packages that random
> changes in the dependency tree might cause many packages to FTBFS at
> any time if it does not stay build essential.

I fail to see how this is any different from any other transitive
dependency being pulled automatically due to some implementation
detail somewhere.

> It is required to provide glibc functionality that is frequently
> used during the build.

This seems like a reiteration of the above "it's used a lot". There
are other things that are used a lot during builds and yet they are
not build-essential (python3 for an obvious example).

> > So, for one, this is getting in the way of making our minimal
> > (non build) systems smaller.
> 
> No, it is not.
> 
> There are 3 different topics:
> 
> 1. making minimal (non build) systems smaller
> 
> Being able to exclude tzdata from a minimal system was achieved when it 
> was removed from the essential set in stretch.
> debootstrap not installing it by default would make that easier.
> Whether build-essential depends on tzdata does not make any difference.

Sure, you can, but it's not the _default_, so it *is* getting in the way.
And some seem to be even trying to tie it into the build-essential set by
proxy, wanting to bolt Priority:required into it… People keep bringing up
in this discussion the sanctity of these default installations, so it
again seems unfair to then disregard it when it's not convenient.

> 2. normal systems
> 
> In a normal (non-minimal) installation not having tzdata installed 
> would be a bug harming many users, no matter what priority it will
> have.

Sure, this can easily be achieved by making it Priority:important,
so I don't see this being a concern.

> 3. build essential
> 
> That's separate from 1. and 2.

Sure again (as long as it is not kept as Priority:required "for
reasons").

> > > I have so far not seen any technical arguments why removing tzdata from 
> > > the build essential set would be better for Debian than keeping it there.
> > > Removing tzdata reduces the size of a chroot that has the build 
> > > dependencies for the hello package installed by ~ 0.5%, this size
> > > decrease does not strike me as a sufficient reason for reducing the
> > > build essential set.
> > 
> > I don't see how this can be a pure technical decision, TBH. To me this
> > looks more like either a matter of trade-offs,
> 
> It is a tradeoff between less work and saving ~ 0.5% space in build
> chroots.

That would be the trade-off you see, yes. :)

> >

Re: Please, minimize your build chroots

[Guillem Jover]

On Sat, 2023-01-28 at 21:35:01 +0200, Adrian Bunk wrote:
> I don't think such arguments are bringing us forward,
> we should rather resolve the problem that these differ.
> 
> All/Most(?) packages where they do differ are packages that were until 
> recently part of the essential set, and since debootstrap still installs 
> them they are in practice still part of the build essential set.

Sure.

> Removing tzdata from the essential set made sense, and I do see your 
> point that tzdata does not fit the "totally broken" definition of
> "required".

> What we need are technical discussions like whether packages like tzdata 
> should also be removed from the build essential set, or whether tzdata 
> being a part of the build essential set should be expressed by making 
> the build-essential package depend on tzdata.

I guess my question is, what makes tzdata build essential, besides
that it's currently kind-of implicitly there? To me it does not look
like it has the properties to be considered as such, besides that if
we lower its priority (as it deserves) it makes a bunch of packages
FTBFS. So, for one, this is getting in the way of making our minimal
(non build) systems smaller.

> I have so far not seen any technical arguments why removing tzdata from 
> the build essential set would be better for Debian than keeping it there.
> Removing tzdata reduces the size of a chroot that has the build 
> dependencies for the hello package installed by ~ 0.5%, this size
> decrease does not strike me as a sufficient reason for reducing the
> build essential set.

I don't see how this can be a pure technical decision, TBH. To me this
looks more like either a matter of trade-offs, or IMO more importantly
of clean definition of a specification (which seems rather more
important than the other concerns). The work to spot these problems has
already been done, and the changes required are trivial and minimal
(disregarding any severity consideration here).

> Everyone can feel free to disagree with me on the previous paragraph,
> but please argue technically and not based on wording in policy.

I can see the appeal, but the problem is that if we completely ignore
policy, then we are starting from nothing, which means deciding over
this is rather open-ended. And of course, policy is not set in stone
and can be fixed, amended, extended as needed, but I think it's our
base from where to start, so requesting to completely ignore it for
this discussion seems not ideal?

The current definition of what the build essential set is supposed to
mean, from policy, seems rather easily mappable to actual package
names (which was the intention, as to not hardcode those or their
organization in policy itself):

  ,---
  It is not necessary to explicitly specify build-time relationships on
  a minimal set of packages that are always needed to compile, link and
  put in a Debian package a standard “Hello World!” program written in C
  or C++. The required packages are called *build-essential*, and an
  informational list can be found in "/usr/share/doc/build-
  essential/list" (which is contained in the "build-essential" package).
  `---

Which maps to a C and C++ compiler (including linker and C library),
make for debian/rules, and deb toolchain.

I appreciate the minimalism and simplicity of the definition. I've
also heard from time to time complains that we even require a C/C++
compiler as build-essential, when for many packages that's not even
needed (although a C compiler is currently needed by dpkg-dev to
determine the host arch).

Policy also has this to say:

  ,---
  If build-time dependencies are specified, it must be possible to build
  the package and produce working binaries on a system with only
  essential and build-essential packages installed and also those
  required to satisfy the build-time relationships (including any
  implied relationships). […]
  `---

Given that "tzdata" does not fit into the "required" definition anyway,
so the proposal to extend build-essential to include the "required"
priority does not make sense; having to slap in there, that “oh BTW, in
addition we also need timezone data, which is not going to be needed at
all by that "Hello World!" program”, seems like a rather weird and
gratuitous requirement, and hackish definition wise, just to avoid
fixing those bug reports?

There are many things we could also include in build-essential to
avoid declaring them, which we still not do, so I'm not sure why we
need to special case timezone data here.

Regards,
Guillem

Re: Please, minimize your build chroots

[Guillem Jover]

On Sat, 2023-01-28 at 16:32:17 +0100, Adam Borowski wrote:
> On Sat, Jan 28, 2023 at 01:59:40PM +0100, Santiago Vila wrote:
> > Unsupported by whom? What is supported or unsupported is explained in 
> > policy.
> > Policy says it must work. Therefore it should be supported (by fixing the 
> > bugs).
> 
> Policy §2.5:
> # "required"
> #Packages which are necessary for the proper functioning of the
> #system (usually, this means that dpkg functionality depends on
> #these packages). Removing a "required" package may cause your
> #system to become totally broken and you may not even be able to use
> #"dpkg" to put things back, so only do so if you know what you are
> #doing.

As stated several times now this passage seems wrong, or inaccurate at
best. See #950440. And I don't see how tzdata would ever fall into
this definition even if that paragraph was correct. As mentioned
before, the tzdata package does not seem like a "required" package at
all, and this should be fixed by lowering its priority. Whether
debootstrap can be fixed to not use the Priority workaround, seem
orthogonal to the issue at hand.

> > That's a straw man. I'm not proposing anything of the sort. Policy says
> > packages must build when essential and build-essential packages
> > are installed (plus build-dependencies).
> 
> Build-essential _packages_.  Not the "build-essential" package which very
> clearly says its dependencies are purely informational.

It does not seem fair to argue both that the build-essential package is
just informational (when it's in fact the canonical declaration of what
is Build-Essential, and what every tool uses to install or check for the
Build-Essential package set), and then also argue that whatever
debootstrap installs (which is based both on build-essential plus a
workaround due to lack of proper dependency resolution) is the canonical
thing.

Regards,
Guillem

Re: depends-on-obsolete-package lsb-base

[Guillem Jover]

On Wed, 2023-01-18 at 18:23:16 +0100, Adam Borowski wrote:
> On Wed, Jan 18, 2023 at 08:19:15AM -0800, Otto Kekäläinen wrote:
> > Lintian just started erroring on 'depends-on-obsolete-package
> > lsb-base' on many of my packages yesterday.
> 
> It's a very low priority cleanup; the Depends is redundant but
> harmless.

I don't think it is redundant though? Just removing the lsb-base
Depends can break packages on partial upgrades in the same way lsb-base
broke stuff before it grew a versioned dependency on sysvinit-utils.

While lsb-base is Priority required in bullseye, it is optional in
bookworm (although I think at least for apt these might be sticky?),
but regardless, if there are no remaining dependencies on it, these
could end up being removed by eager users.

> > There are no new uploads of lsb-base recently
> 
> Relevant changes are from September.  The lsb-base package is empty,
> and would have been gone completely if debootstrap understood Provides.

That would have also broken partial upgrades iff packages then removed
the dependency.

> > and I did not find any news about this topic. The
> > Lintian page https://lintian.debian.org/tags/depends-on-obsolete-package
> > is only about the error in general, not any mention of lsb-base
> > specific changes.
> 
> The description of "$PACKAGE1 => $PACKAGE2" are clear enough, of
> "$PACKAGE => nothing" a bit less, indeed.  You can just drop the dependency.
> 
> > Does somebody know what is going on?
> > 
> > Example:
> > E: mariadb-server: depends-on-obsolete-package Depends: lsb-base (>= 3.0-10)
> 
> The severity of this warning is grossly overstated.  It shouldn't be an
> "error" but a "mild warning to fix if you're bored".

I think the lintian tag should either tell to replace lsb-base with
«sysvinit-utils (>= 3.05-4~)» or it should say nothing and packages
should simply live with that dependency until the next release, then
it can be dropped. The advantage of the second option is that it
implies less packaging churn.

Regards,
Guillem

Re: Please, minimize your build chroots

[Guillem Jover]

On Fri, 2022-12-16 at 18:55:42 +0100, Andreas Metzler wrote:
> or apt.
> 
> I am wondering if there is point to this or whether policy should be
> changed? Is there some value in investing work in having packages
> buildable without Prioriry required packages?
> 
> Such installations can only be found as artifacts since there does not
> seem to be a supported way to upgrade them or (un)install packages
> (quoting policy: "Removing a "required" package may cause your system to
> become totally broken and you may not even be able to use "dpkg" to put
> things back, so only do so if you know what you are doing.") Essentialy
> policy is telling us it might work to install b-d, and uninstall
> Prioriry required, however dpkg might break halfway through and it would
> not be a bug.

  

Thanks,
Guillem

Re: Bug#1026087: ITP: distribution-gpg-keys -- GPG keys by various Linux distributions

[Guillem Jover]

Hi!

On Wed, 2022-12-14 at 15:27:18 +0100, Juri Grabowski wrote:
> Package: wnpp
> Version: 1.79
> Severity: wishlist
> Owner: Juri Grabowski 
> X-Debbugs-Cc: debian-devel@lists.debian.org, deb...@jugra.de

> * Package name: distribution-gpg-keys
>   Version : 1.7.9
>   Upstream Author : Miroslav Suchý 
> * URL : https://github.com/xsuchy/distribution-gpg-keys/
> * License : CC0-1.0
>   Programming Lang: data
>   Description : GPG keys by various Linux distributions
> 
> used by various Linux distributions to sign packages.
> 
> needed by mock/3.5 and I need a sponsor for this package. See current 
> packaging in
> https://salsa.debian.org/pkg-rpm-team/distribution-gpg-keys
> After I know ITP bug number I upload this source package to
> https://mentors.debian.net/package/distribution-gpg-keys/

The project name talks about gpg keys, but those are really OpenPGP
keys (or even better, certificates). I've asked upstream to rename the
project to avoid this common confusion. So you might want to wait until
that's settled to avoid multiple trips over NEW.

  

(If this project is also intended to only cover RPM-based distributions,
as Adam brought up, you might want to further ask them to make that clear
from the project name, say rpm-distribution-openpgp-keys or similar. But
in any case regardless of the intended target use, it still seems like a
very generic name which can be easily confused for a package that would
be needed for Debian, or derivatives.)

Thanks,
Guillem

Re: R³ by default: not for bookworm

[Guillem Jover]

Hi!

On Sun, 2022-09-18 at 03:39:43 +0200, Adam Borowski wrote:
> A few packages had a value of R³ other than "no" / "binary-targets",
> these are deprecated now; bugs filed.

Deprecated by who or what?

> The process of adding/changing a field in "control" differs between the
> three source formats we have.

Hmm, I'm not sure I understand this statement.

> Of these, the most involved format is 1.0 -- you need to repack the
> whole source. And quite a bunch of packages fail that step, not even
> letting me to modify anything.  I guess FTBS bugs need to be enforced...

Nor this one. Could you give more details?

> Almost any format 1.0 package with R³ unset does so not because of an
> actual need for fakeroot, but because of an ancient build system and a
> decade or two of neglect.

Lack of debhelper/dh usage certainly makes adding the field more
challenging, yes.

> Format "3.0 (native)":
> The complete list of packages that FTBFS if you set them to R³:no is:
[…]

> Format "3.0 (quilt)":
> In a pile of build logs that looks incomplete:
> 
> 408 Status: attempted
>   6 Status: failed
>  32 Status: given-back
>  15 Status: skipped
>   12387 Status: successful

Thanks for these checks! But in addition to checking whether these failed,
did you check that they ended up with the same user:group and perms (such
as SUID), as before setting the field?

> Thus: let's revisit R³ being required after Bookworm.

My current thinking though, has been to change the default via something
like:

  

Thanks,
Guillem

Re: packages expected to fail on some archs

[Guillem Jover]

Hi!

[ Mostly to summarize the status re dpkg. ]

On Sun, 2022-09-11 at 17:08:57 +0200, Samuel Thibault wrote:
> The issue we see is that some DDs end up setting a hardcoded list in
> the "Architecture" field, rather than just letting builds keep failing
> on these archs (and then possibly succeeding after some time whenever
> somebody contributes a fix upstream that gets propagated to Debian).

> That said, I guess DDs would still set a manual Arch list so as to get
> the red flags away from their sight on the buildd page status, e.g. from
> https://buildd.debian.org/status/package.php?p=sthibault%40debian.org=yes=compact
> so that they can easily notice which build failures are actually not
> expected, to be able to efficiently work on those.
> 
> So perhaps what is missing is a way to express that an FTBFS is
> expected, so that the buildd page status represents it differently, e.g.
> in orange rather than red?  The idea would be that it should be really
> easy to set (as easy as setting an Architecture list) so that we can
> easily recommend it.

Yes, I agree that this seems like a problem with representing state,
where people feel the need to move from the red to green. I think the
proposals that have been mentioned elsethread are nice, where a
regression would be red but whenever a package was not previously
built (because it never has or it got removed) for an arch, that
is shown in a way consistent with how testing transitions would take
it (orange or another non-red color seems good).

Something else to consider is that, for packages that make sense
porting, deny-listing them from building means we do not have build
failure logs, so deciding what to port or trying to check for patterns
becomes more costly for humans, of course at the cost of potentially
throwing at it buildd resources.

> We could for instance:
> - Add an Architecture-FTBFS field to debian/control
> - Add an environment variable to debian/rules so that on these archs dh
>   fails with a different exit code that buildds would notice.
> - Add a Architecture-FTBFS field in the wb database that DDs can set
> 
> The tricky part I see is that AIUI the buildd status page doesn't
> have direct access to debian/control, only to the wb database, so the
> first solution is not immediate to implement. The third option would
> be the most obvious from the buildd point of view, but that's not very
> convenient for DDs. Possibly such field could be automatically set
> according to the Packages entry when a newer upload is done?

While these might be worth it on their own, it feels (as has been
covered elsethread) to be a bit of a workaround for lack of better
expressiveness in our Architecture declarations, and build state
representation. But see my reply to #807263 about how I see the
different cases. Also for the third we already had something similar
with the Packages-arch-specific file, but my understanding is that we
are moving away from it?

The common mentioned omissions are:

  * Lack of negated architectures in the field. As mentioned on #807264
this is kind of supported already by dpkg, but it would need to be
accepted by the project, and the actual syntax and semantics
documented. (Personally I think this should not generally be used,
and there are very few cases where this seems legitimate?)

  * Lack of bits/endianness arch "aliases" (#962848). The main problem
with this one is that we cannot simply add such aliases, as then
those would silently be considered as regular arches, and they do
not map into the current naming convention at all. These would need
to be added with a breaking syntax (say with some non-accepted
char, such as % or whatever) so that they do not introduce silent
breakage. This would then need to be supported by anything
handling arch restrictions (field and dependencies) which can be a
rather large surface. Then there is the problem that architectures
are evaluated as ORed lists, and the bits/endianness might require
to be treated as ANDed lists some times (of course the latter
could be handled by combining them into single aliases, but meh).

Thanks,
Guillem

Re: Transition proposal: pkg-config to pkgconf

[Guillem Jover]

Hi!

On Mon, 2022-08-29 at 15:38:08 +0200, Andrej Shadura wrote:
> It seems like the proposal went mostly unnoticed. Any comments,
> ideas, anything?

I at least read it, and it looked great. But…

> Most importantly, I need a green light from the current pkg-config
> maintainer (Tollef) to proceed with the plan.

…as it was waiting for this, it slipped my mind.

> Meanwhile, I have uploaded an updated version of the package which
> dropped the crosswrapper, and instead uses the "personalities"
> feature to implement its functionality. The package also no longer
> needs a dpkg hook, as it ships a per-architecture symlinks in
> arch-specific packages.

Thank you for getting rid of both of these annoyances! I switched all
my hosts and chroots to pkgconf, when I watched your DebConf presentation
and it's been working great.

> On Sun, 24 Jul 2022, at 17:33, Andrej Shadura wrote:
> > Following a discussion at DebConf, I’d like to officially propose a 
> > transition
> > from pkg-config to pkgconf in Debian.
> >
> > pkgconf is a newer, actively maintained implementation of pkg-config that
> > supports more aspects of the pkg-config file specification and provides a
> > library interface that applications can use to incorporate intelligent
> > handling of pkg-config files into themselves (such as build file generators,
> > IDEs, and compilers).

Right, I track the pkg-config upstream git repo and it unfortunately has
seen no major development since 2019, and not upstream release since 2017.

I've also taken a peek to pkgconf from time to time, and it looks like
a nice project, code wise, etc.

> > Debian would benefit from switching to pkgconf by utilising a more complete
> > implementation of pkg-config that handles aspects of .pc files that 
> > pkg-config
> > does not. For example, Provides rules and Cflags.private rules are 
> > supported,
> > and pkgconf has a better (and less costly) dependency resolver for .pc 
> > files.

Yes.

> > Migration plan
> > ==
> >
> > I believe that it would in the best interests of Debian to only ship one
> > pkg-config implementation, so I propose to make pkgconf provide the binary
> > package pkg-config without a possibility to fall back to the original
> > implementation. This means that after pkgconf takes over the pkg-config
> > binary package, all packages depending on pkg-config will be using the 
> > pkgconf
> > implementation without being able to opt out and use the one from
> > freedesktop.org. While this may seem limiting, it reduces the probability of
> > some packages staying with the freedesktop.org implementation indefinitely 
> > and
> > missing out on bug fixes or suffering from other differences in 
> > implementation
> > details in future when pkgconf becomes even more widespread.

If pkg-config was showing signs of activity, then this could perhaps
be a potential concern, but given its current state, I think this makes
perfect sense.

> > In preparation for this migration, I ran a rebuild of all packages depending
> > on pkg-config and found only very few failures [5], which may or may not be
> > related to the difference in behaviour between pkg-config and pkgconf. I 
> > will
> > perform another rebuild and continue investigating them to provide patches
> > during the transition period.

Great!

Thanks,
Guillem

Re: Automatic trimming of changelogs in binary packages

[Guillem Jover]

Hi!

On Thu, 2022-08-18 at 21:18:35 +0200, Gioele Barabucci wrote:
> in 2020 there was a brief discussion on debian-devel@ about trimming
> changelogs [1,2].

My objections from that time still stand:

  

I would also like to highlight David Kalnischkies reply:

  

Thanks,
Guillem

Re: Done: Switch default from netkit-telnet(d) to inetutils-telnet(d)

[Guillem Jover]

Hi!

On Sun, 2022-07-17 at 04:18:59 +0200, Guillem Jover wrote:
> There's been talk about switching away from netkit-telnet and
> netkit-telnetd as the default implementations for some time now,
> and replacing them with the ones from inetutils, which is a maintained
> project and does see releases (even though with a long cadence).

The described plan is implemented and all done:

  - Upload of inetutils taking over the packages and updating
alternative priority.
  - Upload of netkit-telnet renaming the packages.
  - Migration of inetutils to testing.
  - Overrides update request filed and applied.
  - Bugs files for the direct telnet/telnetd usage:

<https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=guillem%40debian.org=inetutils-default-telnet-switch>

<https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=guillem%40debian.org=inetutils-default-telnetd-switch>

Thanks,
Guillem

Re: RFC: Additions to dpkg's Pre-Depends

[Guillem Jover]

Hi!

On Wed, 2022-07-06 at 05:13:05 +0200, Guillem Jover wrote:
> As per Debian policy §3.5, and given dpkg “Essential: yes” nature, I'm
> bringing up the following potential additions to dpkg's Pre-Depends,
> and whether there is consensus about each of them individually and
> independently.

Given there's been no major concerns expressed, I'll start
incrementally adding these dependencies, when the code is ready.

Thanks,
Guillem

Re: RFC: Switch default from netkit-telnet(d) to inetutils-telnet(d)

[Guillem Jover]

Hi!

On Sun, 2022-07-17 at 04:18:59 +0200, Guillem Jover wrote:
> There's been talk about switching away from netkit-telnet and
> netkit-telnetd as the default implementations for some time now,
> and replacing them with the ones from inetutils, which is a maintained
> project and does see releases (even though with a long cadence).

Ok, so given the comments, we'll be starting with the outlined plan.

Thanks,
Guillem

RFC: Switch default from netkit-telnet(d) to inetutils-telnet(d)

[Guillem Jover]

Hi!

There's been talk about switching away from netkit-telnet and
netkit-telnetd as the default implementations for some time now,
and replacing them with the ones from inetutils, which is a maintained
project and does see releases (even though with a long cadence).

This has been discussed somehow in #982253 and #987729.

These packages currently use a pair of virtual packages to denote
that they are a telnet or telnetd implementation (telnet-client and
telnet-server). One problem is that currently the netkit implementations
use the generic telnet and telnetd package names, which is a clear way
to mark them as the default implementations (instead of say the other
convention of naming or providing a default-foo package). Another is
that several packages depend on these generic names instead of the
virtual packages, see below for a list that would deserve a non-blocking
"mass" bug filing, which I can handle as part of the switch.

The inetutils-telnet recently got support for the missing «-b» option
for compatibility with netkit-telnet.

The inetutils-telnetd and netkit-telnetd have diverging options and some
conflicting ones, but after pondering about it I don't think this should
be a major issue, as the daemon does not tend to get called by users from
scripts and similar. For completeness the divergences are these:

   inetutils-telnetd   netkit-telnetd
   -   --
   short and long options  short options
  -D (unimplemented «exercise» mode)
   -D (debug modes «auth», «encr») -edebug
   -S, --server-principal  -S (used to set the IP TOS)
   -E, --exec-login-L
   -l, --linemode  
   -U, --reverse-lookup-N (related but not exactly the same)


Simon Josefsson (CCed), who is one of the inetutils upstream maintainers,
recently adopted the netkit-telnet source package in Debian, which he'd
prefer to keep around for a smoother transition period, in case users
want or need to revert back.



So, the idea would be for src:inetutils to take over the telnet and
telnetd binary packages and make them transitional packages depending
on the inetutils variants, for a smooth upgrade, and in addition also
start providing them by the inetutils- packages.

The src:netkit-telnet would then switch to ship netkit-telnet and
netkit-telnetd binary packages (this would ideally be uploaded to
experimental first, so that once ACCEPTED it can be uploaded to sid
once we start the switch, with no missing implementation in between).

I'm inclined to do it in this order to potentially avoid two trips over
NEW, and to reduce any potential disruption period.

In the future (after the next stable release) the telnet/telnetd
packages could be switched to be pure virtual packages, taking the role
of denoting the current default implementation, instead of introducing
default- variants, as that's what users are currently used to, and
it would keep working even if the depending packages below do not update
their dependencies.

We'd file an override request against ftp.debian.org to get the
inetutils-telnet Priority bumped to standard to match the current
telnet package (which could get then its Priority lowered to optional).

Currently inetutils and netkit have the same alternative priority
for telnet, I'd probably bump it also to 150 for inetutils to take
precedence.


If there are no objections, we could probably start working on this
switch in a couple of weeks or so.



List of packages depending on telnet (but not telnet-client):

  forensics-extra (Depends)
  lava (Depends)
  live-task-standard (Depends)
  mininet (Depends)
  vland (Depends)
  zssh (Depends)

  dish (Recommends against all current implementations)
  lava-dev (Recommends)
  lava-dispatcher (Recommends)
  live-task-extra (Recommends)
  pdudaemon (Recommends)

  libtelnet-dev (Suggests)
  libtelnet-utils (Suggests)
  procserv (Suggests)
  ser2net (Suggests)
  tucnak (Suggests)

List of packages depending on telnetd (but not telnet-server):

  telnetd-ssl (Conflicts)
  nyancat-server (Conflicts)


Thanks,
Guillem

Re: Bug#1013992: ITP: session-migration -- tool to migrate in user session settings

[Guillem Jover]

On Tue, 2022-06-28 at 21:51:44 -0400, Jeremy Bicha wrote:
> On Tue, Jun 28, 2022 at 8:07 PM Guillem Jover  wrote:
> > > Package: session-migration
> > > Description: Tool to migrate in user session settings
> > >  This tool is used to migrate in session user data when a program is 
> > > evolving
> > >  its configuration, or needing to have files moved and so on.
> > >  .
> > >  This program is generally autostarted at the very beginning of the 
> > > session
> > >  and integrates caching capability.
> >
> > This looks like an extremely generic name for such tool and package,
> > when it appears to be restricted to gsettings session data only?
> 
> It is not restricted to gsettings although gsettings is a good use case for 
> it.
> 
> Here's an example where it's used for something else:
> https://salsa.debian.org/gnome-team/gnome-boxes/-/commit/b536a968eb192
> 
> It would be nice if the upstream developers would handle user session
> migrations tasks themselves, but they often don't.

Ah thanks, looking into it, it seems more generic indeed.

> > > Package: dh-migrations
> > > Provides: dh-sequence-migrations
> > > Description: debhelper extension for session-migration support
> > >  This package provides a debhelper extension to perform session migration
> > >  operations on the installed packages.
> >
> > This also seems extremely generic. Migrations could refer to anything,
> > from databases, to any other data source. Something like
> > dh-gsettings-migrations seems like would be way better?
> 
> Despite being around for a decade, it looks like it's only used by
> about 6 current Ubuntu source packages so a rename is doable if
> needed. I think I wouldn't even need a transitional package since we'd
> rebuild all those Ubuntu packages which would get them the properly
> named dependency.
> 
> Here's a suggestion:
> user-session-migration
> dh-migrate-user-session Providing dh-sequence-migrate-user-session

Personally I'd perhaps try to keep both names consistent, also
the name you propose for the dh helper looks as if it would be
performing the migration itself which can be misleading, so perhaps
something like dh-user-session-migration would be better? In any case
I'd take either (or similar variants) over dh-migrations. :)

Thanks,
Guillem

Re: RFC: Additions to dpkg's Pre-Depends

[Guillem Jover]

On Wed, 2022-07-06 at 17:30:22 +0200, Guillem Jover wrote:
> > Unless I'm mistaken if dpkg dep on libcap(2) and libaudit, then dpkg
> > would transitively dep on both libcap(2) and libcap-ng which seems
> > suboptimal.
> 
> Yes, but its an internal implementation detail, so libaudit could be
> switched, and for now, at least they will not end up in the same address
> space. Of course pulling both is not ideal, but on Linux they are both
> already part of the pseudo-essential set anyway, and they are tiny
> compared to say the libzstd one, so…

I just noticed that libcap-ng and libaudit seem to share the same
upstream, so while I might be wrong, I'd expect there could be some
resistance to switch or add support for another POSIX capabilities
library.

If this was question where only one library can end up being used, I
think I'd still take libcap over libcap-ng *and* libaudit.

Thanks,
Guillem

Re: RFC: Additions to dpkg's Pre-Depends

[Guillem Jover]

On Wed, 2022-07-06 at 16:45:18 +0200, Andreas Henriksson wrote:
> On Wed, Jul 06, 2022 at 05:13:05AM +0200, Guillem Jover wrote:
> [...]
> > * libaudit-dev
> >   - Rationale:
> > This could allow to add Linux audit support to dpkg on package action
> > events. I've got a branch that might need minor polishing, but could
> > otherwise be merged.
> > 
> >   - Essential/Build-Essential:
> > On Linux it is already part of the pseudo-essential set.
> [...]
> > * libcap-dev
> >   - Rationale:
> > This could allow to add support to start-stop-daemon (already code
> > available) to drop POSIX capabilities. And also in the future (either
> > later in 1.21.x or 1.22.x) to support fsys POSIX capabilities as part
> > of the fyss metadata tracking support that is upcoming.
> > 
> >   - Essential/Build-Essential:
> > On Linux it is already part of the pseudo-essential set.
> [...]

> IIRC there are several competing capability libraries already part of
> pseudo essential. I think it would be good if we could identify one and
> work towards having only that in pseudo essential (and in theory maybe
> the entire archive use it). Maybe it's out of scope for this discussion
> though...

Ah, sorry, should have included that as part of the rationale too.

When I looked into this, I checked both libraries, their size, the
amount of users and their APIs. And my conclusion was that libcap-dev
looked like a better choice, even if it might be supposedly harder to
use (which I found it to be the reverse, given it has an explicit API,
instead of one based on hidden global state), but then libcap-dev API
could always be improved or a new higher-level layer added on top.

  * libcap-ng-dev
- Installed size 66 KiB.
- 27 source packages Build-Depend* on it.
- Uses (thread-local) hidden global state, when dealing with files
  f.ex, which makes this unintuitive and rather unattractive when
  being used from a library (eventually this would end up in
  libdpkg.so), after that I didn't look further into this library.
  I think ideally this this library could have been either built on
  top of libcap-dev, or any more higher-level interfaces added there.

  * libcap-dev
- Installed size 65 KiB.
- 121 source packages Build-Depend* on it.
- The API looks less constrained and better to use from another
  library.

(FWIW, I also think unifying on a single library distro-wide would be a
nice goal, though.)

> Unless I'm mistaken if dpkg dep on libcap(2) and libaudit, then dpkg
> would transitively dep on both libcap(2) and libcap-ng which seems
> suboptimal.

Yes, but its an internal implementation detail, so libaudit could be
switched, and for now, at least they will not end up in the same address
space. Of course pulling both is not ideal, but on Linux they are both
already part of the pseudo-essential set anyway, and they are tiny
compared to say the libzstd one, so…

Thanks,
Guillem

RFC: Additions to dpkg's Pre-Depends

[Guillem Jover]

Hi!

As per Debian policy §3.5, and given dpkg “Essential: yes” nature, I'm
bringing up the following potential additions to dpkg's Pre-Depends,
and whether there is consensus about each of them individually and
independently.

* libmd-dev
  - Rationale:
src:dpkg currently has its embedded MD5 implementation. On non-Debian
(and derivatives) it will default to use the message digest function
from either the system libmd or libc. I'd eventually like to be able
to remove the embedded code, which is there mostly for Debian. And to
be able to rely on SHA variants, for fsys metadata tracking and
similar (and not have to embed those too). I'd rather not add
support nor link against a crypto library like libgcrypt which might
already be present on the pseudo-essential set (currently, but could
go away easily with the OpenSSL license change), as these functions
are currently used for data integrity rather than for security, and
would/could get broken with stuff like FIPS enabled.

  - Essential/Build-Essential:
AFAIR it is already part of the cross bootstrap set? But not of the
pseudo-build-essential or pseudo-essential sets. Most systems have
it installed via libbsd. On minimal systems it would increase the
installed size by around 80 KiB.

  - Disclosure:
I maintain this in Debian and I'm upstream.

* libaudit-dev
  - Rationale:
This could allow to add Linux audit support to dpkg on package action
events. I've got a branch that might need minor polishing, but could
otherwise be merged.

  - Essential/Build-Essential:
On Linux it is already part of the pseudo-essential set.

* libacl-dev
  - Rationale:
This could allow in the future (either later in 1.21.x or 1.22.x) to
support ACLs as part of the fsys metadata tracking support that is
upcoming.

  - Essential/Build-Essential:
It is already part of the pseudo-essential set.

  - Disclosure:
I maintain this in Debian.

* libcap-dev
  - Rationale:
This could allow to add support to start-stop-daemon (already code
available) to drop POSIX capabilities. And also in the future (either
later in 1.21.x or 1.22.x) to support fsys POSIX capabilities as part
of the fyss metadata tracking support that is upcoming.

  - Essential/Build-Essential:
On Linux it is already part of the pseudo-essential set.

* libzstd-dev
  - Rationale:
This could allow to add zstd support for .debs via the library. This
is mostly to leave the door open to that possibility, as I'm still
pondering whether to perhaps add the support in Debian via the CLI
and just add those via Recommends or similar. Also there are still
concerns about the library and I have to note I'm rather unhappy
with how Ubuntu pushed this, and diverged the .deb ecosystem,
forcing upstream's hand here. :(
More details at .

  - Essential/Build-Essential:
On Linux it is already part of the pseudo-essential set.

Thanks,
Guillem

Re: Bug#1008304: ITP: http-nio -- http/s file system provider for Java NIO.2

[Guillem Jover]

Hi!

On Sat, 2022-03-26 at 14:59:17 +0100, Pierre Gruet wrote:
> Package: wnpp
> Severity: wishlist
> Owner: Debian-med team 
> X-Debbugs-Cc: debian-devel@lists.debian.org, debian-...@lists.debian.org
> 
> * Package name: http-nio
>   Version : 0.1.0
>   Upstream Author : Daniel Gomez-Sanchez
> * URL : https://github.com/lbergelson/http-nio
> * License : BSD-3-Clause
>   Programming Lang: Java
>   Description : http/s file system provider for Java NIO.2
> 
> This package provides a http or https file system that can be used in
> conjunction with Java NIO.2. This lightweight library provides a few classes
> related to file systems and paths.
> 
> It is provided by the developers of gatk, which is a long-term packaging 
> target
> of Debian-med team. I am packaging it as a reverse dependency of gatk.
> For this reason, it will be team-maintained inside Debian-med team.

The http-nio name seems rather generic to be used as either source or
binary package name. Could you namespace it? I don't see a very clear
naming convention in the archive for Java packages, but then I have not
checked the java-policy, TBH. What I see is either -jave or
lib-java, either would work here.

Thanks,
Guillem

Re: Epoch bump for gnome-shell-extension-autohidetopbar

[Guillem Jover]

Hi!

On Mon, 2022-03-28 at 13:21:03 +0200, Tobias Frost wrote:
> gnome-shell-extension-autohidetopbar is currently packaged with a date/name 
> version, eg.
> 20209, unfortunatly without a leading 0~...
> 
> I recently agreed [1] with upstream to tag their releases to
> match the version as on extensions.gnome.org as this has been source of
> confusion for the users of the extension and package*. The versions on
> extensions.gnome.org is currently 107.
> 
> Therefor I'd like to bump the epoch for the pacakge to 1:
> with the next upload.

Yes, this looks like what epochs were intended for.

Thanks,
Guillem

Re: proposed MBF: packages still using source format 1.0

[Guillem Jover]

Hi!

On Tue, 2022-03-15 at 15:36:48 +, Ian Jackson wrote:
> However, given that I perceive that:
>   - there is a campaign to abolish 1.0
>   - there are important use cases where 1.0 is needed
>   - the campaign to abolish 1.0 is being prosecuted anyway
> I have deliberately chosen to continue to upload even pure-native
> packages as 1.0, to try to slow this process down in the hope that the
> situation improves and/or people stop trying to forbid useful things.

Ian, just to try to give some kind of reassurance. I'm not involved
with this MBF in any way (neither suggested nor prompted it, etc.).
While I'd like to see the practice of mismatched format vs version
disappear, I also neither did nor have immediate plans to make that
an error for 1.0 formats, as long as there are users at least in
Debian, or there are no viable alternatives, because I consider 1.0
confusing enough and not salvageable, that it makes less of a
difference there. If there ever comes the time where it's feasible to
error out on 1.0 on those conditions, I'd even be fine with a force
option but only for 1.0 formats, because I really want to preserve
2.0+ formats as clear as possible, and improve on those. I also have
no plans to abolish building source format 1.0 in dpkg.

Something I might want to see though (although I hold not much hope
for) is a possible move away from the default behavior when no
debian/source/format is present, as I think that gives bad defaults
for newcomers or inexperienced users, and even there just emitting
warnings tend to be ignored. Possible alternatives could be, either
erroring out, or changing the default format depending on say a
dpkg-compat level, or similar, I don't know, have not thought this
through though. But explicitly marking sources as 1.0 (as has been
warned for a long time now) would of course keep working as of right
now.

HTH,
Guillem

Re: proposed MBF: packages still using source format 1.0

[Guillem Jover]

On Thu, 2022-03-10 at 12:09:14 -0700, Sam Hartman wrote:
> You're trying to produce packages from CI builds or other automation
> where you sometimes have native Debian revisions.
> 
> * you are producing a package where you have distinct upstream and
>   debian branches, and you cannot control  the upstream version number.
>   You want to do everything in git, and not have to deal with quilt
>   patches.
>   Say you don't even have any patches, but you sometimes do  produce new
>   revisions simply for changes to debian control files and the like.

As I mentioned last time around, it has never been made clear why a
different “revision”-separator character cannot be used here. (Perhaps
out of doctrine? :)

> * You are trying to local (or downstream) builds of debian packages that
>   do have debian revision numbers.
>   You want to do everything in git because honestly dealing with dscs is
>   a PITA and if you can avoid it you want to.
> You need to produce dscs to feed to sbuild, or mini-buildd or something.
> But you just want to do that easily from your git CI pipelines.
>
> My frustrations with the number of hours I've lost because of this
> doctrinal issue has caused me to come close to giving up on Debian more
> than once.
> Part of that is frustration around how the change was handled and how
> concerns and use cases where not considered and dismissed without
> consideration.
> But part of that is how I've had to hack around the isue in every
> downstream CI environment where I took Debian packages and modified
> them.

In this second scenario, you seem to be using .dsc in anger, when you
don't really want to have to be using them at all. And when doing that
you seem to have decided that using them in a way that makes our packaging
stack more confusing, incoherent and error-prone is better, than say
trying to get those other tools to avoid using the .dsc format at all.


The other thing, is that people keep confusing the transport source
format (.dsc, etc.), with the unpacked source. Of course it is not
helped that dpkg-source seems to be conflating these, but this is not
really the case. I've for example been pondering adding an extract
mode where instead of generating a quilt .pc/ compatible directory it
would instead generate a git repo. There's also the «3.0 (custom)»
format, which exemplifies this detachment.

But in any case, in your second scenario, if you are doing CI builds,
and only want to be working with git, you could simply do something
like this from your git checkout:

  $ dpkg-source --format="3.0 (git)" --build source-dir-4.0/

regardless of the declared source format in the unpacked tree
(including 1.0, 2.0 or any 3.0 format). For binary builds you could
do:

  $ dpkg-buildpackage --source-option=--format="3.0 (git)" -us -uc

Regards,
Guillem

Re: proposed MBF: packages still using source format 1.0

[Guillem Jover]

Hi!

[ But, this one again… ]

On Thu, 2022-03-10 at 18:17:15 +, Steve McIntyre wrote:
> Why on earth *would* you mess around using Debian revisions on a
> native-format package, though? IMHO it's pointless and is just going
> to confuse people. Unless you can explain a good reason to need this,
> I'd argue strongly that the 3.0 native support is DTRT for the
> principle of least surprise if nothing else!

Yes. People complain about the Debian packaging toolchain being too
complex or too confusing. This is one of such cases. As has been stated
countless times, this subverts the semantics of both the source and
version formats. At least we only have one case remaining, the even
more senseless 1.0 non-native source with native version was turned
into an error with dpkg 1.20.1. Recall that dpkg-source currently needs
to use heuristics to decide whether to use an orig tarball + diff or not
for format 1.0. :(

We currently have 21 source packages in the archive with format 1.0
native source and non-native version:

  $ grep-deb-sources -sPackage -FFormat '1.0' -a \
 -FVersion '-' -a --not -FFiles '.diff.'

My impression is that _most_ of those are packaging mistakes. I just
filed two new bugs on packages that have recently been (apparently)
accidentally uploaded with such broken source tarballs (vde2: #1007087;
etbemon: #1007088).

  

  

Then we are left with at most a literal (vocal) handful of maintainers
potentially doing this on purpose. And the main reason we cannot have
coherent, and understandable semantics here.

I've also asked before why this attachment to subvert the source
format, when the version could be modified to use some other character
instead for the “revision”, such as ‘+’, but I don't recall ever
getting any answer (not even a satisfactory one) to that.

Thanks,
Guillem

DKIM and Exim (was Re: Gmail bounce unauthenticated @debian.org addresses)

[Guillem Jover]

Hi!

On Fri, 2022-03-04 at 14:36:01 +, Colin Watson wrote:
> I reproduced a similar problem, then set up DKIM for myself and
> everything then worked, so I think you're correct.
> 
> The links in the original d-d-a email were mostly stale, but I found
> https://bynicolas.com/server/exim-multi-domain-dkim-custom-selector/
> helpful in getting this going with my local Exim setup.

You might want to also fix the DKIM_SIGN_HEADERS macro in the Exim
config, as its default is currently broken (see #939808). The patch
attached there is not helpful for local usage, so you might want
something like what I've got in my config:

,--- exim4.conf ---
[…]

# The default headers to sign is broken, and includes things that should
# not be signed by default if they are missing, or they will break mailing
# lists.
DKIM_SIGN_HEADERS = \
  From:From:Reply-To:Subject:Subject:Date:Message-ID:To:Cc:MIME-Version:\
  Content-Type:Content-Transfer-Encoding:Content-ID:Content-Description:\
  In-Reply-To:References:X-Debbugs-Cc:\
  =Sender:\
  =Resent-Date:=Resent-From:=Resent-Sender:=Resent-To:=Resent-Cc:\
  =Resent-Message-ID:\
  =List-Id:=List-Help:=List-Unsubscribe:=List-Subscribe:=List-Post:\
  =List-Owner:=List-Archive

[…]
`---

Thanks,
Guillem

deb-porterbox (was: Re: No mips64el porterbox?)

[Guillem Jover]

Hi!

On Tue, 2022-03-01 at 10:28:28 +0100, Julien Puydt wrote:
> one of my package has a failure on mips64el and upstream is ready to
> help me find the cause and debug the issue.
> 
> Unfortunately, on https://db.debian.org/machines.cgi I only see five
> developer machines on this architecture -- all buildd!
> 
> Is there really no mips64el porterbox, or is it only a transitory
> situation?

As others have mentioned there's one such box. You can use the
following script to get the available porterbox for any arch:

  

I'll check whether I did submit that to something like devscripts,
which I cannot recall.

Thanks,
Guillem

Re: The future of src:ntp

[Guillem Jover]

On Mon, 2022-01-24 at 08:08:16 +0100, Guillem Jover wrote:
> On Sun, 2022-01-23 at 15:12:49 -0600, Richard Laager wrote:
> > On 1/19/22 15:04, Bernhard Schmidt wrote:
> > > On 19.01.22 20:34, Richard Laager wrote:
> > > > 2. I create transitional binary packages in src:ntpsec:
> 
> > I got to thinking about this more. This won't work, because src:ntp is
> > 1:4.2.8p15+dfsg-1 and src:ntpsec is 1.2.1+dfsg1-2. I would need an epoch (of
> > 2, since ntp already has an epoch of 1) on ntpsec in order for src:ntpsec's
> > transitional bin:ntp package to be newer than src:ntp's bin:ntp package.
> 
> Bumping the epoch to 2 here is completely gratuitous and can make a mess
> for ntpsec *and* potentially ntp iff upstream got to be improved and we
> wanted to reintroduce it in the future.
> 
> > It might be technically possible to build a binary package with different
> > versioning, but even if it is: 1) I don't know how, and 2) I'm not sure
> > whether that's a good idea, especially compared to the alternatives.
> 
> Yes, this is the recommended method, that has been used in the past,
> and which is mentioned in the dpkg FAQ. You can set arbitrary versions
> via dpkg-gencontrol (or indirectly via dh_gencontrol).

To clarify, which seems I had included initially but removed while
editing. If generating the transition packages from ntpsec, then you
can set the binary versions, for example, to be something like:

  «1:$(ntp_upstream_version)+$(ntpsec_binary_full_version)»

so you'd end up with something like:

  1:4.2.8p15+dfsg+1.2.1+dfsg1-2
  1:4.2.8p15+dfsg+1.2.1+dfsg1-3
  1:4.2.8p15+dfsg+1.2.2+dfsg1-1
  …

and while ugly, it serves its intended purpose quite well w/o messing
with epochs and potentially muddling any package's future.

Thanks,
Guillem

Re: The future of src:ntp

[Guillem Jover]

On Sun, 2022-01-23 at 15:12:49 -0600, Richard Laager wrote:
> On 1/19/22 15:04, Bernhard Schmidt wrote:
> > On 19.01.22 20:34, Richard Laager wrote:
> > > 2. I create transitional binary packages in src:ntpsec:

> I got to thinking about this more. This won't work, because src:ntp is
> 1:4.2.8p15+dfsg-1 and src:ntpsec is 1.2.1+dfsg1-2. I would need an epoch (of
> 2, since ntp already has an epoch of 1) on ntpsec in order for src:ntpsec's
> transitional bin:ntp package to be newer than src:ntp's bin:ntp package.

Bumping the epoch to 2 here is completely gratuitous and can make a mess
for ntpsec *and* potentially ntp iff upstream got to be improved and we
wanted to reintroduce it in the future.

> It might be technically possible to build a binary package with different
> versioning, but even if it is: 1) I don't know how, and 2) I'm not sure
> whether that's a good idea, especially compared to the alternatives.

Yes, this is the recommended method, that has been used in the past,
and which is mentioned in the dpkg FAQ. You can set arbitrary versions
via dpkg-gencontrol (or indirectly via dh_gencontrol).

Thanks,
Guillem

Bug#1002056: ITP: zlib-ng -- optimized zlib compression library

[Guillem Jover]

Package: wnpp
Severity: wishlist
Owner: Guillem Jover 
X-Debbugs-Cc: debian-devel@lists.debian.org, Mark Brown 

* Package name: zlib-ng
  Version : 2.0.5
  Upstream Author : zlib-ng Team
* URL : http://github.com/zlib-ng/zlib-ng
* License : Zlib, Zlib-RFC, CC-BY-3.0, CC-BY-4.0, Public-Domain
  Programming Lang: C
  Description : optimized zlib compression library

 zlib-ng is a fork of the zlib library implementing the deflate compression
 method found in gzip and PKZIP.
 .
 It includes and consolidates many optimizations found in alternative forks,
 that have not been merged in the official zlib library.


I just discovered this project, and started packaging [P] it to be able
to play with a dpkg branch switching its zlib support to zlib-ng. The
speed up is quite significant, for example when packing the 0ad-data,
on my system it takes 5m50s~ with zlib and 4m30s~ with zlib-ng.

  [P] https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/

The project has a compat mode which generates API "compatible" zlib
replacement libraries, but unfortunately it is stated not to guarantee
to be ABI compatible, so no compat packages or similar could be
produced right now as that could potentially break other packages. I've
filed a report [S] upstream to request a more usable shim instead.

  [S] <https://github.com/zlib-ng/zlib-ng/issues/1081>


I'm still pondering whether to upload this, although the packing is
already done, but w/o the above mentioned shim its utility seems
restricted as most upstream projects use it via its compat mode,
instead of with its native API. But if that happens, I think it would
make sense to upload, as it's currently being embedded in several
upstream projects and even if dpkg would not switch to it, it would
still help with removing embedded code copies, and speeding up other
packages. Or make other RPFs such as #901490 (an alternative fork)
unnecessary.

Thanks,
Guillem

Re: Unmet :native dependency error when cross-compiling with dpkg-buildpackage

[Guillem Jover]

Hi!

On Wed, 2021-12-08 at 15:26:10 +0300, Uladzimir Bely wrote:
> The conditions are the following:
> Build Architecture: amd64
> Host Architecture: arm64

> The package I'm trying to cross-build is 'u-boot' with some patches. It has 
> build-time dependency 'swig:native'. But swig from Debian repos doesn't fit 
> all needs, so it's also compiled from sources with some patches applied. 
> Than, 
> local Debian repository is created (using reprepro), where 'swig:amd64' is 
> available.

> Later, 'u-boot' itself is being compiled using sbuild. During "Install 
> package 
> build dependencies" stage, 'swig:native' is correctly resolved to 
> 'swig:amd64'. All deps are correctly installed, then. But later, when it 
> comes 
> to dpkg-buildpackage stage, it fails with the following error:
> > Command: dpkg-buildpackage --sanitize-env -aarm64 -Pcross,nocheck -us -uc 
> > -b 
> -rfakeroot
> > "dpkg-checkbuilddeps: error: Unmet build dependencies: swig:native:

The swig package is arch:all m-a:foreign, which is disallowed as the
target of a :native arch-qualified dependency, and thus the dependency
is not satisfiable. See:

  
https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/tree/doc/multiarch.txt?h=pu/doc-multiarch-spec

BTW, you might want to rise this kind of question in the
debian-cross@l.d.o mailing list.

Thanks,
Guillem

Re: devscripts: wrap-and-sort should default to -ast

[Guillem Jover]

On Wed, 2021-02-10 at 11:47:12 +, Phil Morrell wrote:
> As a user of -satb, I'd like to point out that the flags are not all
> equal. Two of them support a (more objective?) desire that "addition to
> a list in line-based VCS should have no deletions". That is -at, whereas
> -s is a subjective prettifier and -b could remove information, hence -k.

To me, all the flags in -ast are objectively superior to their
counterparts. In the case of -s even more so, in addition to what has
been mentioned of avoiding reindenting on field renames which does not
necessarily mean the official field name gets changed (which is indeed
very unusual), but the field being renamed in debian/control, say from
Recommends to Suggests; it is also superior because:

 1) it avoids VCS noise, by not putting the first value on the first
line as the field itself,

Depends: foo,
 bar,

instead of:

Depends:
 foo,
 bar,

 2) it allows trivial copy or syncs, between say build-dependencies
and run-time dependencies. Say you have this:

Source: foo
Build-Depends: some-dep-dev,
   other-dep-dev,

Package: foo-dev
Depends: some-dep-dev,
 other-dep-dev,

which means you have to copy, possibly remove the field, then
reindent, which is rather annoying.

I agree that -b is something else entirely, and in many cases I prefer
ordering depending on relationship, layers or importance of the
packages involved.

> To make that work as a default, there would need to be something like an
> --short-preferred-unless-existing-indent to prevent constant
> reformatting. I disagree with jonas on the importance of -s because I'm
> not convinced that field names change, especially not often.

See above. But, I'd actually say it's perhaps as important or more
than -t! :)

Thanks,
Guillem

Re: Epoch bump for golang-github-valyala-fasthttp

[Guillem Jover]

On Fri, 2021-11-12 at 10:55:26 +0530, Pirate Praveen wrote:
> On 12 November 2021 12:38:23 am IST, Guillem Jover  wrote:
> >The golang-github-valyala-fasthttp package used to have date-based
> >release numbers (current Debian version 20160617-2). Upstream has
> >since switched to semver (latest upstream version 1.31.0).
> >
> >So the version scheme has been reset, and unfortunately given that no
> >prefix was used when initially packaging this, an epoch seems to be in
> >order now.
> >
> >I'm planning on updating in the coming days to the latest upstream
> >release and bump the version using an epoch.
> 
> How about golang-github-valyala-fasthttp-v1 ?
> Though it won't match import path, it can avoid the epoch.

While interesting, I think this might be worse. I'm not a fan of
epochs, but this is precisely the case they were intended for. The way
I see it, the source package name is now burned, and even if we played
games and used a different source+binary name w/o epoch, but breaking
the current convention and expectation of package names trying to map
closely to import paths, the current source+binary one could/should not
be reused anyway, so we might as well bump the epoch there, and if in
the future there's a «.v1» kind of import path bump, then we can simply
drop the current one and completely get rid of the epoch.

> Most projects change import paths on incompatible bumps.

But this didn't happen upstream in this case.

Thanks,
Guillem

Re: merged /usr vs. symlink farms

[Guillem Jover]

unblock 848622 by 134758
thanks

On Sun, 2021-08-22 at 11:21:38 +0100, Luca Boccassi wrote:
> On Sat, 2021-08-21 at 22:57 +0200, Guillem Jover wrote:
> > On Sat, 2021-08-21 at 18:47:50 +0100, Luca Boccassi wrote:
> > > The bug is real, nobody doubts that - it has been filed on dpkg 20
> > > years ago.
> > 
> > You keep repeating this, but I have no idea what bug you refer to.
> > 
> > There's #148258 (from 2002), which is conffile related, and not
> > actionable and should probably just be closed.
> > 
> > There's #182747 (from 2003), which while apparently similar is
> > something else completely. This is about the (IMO) misfeature of
> > supporting a local admin to redirect (not alias) a directory using a
> > local symlink (mainly for space management reasons). For an
> > explanation
> > see <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779060#10>.
> > 
> > There's #406715 (from 2007) which is related to the above misfeature.
> 
> I am referring to #134758 since it's linked as the root cause from
> usrmerge's #848622.

Well, that's a bogus block then, because that's obviously not the root
cause. I see I was CCed when that block was set, I guess I missed it.
:/ Fixed it now…

> "dpkg-query: Make -S handle unowned symlinks resolving to owned
> pathnames" filed in February 2002 - 19 years and a half ago. I refer to
> that because it's linked as the root cause in the BTS of the relevant
> issue with usrmerge we are discussing.

Even if the wishlist from that report got implemented, it would still
not fully solve all the problems, where among them «dpkg-query -S»
is probably the lesser one, which would not work in the other direction
anyway (querying a path under /usr/ known to dpkg as being under /).

And then I'm not convinced this should even be implemented at all,
as it would introduce behavior differences between literal pathnames
and patterns, and making them slower (for the first case) or potentially
extremely slower (for the second case), in addition to making the queries
dependent on the on-disk layout (so unreliable from the packaging PoV,
as it would invent on the spot, pathnames not truly coming from any
package nor otherwise known to dpkg).

This for a misfeature in dpkg (supporting redirecting symlinks) that
allowed the current mess anyway. So I'm inclined to wontfix and close
that one.


Not looking forward to further interactions…
Guillem

Epoch bump for golang-github-valyala-fasthttp

[Guillem Jover]

Hi!

The golang-github-valyala-fasthttp package used to have date-based
release numbers (current Debian version 20160617-2). Upstream has
since switched to semver (latest upstream version 1.31.0).

So the version scheme has been reset, and unfortunately given that no
prefix was used when initially packaging this, an epoch seems to be in
order now.

I'm planning on updating in the coming days to the latest upstream
release and bump the version using an epoch.

Thanks,
Guillem

Re: merged /usr vs. symlink farms

[Guillem Jover]

Hi!

On Sun, 2021-08-22 at 09:18:25 +0200, Andreas Metzler wrote:
> Afaict we have still no idea on how to move on.
> 
> 1 I think you agree that there is a significant number of usrmerged Debian
>   installations out there. It does not really matter whether there are 7% or
>   40%. They exist and should (keep) working.

By my definition, these have never been working correctly, but
semantics I guess. My wish would be to indeed salvage those systems,
that's why I implemented dpkg-fsys-usrunmess. But if people refuse,
then at some point I'd personally consider them lost, TBH. :/

> 2 As you have stated there are known issues with dpkg and usrmerged
>   systems. Some of them are are triggered by moving files from / to /usr.
> 
> Starting from here it is hard to see a way forward.

Well, in my mind the first and most immediate action that would be
done, is to stop the bleeding, by:

  - reverting the changes in deboostrap in sid, bullseye (and ideally
in buster too),
  - reverting the notion that split-/usr is unsupported (which includes
the extremely confusing interpretation about this applying to
sid/testing too), and update documentation such as release-notes,
etc.,
  - adding a Conflicts against usrmerge somewhere (dpkg, base-files or
similar) for extra caution,
  - (optionally) removing usrmerge from buster, bullseye and sid,

I'd be ready to prepare patches or file reports for any of these.

But then, my expectations regarding Debian have plummeted so…

> Is it a realistic option to require something like
> -
> dpkg --purge usrmerge
> dpkg-fsys-usrunmess
> -
> pre dist-upgrade to bookworm to get back all systems to a sane (from
> dpkg POV) state and start fresh?

If by requiring you mean documenting the requirement, then I don't
think that would be enough.

But I could see introducing a preinst somewhere (perhaps base-files,
to permit upgrading dpkg itself) which would check for those unsupported
systems and emit a big fat message explaining the situation and
instructions to follow to be able to proceed, then abort. Similar in vein
with how other preventions (AFAIR) have been implemented for example on
glibc or udev for environment, multi-arch or kernel requirements.

Sadly, even introducing that into bullseye would not guarantee that the
bookworm upgrade would already be in a sane state, as there's no
guarantee people do not skip point releases. But I assume it would cover
most of the installations. So the problem would be greatly minimized.
Only after the bookworm upgrade has been completed this would be
guaranteed.

> Is this robust (except for crash as stated in the manpage), or would
> you consider it beta-quality?

There's a known problematic interaction with a systemd's emergency mode
misbehavior/bug, which I've worked around, plus some other robustness
improvements I've prepared that will be included in 1.21.0, and targeted
for 1.20.10. I should probably also add a temporary policy-rc.d to
batch and defer all service actions for after the mass reconfiguration,
which I'll try to code up soonish to further improve robustness.

Otherwise I've taken great care and consideration on trying to make it
as robust as possible, but I guess out of cautiousness and given the
nature of what the script is doing (which is still a hack, although a
self-contained one at least), I'd be hard pressed to probably ever
declare it 100% safe. :/ Of course more testing is always helpful!

Thanks,
Guillem

Re: merged /usr vs. symlink farms

[Guillem Jover]

On Wed, 2021-08-25 at 09:57:09 -0700, Russ Allbery wrote:
> Wouter Verhelst  writes:
> > The problem here is also that if there are two packages like that, on an
> > usrmerge system, we would not know this is happening.

Also this does not need to come from "buggy" packaging practices.

> I agree, of course, but I don't see a way in which Policy can help with
> that problem unless this packaging decision was intentional and the person
> who made that decision would have chosen otherwise if Policy had said to
> not do it.

> This seems more like an appropriate check for an archive-wide QA tool
> looking for cross-package problems.

I've said this many times over the months (years!?), that while this
can easily affect stuff from the archive, which we do control, where
policy applies and where we could try to poorly reimplement the checks
that dpkg does to detect them in some QA checker, even though the
following problems would still apply:

  - the checks cannot be performed (only) over static snapshots of
the archive, as this would affect partial upgrades too,
  - the checks would need to take into account packages we have
stopped shipping way in the past as those can linger around
installed,
  - the checks would have a hard time with the non-declarative side
of the packaging stack,

is that something else I expect to have a non-zero chance to bite users
are all those common practices that people give for granted and that we
have supposedly supported in the past, as guaranteed by our packaging
system, such as:

  - keeping installed packages that have stopped shipping in Debian,
  - installation of packages from third-parties, or from local
overlays or similar, or even rebuilt forks of packages from Debian,
  - installation or holding of Debian packages from older releases,
  - local diversions or alternatives,

which are out of our QA reach.


The fact that the supporters of a *filesystem layout* have been happy
to dismiss and ignore this and have been pushing for what I think can
be easily described as the worst ever "transition" done in Debian, very
sadly, for me this whole topic marks a before and after in Debian, and
has put my trust in the technical side of the project into question.

Of course some of those supporters are now agreeing these problems can
be insidious, progress I guess. And at the same time others are claiming
that acknowledging those problems would hold back the entire distribution,
while others are now saying we should stop doing packaging changes because,
well, these problems perhaps are potentially problematic, the irony.

And then we get people raving over what's the worst and most atrocious
hacks to pile on into dpkg to try to workaround the actual root cause
(turtles^Wnasty hacks and kludges all the way down I guess). Which I
obviously expect to eventually be coerced into merging into dpkg at
some point or another via our esteemed Authority.

From where I'm sitting Debian is the project that lost its way…

Sadly,
Guillem

Re: merged /usr vs. symlink farms

[Guillem Jover]

On Fri, 2021-08-20 at 07:56:33 -0600, Sam Hartman wrote:
> > "Theodore" == Theodore Ts'o  writes:
> Theodore> FWIW, from following the discussion, I've become more and
> Theodore> more convinced that a symlink farm is *not* the right
> Theodore> answer, regardless of whether it is done centrally or via
> Theodore> individual packages moving files and created symlinks if
> Theodore> necessary in individual maintainer scripts.

There was talk about the huge amount of symlinks required in a symlink
farm setting, but that might have been true for a scenario where those
symlinks would have been handled automatically and transparently.

The huge majority of files under /lib* (which is the actual bulk of them)
should require no symlink farms. Many of the ones under /bin and /sbin
(we are talking about around 240 packages here) might be switchable w/o
compat symlinks after careful consideration (or not…), many of the ones
that require symlinks would need these just for a period of time, and
only a handful would remain (the ones that are part of standard
interfaces, which I'd expect would be mostly shells?). I think the amount
of symlinks currently provided by f.ex. lvm2 and e2fsprogs combined would
amount to more symlinks than what we would eventually end up with TBH.

> Ted, in terms of judging consensus, I'd like to explicitly ACK your
> summary.
> I think that you have accurately described where we are in the
> discussion.
> 
> As you know, one of the ways we can see  how close we are on consensus
> is to look at what happens when someone proposes a summary like you did.
> 
> Simon Richter tried to challenge your summary effectively saying that we
> couldn't have an informed consensus because there were open technical
> issues that had not been addressed.  This was roundly rejected by
> yourself, Philip Hands and Luca Boccassi.

Well then that's a very underwhelming way to judge consensus TBH.

Philip remarked on the falling-sky theme (which I covered in a reply to
Luca elsewhere), and then used a crystal ball to predict the future,
where a particular filesystem layout would pass the test of time in
contrast to dpkg, which apparently (obviously a bit biased here) is such
a trivial detail in our distribution that it can be easily replaced w/o
requiring to scrap everything and starting a new distribution from
scratch.

Luca posted a mess of disinformation.

Ted's reply I've partially covered at the top of this mail. The rest
I've covered in the aforementioned reply to Luca. And some of the
remaining parts I've heard the "dpkg team" think are wild conjectures
on motivations and similar.

Or there's the following for example:

> IN particular, most systems are usrmerged today, and while these bugs
> are annoying, many people get along just fine.

That would imply that most people have either gone out of their way to
explicitly install and run usrmerge or they have reinstalled from
scratch. I cannot believe the first to be of any significance, the
second would put into question why we bother supporting release upgrades
(after all many other distros do not support that, we perhaps should
catch up with what the rest of the world is doing!).


I've also tried to stop replying to all misconceptions, inaccuracies
and plain wrong stuff on this thread (f.ex. people vocal in this thread
apparently do not exactly seem to know how upgrades or our package
manager works in Debian?), first to avoid dominating it with replies
(like some kind of deranged person, where I think I've probably already
surpassed my quota), and because it just all seems like a monumental
waste of time, energy and motivation. Which I'd rather spend elsewhere.



I'm personally just not seeing such consensus, despite the attempts of
some to make it pass as so. My perception is that this topic has become
such a black hole of despair, that people that take issue with it, are
simply stepping away.

Exhausted,
Guillem

Re: merged /usr vs. symlink farms

[Guillem Jover]

On Sat, 2021-08-21 at 18:47:50 +0100, Luca Boccassi wrote:
> My recollection (which might be wrong, but a quick look at release
> notes seems to support it with 11.04 having multiarch 2 years before
> Wheezy) is that Canonical led the way with the multiarch effort in
> Ubuntu, and Debian followed with lots of huffing, puffing and
> grumbling.

You mean that time when Ubuntu merged an implementation based on a broken
design with broken interfaces, that was causing dpkg database damage, where
the tech-ctte also tried to force through, in all its wisdom? Right, great
example. (And let's ignore release cadences for more spectacular effect.)

  

But just wow, such a mischaracterization and deformation of the events.
Sadly, at this point I'm not surprised. This goes along comments such as
that the intersection of packages not using debhelper and shipping
split-/usr files are in the "thousands" (way less than the actual number
of packages shipping those pathnames), or the ones in the paragraphs
below about that mythical bug report, and the single failed attempt to
symlink farm, etc, etc…

> On Sat, 2021-08-21 at 16:20 +0200, Wouter Verhelst wrote:
> > I'm not saying the solution which the dpkg maintainers are proposing
> > is the only valid solution, but if you go and tell them "ah the real
> > problems you point out are irrelevant" then You! Are! Doing! It!
> > Wrong!
> 
> Again, if the magnitude of this dpkg bug was really that serious there
> would be visible consequences after almost 3 years of deployments
> across two distributions with who knows how many million instances, and
> yet "having to run dpkg -S again" is all we can see. Where are the bug
> reports? Where are the enraged users with unusable broken system and
> lost data? Where are the reports of Canonical going out of business
> because Ubuntu is unusable?

Just like no one had detected the database corruption in Ubuntu before
I spotted the problem via code review and analysis (which I guess in
your world translates to "opinion"). I'd expect the problems with
aliased directories to be that kind of insidious issue that people
have a very hard time trying to pin point, and which will be getting
worse as time passes.

> The bug is real, nobody doubts that - it has been filed on dpkg 20
> years ago.

You keep repeating this, but I have no idea what bug you refer to.

There's #148258 (from 2002), which is conffile related, and not
actionable and should probably just be closed.

There's #182747 (from 2003), which while apparently similar is
something else completely. This is about the (IMO) misfeature of
supporting a local admin to redirect (not alias) a directory using a
local symlink (mainly for space management reasons). For an explanation
see .

There's #406715 (from 2007) which is related to the above misfeature.

> What I am taking issues with is the representation of its
> actual, real effects, and thus its severity and the consequences for
> the project. There are a lot of words being spent on how terrible and
> broken and unacceptable the status quo is, and yet not a single link
> to a bug report.

What I'm appalled at is the sloppiness and dogma shown in the name of
a filesystem layout that will have very minimal benefit for final users
(in contrast to some use cases for some admins or installations that
should already know what they are doing, and can manage all potential
downsides in a controlled way through a hack like usrmerge), knowingly
in detriment of robustness and stability.

> By all means, go and fix it, make it a top priority for dpkg to sort
> out, all hands on deck, whatever needed -

To even consider the possibility to support this missing feature in
dpkg would require for it to get support for at least tracking
filesystem metadata (which is has *never* *ever* supported), which is
currently not deployable:

  

Even with that support in place, that just would not give automatic
aliased directory support. It would need no package to ship anything
inside those directories, and it would also need for some package to
ship those aliased symlinks. And then many corner cases would need to
be considered as then dpkg will need to reconcile what's on the
filesystem, on its database and on the various .deb (given that these
have been a shared resource, that it cannot possibly and safely switch
type by itself), w/o also breaking previous expectations. Not to mention
that the general aliasing issues still would not disappear.

> but to demand the entire project has to stand still,

So wait, when it suits you the "entire project" is involved and cannot
do stuff, but when it does not the "entire project" is not required to
do anything because the proposed solution magically solves stuff for
free with no effort involved… right.

> and to de-facto derail the 

Re: merged /usr

[Guillem Jover]

On Fri, 2021-08-13 at 07:53:20 +0200, Marco d'Itri wrote:
> Implementations with real /bin /sbin /lib* directories and symlink farms
> are not useful because they would negate the major benefits of 
> merged-/usr, i.e. the ability of sharing and independently updating 
> /usr.

Yes, that major benefit that is completely broken by design and
unsupported anyway, because /etc and /var can also easily get out
of sync. If you rely on this then you are on your own anyway…

Also nothing prevents sharing /usr with symlink farms, iff all systems
are in sync, which they must anyway.

So that argument is pretty void of substance and founded on a base
of unreliability… but what's new.

Guillem

Re: Arch triplet for uefi applications

[Guillem Jover]

On Tue, 2021-08-10 at 12:34:18 +, Bastien Roucariès wrote:
> I am going to compile shell.efi from source.
> 
> I whish to install to something stable, but I need an arch triplet
> in order to put in a multiarch (like) location.

Multiarch-based pathnames should only be used by multiarch-conforming
packages for matching and existing dpkg architectures, otherwise we'll
end up with a mess of conflicting paths if we ever add such real
architecture.

Where 
comes to mind.

In addition the main reason we had to add the multiarch tuples was
pretty much to workaround the problem with i386 using varying CPUs in
the GNU system name (i486, i586, i686), so using anything but i386
there would be adding insult to injury. :)

> I suppose that it will be  x86_64-efi-none (or maybe x86_64-windows-efi  )  
> and 
> i686-uefi-none ? Note that grub use x86_64-efi

The vendor always gets ignored in the dpkg context.

Although is there really a need for a cross-compiler here, or is
something like -ffreestanding enough?

> I suppose we should register these triplet somewhere, and I suppose it is 
> config-patc...@gnu.org ?

https://wiki.debian.org/Teams/Dpkg/FAQ#Q._Can_we_add_support_for_new_dpkg_architectures.3F

Thanks,
Guillem

Re: merged /usr

[Guillem Jover]

On Tue, 2021-07-27 at 13:23:46 -0400, Calum McConnell wrote:
> > Of course, having to unnecessarily add more maintainer scripts to
> > handle something that dpkg can do perfectly fine on its own
> 
> TL;DR: merged-usr-via-symlink-farms cannot be done without changing dpkg,

In my mind that's "false", but whatever yes… given that the reversion does
not appear forthcoming, other new features might indeed be needed in case
people do not want to be forced into this train wreck in slow motion.
And as I mentioned on my first reply in this thread I'm prepared to
devote any necessary volunteer time to implement such things in detriment
of other Debian work if necessary.

> and since the quote above seems to indicate you'd be willing to do that,
> why not just change dpkg to support aliased dirs?

I've mentioned this multiple times now, firstly because I think it
would still be a broken layout. Secondly, because there are different
types of dpkg features, some can be used right away, some others will
still need at least two release cycles to be usable. The features that
I think might be needed to be able to avoid being forced into using
merged-usr-via-aliased-dirs, such as registering arbitrary files are of
the former category. The features that would be needed to add support
for merged-usr-via-aliased-dirs would be the latter.

> Another way forward is to transition existing systems without merged-usr
> to a merged-usr-via-symlink-farms.  To accomplish this, we need the
> ability to create symlinks in installations that are not usrmerge, but to
> not create those links in installations that are.  That requires either
> maintainer scripts or a change to dpkg.  You just criticized maintainer
> scripts, so I would assume that they are not your favorite solution. 

Having to add maintainer script usage would be non-ideal, but would be
better than being forced to use merged-usr-via-aliased-dirs.

> Furthermore, others have pointed out that essential packages need to work
> before maintainer scripts are executed.  This behavior is codified in
> policy: 

> > "Since dpkg will not prevent upgrading of other packages while an
> > essential package is in an unconfigured state, all essential packages
> > must supply all of their core functionality even when unconfigured".

That's not what policy says. "unconfigured" does not mean that
*maintainer scripts* have not been executed.

> In other words, using maintainer scripts to create the symlinks that
> enable the core functionality of these packages during configuration is a
> no-go without a revision to policy and a change to dpkg (which might be
> impossible).  That means the symlinks would need to be included in the
> package declaratively: but simply shipping them would break the existing
> merged-usr installations.  We've already established that un-merging and
> then re-merging every installation isn't going to happen: so we'll need to
> get the file references in place using a method that isn't shipping two
> aliasing files and doesn't require maintainer scripts.

This is based on an incorrect premise. In addition, as I've also said
elsewhere bootstrapping is outside the realms of debian-policy anyway.

> Simply modifying dpkg to automatically produce symlinks from /bin to
> /usr/bin at unpack time is not enough either: dpkg isn't necessarily the
> tool doing the unpacking, and so other tools would need to be modified,
> each one of them checking for a merged-usr and then accounting for it if
> needed.  This solution is actually viable: it would lead to a working
> system, and not break the essential guarantee.

This is still based on an incorrect premise. And even though I'd find
what you propose to be a major kludge, all bootstrappers I know of,
always end up running dpkg over all .debs to do a proper installation
of any possibly manually unpacked package. And not to mention that
bootstrappers that support merged-usr-via-aliased-dirs have required
implementing an explicit kludge for it.

Regards,
Guillem

Re: merged /usr

[Guillem Jover]

On Tue, 2021-07-27 at 16:26:34 +0200, Simon Richter wrote:
> On 7/27/21 11:44 AM, Wouter Verhelst wrote:
> > A package in the essential set could work around the issue by moving a
> > file around and creating a necessary symlink in preinst rather than
> > shipping things. The set of Essential packages is small however, and
> > most packages can ship a compat symlink.
> 
> In debootstrap (which is the important use case for Essential packages and
> their constraints), all Essential packages are unpacked first, and then,
> individually, their preinst is run, the files unpacked again (this time from
> dpkg), and then we're in normal dpkg land, although in a chroot.

The installation bootstrap is currently "undefined behavior" and it's
not specified by policy. The properties of the Essential set do not
apply there. See .

> So the concept of a preinst script for an Essential package is wobbly at
> best.

Not really. Bootstrapping has always been done with strings and tape.
Of course, having to unnecessarily add more maintainer scripts to
handle something that dpkg can do perfectly fine on its own, would regress
the progress we have been making to make the installation bootstrapping
automatable and definable. But that seems less worse than the breakage
induced by the merged-/usr-via-aliased-dirs layout. :/

> For debootstrap --foreign, this might be even more complicated.

Only a few paths are expected to be hardcoded, nothing that couldn't
be special-cased by debootstrap along all other bootstrapping
knowledge it contains which would ideally be contained in their
respective packages anyway.

But this indeed is, having to pile hack over hack from the original
broken foundation.

> Also, take care when moving shell commands from a shell script: the bash
> shell at least keeps a cache of commands to paths so it doesn't have to do a
> full path search every time. A shell script that calls
> 
> mv /bin/cp /usr/bin/cp
> ln -s ../usr/bin/cp bin/cp
> mv /bin/ln /usr/bin/ln
> ln -s ../usr/bin/ln bin/ln
> 
> could fall over because it cached the location of "ln" as /bin/ln in the
> beginning, then after the move cannot find it anymore. That needs at least a
> "hash -d ln".

As has been mentioned, this is completely unsafe and does not map to
what dpkg would be doing.

Regards,
Guillem

Re: merged /usr

[Guillem Jover]

On Tue, 2021-07-27 at 11:44:32 +0200, Wouter Verhelst wrote:
> On Thu, Jul 22, 2021 at 03:20:05PM +0100, Simon McVittie wrote:
> > On Thu, 22 Jul 2021 at 15:53:32 +0200, Wouter Verhelst wrote:
> > > I've suggested previously that we can easily make it RC for bookworm to
> > > have a file outside a limited set of directories (/etc and /usr would be
> > > OK, but notably /bin /lib and /sbin wouldn't be) that is not a symlink.
> > > This is easy to detect with a lintian check and reasonably easy to
> > > implement
> > 
> > I don't think that works in general without breaking some of Debian's
> > axioms around Essential packages, as previously described here:
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=978636#118
> 
> Yes. Those arguments didn't convince me then, and they don't convince me
> now.

Ack, these are very contrived.

> A package in the essential set could work around the issue by moving a
> file around and creating a necessary symlink in preinst rather than
> shipping things. The set of Essential packages is small however, and
> most packages can ship a compat symlink.

Yes, along those lines. To try to do something resembling dpkg's safe
behavior, we'd need to do in preinst, something like:

  - if /usr/foo does not exist:
- copy /foo to /usr/foo
- replace /foo with a symlink to /usr/foo

Then dpkg would replace /usr/foo with the new version. Of course this
is all kinds of suboptimal, as the .debs will still not ship the actual
symlinks and it's trying to replicate what dpkg is designed and supposed
to do to handle Essential packages safely, even when doing this kind of
switch, where it will delay symlink installation as the last step… but
we cannot do that right now due to the incorrect restrictions imposed by
merged-/usr-via-aliased-dirs. :(

I have very deep and strong regrets about having removed compat symlinks
under /, to make it possible for people that wanted to locally use
the usrmerge hack. I guess the lesson learned with this episode, is
that in the future, similar stuff cannot be let through, when people
promise this will not be pushed into the distro, and it's just for
local deployments and similar, or we end up with this kind of mess. :/

> > I have a longer mail written with possible ways forward, which I'm
> > deliberately not sending right now, because the first step in all of these
> > plans is "release Debian 11" and I don't want to distract the people who
> > are making that happen (any more than has already happened).
> 
> This is so exhausting.

Indeed, very. The impression I'm getting is that instead of stopping the
bleeding effect, this is being let fester to the point any option will
be terrible, so anything, regardless of its badness will seem acceptable
to make progress at that point.

> Yes, I know the release is close, and yes, I know that some people are
> immensely busy working on that. I want to help them do so in any way I
> can, but they're not *required* to read -devel, and "they might read
> this and get distracted" seems like a pretty poor argument.
> 
> I'm not busy with the release. Are you? If not, you *can* actually come
> up with an argument right now, and I promise not to insist on any
> decision being made until the release happens, so that those
> hypothetical people who *are* busy with the release can still chip in
> later if they choose to do so.

Yes, I'd say this is one of Debian's development fallacies. You know
you are dealing with strong arguments when this comes up. Also the
proponents are not worried now that this is being shoved down by
force due to the involvement of the TC…

Thanks,
Guillem

Re: Re: merged /usr considered harmful (was Re: Bits from the Technical Committee)

[Guillem Jover]

[ Barak, I appreciate your mail, but *sigh*, it's still frustrating,
  as pretty much many of the mails related to this, as they keep
  ignoring what has been said, and I feel like talking to a wall. :/ ]

On Thu, 2021-07-22 at 11:48:34 +0100, Barak A. Pearlmutter wrote:
> Seriously? Being able to type
> 
> dpkg -S $(which cat)
> 
> instead of
> 
> dpkg -S $(which cat|sed 's:^/usr::')
> 
> is the big user-level pain point?

No. As I've mentioned before, f.ex.:

 * dpkg, dpkg-divert, or update-alternatives are unable to detect file
   conflicts and thus might allow silent overwrites of random stuff on
   disk (3rd-party/local packages where we have no control over, or
   even packages from old Debian releases come to mind),
 * when moving files across packages and across aliased directories,
   these pathnames might end up disappearing depending on the unpack
   order (new/old Debian or 3rd-party/local packages too),
 * dpkg-deb -x on the root directory (yes, people use this to recover
   systems) with any .deb that contains files on real directories under
   «/» (3rd-party/local packages), will replace the symlinked directories
   with real ones,
 * dpkg-statoverride used with aliased pathnames that exist on disk
   but unknown to dpkg, will fail to apply the overrides (itself and
   dpkg on unpack), this could have security implications f.ex.,
 * dpkg-triggers used with aliased pathnames that exist on disk but
   unknown to dpkg, will fail to activate triggers,

Also, yes, anything using dpkg-query is now broken, take for example the
cruft package, the problem is that there are way more things depending
on this interface.

But the aliasing problem is a general one. Take the /etc/shells
mentioned recently on d-d, the admin needs to remember to always add
two entries (/ and /usr) to make sure things will match by any random
program accessing the file for its checks.


Regarding your dpkg-query example above, using which(1) and assuming
a PATH set to prefer /usr directories (which AFAIR is the current
default) would mean that once all objects have moved in the .debs into
/usr, the former would always work, but at that point the latter would
not. Also that assumes people would use which(1) or similar, or even
fully canonicalize the pathname, which might or might not have anything
to do with what dpkg knows about on its database.

> People seem pretty worked up over this, but honestly I'm not
> understanding why. We already have $PATH which (let's be honest) is an
> ancient crappy workaround for the original Unix sin of not keeping all
> the executables in one place. (And analogous wheel-reinventing goo for
> /lib vs /usr/lib vs /usr/lib/x86_64-linux-gnu, etc.) Given that,
> moving them around isn't supposed to be a big fuss. Oh, but there are
> also shebang #!/bin/foo issues and other hard coding. The shebang is
> already such a mess that people use #!/usr/bin/env foo, just to get
> foo searched in $PATH. (85 of the 890 scripts in /usr/bin/* on the
> machine I'm typing this on use a /usr/bin/env trampoline.) Nobody
> would ever consider having /bin/foo and /usr/bin/foo be different
> programs, that would be madness. The TC basically concluded that the
> distinction between /bin and /usr/bin, etc, had totally broken down
> and there's no advantage to keeping them distinct, plus initrd is the
> new /bin. (Pretty much the same argument as in the design of plan9.)
> I'm not seeing much argument against that, except for nostalgia.

In case this is not clear, and as I've mentioned also countless times
already, I have no problem with merging contents of directories from /
into /usr, I've done that for all packages I maintain (where no compat
symlinks were required). I do have a huge problem with the approach
that has been forced into the distribution disregarding the packaging
system and going on its back, though.

You might want to take a peek at
.


Can any of the above be avoided? Well certainly, in the same way you
can probably cross a mine field safely, you'll just need to always be
aware of any of these things, and verify everything you install, and
all pathnames you use, etc. I'm not sure how this is in any reasonable
universe an acceptable way to expect end users to use the system TBH.


Thanks,
Guillem

Re: merged /usr considered harmful (was Re: Bits from the Technical Committee)

[Guillem Jover]

On Tue, 2021-07-20 at 11:31:37 +0200, Guillem Jover wrote:
> Unfortunately, when the supporters of the merged-/usr-via-aliased-dirs
> pushed their approach into the distribution, that meant that package
> stopped being able to ship compatibility symlinks under «/», and those
> needed to be "handled" in maintscripts (by reimplementing poorly and
> unsafely what dpkg is supposed to do). This means dpkg is not in the
> loop and cannot perform a safe upgrade moving these pathnames safely,
> as long as merged-/usr-via-aliased-dirs is supported.

Sorry, how sloppy of me, let me qualify that last word, to avoid any
confusion, "supported". As in inflicting that into unsuspecting users,
and in having to bear and pay the price for its current existence, not
as in an approach that is supported by the packaging system.

Regards,
Guillem

Re: merged /usr considered harmful (was Re: Bits from the Technical Committee)

[Guillem Jover]

On Mon, 2021-07-19 at 16:41:42 +0200, Johannes Schauer Marin Rodrigues wrote:
> So what what is actually the roadmap after the bullseye release? What is the
> way forward? Should I rather file bugs with patches against individual 
> packages
> to move their files from /(sbin|bin|lib)/ to /usr/(sbin|bin|lib)/ or do we
> already have a debhelper patch to do that move for us?

Unfortunately, when the supporters of the merged-/usr-via-aliased-dirs
pushed their approach into the distribution, that meant that package
stopped being able to ship compatibility symlinks under «/», and those
needed to be "handled" in maintscripts (by reimplementing poorly and
unsafely what dpkg is supposed to do). This means dpkg is not in the
loop and cannot perform a safe upgrade moving these pathnames safely,
as long as merged-/usr-via-aliased-dirs is supported.

This is all kinds of nasty, because a proposal that has a complete
disregard for the packaging system, and is founded on assumptions
about features that do not even exist, is also forcing that design
breakage into a solution that would otherwise require no such hacks.
:(

> This would mean that we only have to bear with the problems mentioned by
> guillem until that move is complete, correct?

No. Even if today we magically could get all .debs shipping object
directly under «/usr» instead of «/», and had the aliased symlinks in
place, the problems I've mentioned would pretty much be in place.

Regards,
Guillem

Re: merged /usr considered harmful (was Re: Bits from the Technical Committee)

[Guillem Jover]

On Mon, 2021-07-19 at 15:10:42 +0200, Michael Biebl wrote:
> Am 19.07.21 um 03:36 schrieb Guillem Jover:
> > What I've also said multiple times, is that
> > merged-usr-via-moves-and-symlink-farms could have been implemented in
> > a fully automated way, by debhelper, w/o requiring any maintainer scripts,
> > all with full cooperation and managed by dpkg, with .debs shipping
> > actual tracked pathnames

> What you propose is, that each and every package does its /usr-merge
> transition on its own. This only works, if packages are independent (enough)
> so this actually works.
> Unfortunately this is not the case. Take PAM for example. You can't just
> recompile src:pam and have debhelper automatically move all files to /usr.
> This would break all packages that install a PAM plugin. You have a
> transition here, involving many packages.
> Same is true for udev rules, systemd service files, basically every package
> that provides interfaces/hooks to other packages is affected.
> So it's not that simple unfortunately. You can't fully automate that.

Not at all. pam or whatever we transition via cooperation from dpkg,
would be kept compiling using the directories it currently uses, and
debhelper would simply move the objects on the .deb from «/» to «/usr»,
and create the compat symlinks. That means pam, and any plugins migrated
or not, would still be available in the current pathnames causing no
breakage. This part would be done automatically.

And as I've said elsewhere, removal of the compat symlinks would
require manual intervention (at the maintainer discretion), at some
later time, to modify the configured installation paths (which is
something that cannot be automated in debhelper, due to packaging
overrides and similar), at which point it can be decided whether to
declare say a local pam flag day, or add the needed Breaks and similar,
or add support to pam to look in both pathnames to avoid the two
previous items.

> According to
> apt-file search -x '^/(lib|bin|sbin)'
> on my Debian sid/amd64 system, we have 1747 packages shipping 24583 files in
> those directories. There are *many* such entangled transitions hidden in
> there, so I fear this is not manageable.

See my comment above, and josch reply.

> As Luca pointed out, even distros with a much stricter governance model were
> not able to do that.

Well if they did it poorly, no wonder, I guess.

> The /usr-merge transition as described and decided on in the TC bug, seems
> to me is the only viable way forward.

I obviously disagree.

> Yes, it does break dpkg -S, but your idea of using a list of mapped paths as
> in [1] seems like an entirely reasonable approach to solve this.

Did you miss the section in the mail you are replying where I mention
that f.ex. dpkg tools (dpkg, dpkg-divert, u-a) missing to then detect
file conflicts, or files disappearing on moves, or dpkg-deb -x (or tar)
destroying merged-/usr-via-aliased-dirs systems?

All these, including dpkg -S (which BTW also breaks after paths have
been moved, as when say bash ships as /usr/bin/bash, then dpkg -S will
also fail to find the expected /bin/bash), are currently affecting any
system being installed by the default installer, or ones having been
switched by the usrmerge hack.

> Once we have this global switch to merged-usr, packages can bit by bit,
> completely independent, update their debian/rules to use --prefix=/usr and
> after a few years, we don't have any packages anymore installing any files
> in /. We could aid this process with a lintian check that flags packages
> that install files in /(sbin|bin|lib)/.

The same can be said about my proposal, except that then dpkg is kept
in the loop, the transition is done safely by dpkg as part of individual
package upgrades, instead of having to use something off-band and as
disturbing as usrmerge during upgrade, on dpkg's back w/o any
cooperation from it…

Regards,
Guillem

Re: merged /usr considered harmful (was Re: Bits from the Technical Committee)

[Guillem Jover]

On Thu, 2021-07-15 at 10:13:47 +0100, Luca Boccassi wrote:
> As it has been said and written many times already, in reality this is
> not broken by design at all and in fact it is the only successful
> strategy that has been deployed by other distros - it's what is being
> called merged-usr-via-moves-and-symlink-farms that is broken. We can
> say this with certainty because both approaches have been tried, so
> there's actual real-world data to look at. Just go look at the absolute
> mess that Suse got themselves into by following that solution - a 10-
> years-long massive half-done-never-finished headache that took an
> inordinate amount of work to back out of, and move to the actual
> working solution that everybody else are using - merged-usr-via-
> aliased-dirs. On the other hand Fedora/RHEL had a smooth and simple
> transition using merged-usr-via-aliased-dirs, and that was the end of
> it.

Yes, a single (a couple!?) data point(s) of a strategy used in another
unrelated distribution, with completely different packaging stacks and
ecosystems, which was done very poorly, has been presented repeatedly.
I'm not sure why that has much value TBH.

The above seems to also be confusing how and if a design has been
deployed, with its inherent (short and long term) properties. A badly
performed *deployment* for a better design does not make its properties
bad, in the same way that *deploying* a flawed design faster does not
make its properties better…

What I've also said multiple times, is that
merged-usr-via-moves-and-symlink-farms could have been implemented in
a fully automated way, by debhelper, w/o requiring any maintainer scripts,
all with full cooperation and managed by dpkg, with .debs shipping
actual tracked pathnames, if it had not been for the mess required
by merged-usr-via-aliased-dirs. :/

> Dpkg has some very minor bugs that rpm/dnf/yum/zypper/whatever do not
> suffer from. So what? It is perfectly normal as it's software and all
> softwares have bugs. They could be fixed, worked around, or ignored,
> like all other bugs.

If by "very minor bugs" you mean that f.ex.:

 * dpkg, dpkg-divert, or update-alternatives are unable to detect file
   conflicts and thus might allow silent overwrites of random stuff on
   disk,
 * when moving files across packages and across aliased directories,
   these pathnames might end up disappearing depending on the unpack
   order,
 * dpkg-deb -x on the root directory (yes, people use this to recover
   systems) with any .deb that contains files on real directories under
   «/», will replace the symlinked directories with real ones,

then, yes, I guess "very minor" indeed. But then I completely object
to this being classified as bugs in dpkg, as this has been shoved in
disregarding that dpkg does *not* support this, and it would require
new *features* to be implemented, so this "transition" is founded on
assuming features that do not exist, or completely going behind
dpkg's back, which sounds great I guess…

The problem in general is that this layout introduces unreliability
and silently induced breakage stemming from this flawed filesystem
layout, which is going to affect the people that are going to benefit
the less from its properties, and are the less experience to deal with
it.

I've tried to update the 
with more explicit information, in case some people might have missed
that from previous instances of this discussion, and added an initial
table of properties for both proposals, to avoid cluttering and repeating
it on the list, and to ease updating it.




We do actually have experience with "an aliased" layout from symlinked
/usr/share/doc/ directories, where those are for optional/removable parts
of the filesystem, and the symlinks are even known and managed by dpkg.
And those have been a major and constant source of packaging bugs.

And here we are getting the project installing by default systems that
are fighting the packaging system and going on its back, to enable a
filesystem design layout mostly experienced admins will benefit from
in very special deployment conditions, where final users can very
easily suffer from its introduced unreliability (from 3rd-party repos
or locally built packages, etc, f.ex.).

Because the above has been brought up before and the proponents are well
aware of these, I'm afraid at this point the only thing that comes to
mind is negligence, TBH. :/

But *shrug*, have at it, I'm tired of the continued and complete
disregard of the unreliability issues and subversion of the packaging
system, which are supposed to be pillars of our project, every single
time. I just replied so that people that might not want to be forced
into this stuff know that there's going to be a way out.

Regards,
Guillem

Re: Need help with Multi-Arch in systemd

[Guillem Jover]

On Wed, 2021-07-14 at 11:59:11 +0100, Simon McVittie wrote:
> On Thu, 08 Jul 2021 at 23:03:48 +0200, Michael Biebl wrote:
> > [a separate libsystemd-shared-249 .deb] would also mean, that on every
> > new upstream release, systemd would have to go through NEW
> 
> It seems like we're rejecting a good technical solution because
> social/organisational factors block it (namely, new binary packages
> triggering manual review from the ftp team even if there has not been
> any significant change in how the package is organised, resulting in
> manual review being artificially frequent for libraries that happen to
> break ABI a lot, but infrequent for packages that aren't libraries or
> don't break ABI).

Yes. I was mentioning exactly this the other day on the
#debian-bootstrap IRC channel.

In addition having this automatic support could make life easier for
many other potential packages.

> This seems like a side-effect of the ftp team's two gatekeeping roles
> (legal review and managing the namespaces of source and binary package
> names) both having the dak override file as their implementation, rather
> than necessarily anything that was designed or intended.

Yes, plus section and priority-spaces. But then, I don't see why a binary
package rename should trigger a new legal audit.

> Would it be feasible for dak to have a list of binary package name
> regexes mapped to a source package and a section/priority, and auto-accept
> packages from the given source package that match the regex, assigning
> the given section/priority, without manual action? That would let the
> ftp team pre-approve src:systemd to ship /^libsystemd-shared-[0-9]+$/
> in libs/optional, for example.

What I had in mind was that DAK would gain support for automatic
ACCEPT of binary package renames due to SONAME version bumps,
something like this:

  * If the new bin:lib:
- replaces an existing bin:lib from the same
  source, where  is lower than .
- contains a shared library mapping to that package name.
- is in section */libs or */oldlibs.
  * Then → auto-ACCEPT, pre-filling the new section/priority from the
old binary package.
  * Otherwise → NEW.

I guess it could potentially be further extended later on to cover
other safe non shared library cases.

But if that's too much to ask, either due to implementation or policy
concerns, I'd take an explicit allowlist letting specific cases
through, such as the systemd one, instead of having to settle for
either suboptimal or wrong solutions for the problem at hand, due
to the currently required workflow being too cumbersome.

Thanks,
Guillem

Re: What are desired semantics for /etc/shells?

[Guillem Jover]

Hi!

On Tue, 2021-06-15 at 13:31:15 +0200, Felix C. Stegerman wrote:
> FYI I just noticed another inconsistency: on my merged /usr systems
> (installed as such, not converted later w/ usrmerge), /etc/shells
> contains both /bin/ and /usr/bin/ paths for some shells, but not all
> (e.g. no /usr/bin/sh, no /bin/screen).
> 
>   $ sort /etc/shells
>   # /etc/shells: valid login shells
>   /bin/bash
>   /bin/dash
>   /bin/rbash
>   /bin/sh
>   /usr/bin/bash
>   /usr/bin/dash
>   /usr/bin/rbash
>   /usr/bin/screen
>   /usr/bin/tmux

Yeah, welcome to the wondrous world of merged-/usr-via-aliased-dirs,
where these paths are aliased, and can be used in places that will
end trying to match against literal entries in /etc/shells, and fail.

While I think I might have been aware of this issue, I never explicitly
tracked it in , added now,
thanks!

Regards,
Guillem

Re: merged /usr considered harmful (was Re: Bits from the Technical Committee)

[Guillem Jover]

On Wed, 2021-07-14 at 19:54:56 +, Thorsten Glaser wrote:
> Sean Whitton dixit:
> >* #978636 move to merged-usr-only?
> >
> >  We were asked to decide whether or not Debian 'bookworm' should
> >  continue to support systems which are not using the merged-usr
> >  filesystem layout.  We decided that support should not continue beyond
> >  Debian 'bullseye'.
> 
> What? WHAT? ＷＨＡＴ？
> 
> >  The decision is captured here:
> >  
> 
> No reason provided either. This stinks. I’m v̲e̲r̲y̲ disappointed.
> Debian is becoming untenable. Years ago, I had hoped it won’t.

I've been meaning to send a note about this for some time now, but
as I feel it keeps getting ignored, it always seems a bit pointless.

But in any case, given that merged-usr-via-aliased-dirs is not really
supported by dpkg anyway, it is broken by design [B], I have no
intention whatsoever to break any of my systems with such layout going
forward, I'm thus planning to spend any necessary volunteer time
implementing any fix, workaround or solution required to avoid having
to use it, in detriment of other Debian volunteer time. I already
started some time ago with dpkg-fsys-usrunmess(8), present already in
the upcoming bullseye release.

[B] 


Thanks,
Guillem

Re: Need help with Multi-Arch in systemd

[Guillem Jover]

On Fri, 2021-07-09 at 12:29:19 +0200, Michael Biebl wrote:
> Am 09.07.2021 um 11:37 schrieb Timo Röhling:
> > * Michael Biebl  [2021-07-09 11:01]:
> > > Splitting out libsystemd-shared (once) will make PID1 very brittle.
> > > It can lead to situations where old systemd + new libsystemd-private
> > > is installed. If the installation is aborted at this point, you have
> > > an unbootable system.

> > Helmut suggested a tightly versioned dependency, so such a state
> > would never be consistent.

> That tightly versioned dependency doesn't help unfortunately. There is still
> a time window between the new libsystemd-shared and the new systemd being
> unpacked.

If the private library has no backwards or forward compatibility (due
to the SONAME used) the time window where the library does not match
the expectations of the stuff linked to it might indeed be too big.
But the reported bug is just a symptom of a bigger issue. This problem
already exists for any of the other binary packages built against this
private shared library, there's a time-window where they will not work
if the installation gets interrupted or fails, compared with public
shared libraries.

This implies to me the correct solution is a name-versioned package,
even though that seems cumbersome given our current archive handling
practices this is IMO the only correct solution. The alternative of
using Multi-Arch:allowed is a workaround for this packaging problem.

Another solution might be to request upstream to stabilize this library?
Yet another solution, which seems suboptimal, might be to statically
link it, but the shared library seems rather big, but perhaps if the
programs linked only use parts of it, that might not be too bad?

Thanks,
Guillem