Re: Proposed changes to sbuild and debootstrap

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting RhineDevil (2020-11-29 20:33:22)
> A little proof of concept of what I'm aiming to is in the latest commits of
> https://salsa.debian.org/TanyaEleventhGoddess/sbuild, sbuild-createschroot is
> able to work with the commands as-is, but also features automatic detection
> of a possible default configuration, plus I fixed a possible problem in
> filehandlers handling that may lead to deadlocking

I've had a look at the commits in that repo and think it would be better to
move the flavor support to debootstrap. sbuild-create-chroot is supposed to be
only a very thin layer on top of debootstrap and should not do much more than
all the schroot specific stuff. That way, a new --vendor switch would just be
passed on to debootstrap which would then do the right thing. If I understand
your problem correctly (derivatives having the same name as Debian releases but
differing content) then in the end you need to do something about debootstrap
anyways. By starting with debootstrap support, you avoid duplicate work in
sbuild-create-chroot.

When you open the bugs against debootstrap, please CC me. I'm developing a
debootstrap alternative called mmdebstrap which would also need to gain support
for a --vendor switch or similar.

Another package that might be of interest to you is distro-info which carries
distributino meta-data for different flavors like debian or ubuntu.

https://salsa.debian.org/debian/distro-info

And finally, there is vendor support in dpkg itself:

https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/tree/scripts/Dpkg/Vendor

Tools written in Perl like sbuild and mmdebstrap make use of this, so
implementing things there, again avoids having to touch other tools.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Bug#976073: sbuild: support "podman" as chroot mode and provide a sbuild-create-oci command (built on top of buildah)

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Raphaël Hertzog (2020-11-29 11:42:16)
> I know that multiple developers started using podman and buildah to manage
> containers where they build their Debian packages. With user namespace
> supports, this allows rootless building (like the "unshare" chroot
> mode)... you don't even need root to setup the "build chroot" since those
> are containers that you can download (or rebuild if you prefer).

already today you don't need to be root to setup the build chroot, by using
mmdebstrap as a debootstrap drop-in-replacement in sbuild-createchroot like so:

$ sbuild-createchroot --debootstrap=mmdebstrap --make-sbuild-tarball 
~/.cache/sbuild/unstable-amd64.tar.gz unstable $(mktemp -d)

The resulting tarball can then be used with the sbuild unshare backend. The
only time you need be root is to execute

$ sudo sysctl -w kernel.unprivileged_userns_clone=1

But I guess you also need this for podman and buildah?

> Thus it would be nice if sbuild had a "podman" chroot mode where it could
> use podman containers to build the packages.
> 
> A "sbuild-create-oci" command would also be helpful to build the various
> container images, either from scratch (so that you don't have to trust
> images that you download) or on top of pre-existing OCI images (to
> save time and effort). That command should not be hard to build on top
> of buildah.
> 
> Some links:
> http://tauware.blogspot.com/2020/04/building-packages-with-buildah-in-debian.html
> https://developers.redhat.com/blog/2019/02/21/podman-and-buildah-for-docker-users/

I'm absolutely for it! If somebody wants to implement and maintain it, please
send patches for me to review. The person can then keep maintaining the podman
chroot mode easily because sbuild is in the Debian group on salsa.

What I would like even more, would be to add a podman backend to autopkgtest.
This has the following advantages:

 - it would already work with sbuild today (no changes in sbuild required)
 - no duplicated work to have podman support in both sbuild and autopkgtest

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Russ Allbery (2020-12-14 23:54:37)
> > The point I'm making is that i386 processors are still incredibly common,
> > and we shouldn't abandon their users.
> 
> Not abandoning users is a powerful motivating force, but it still has to
> succeed in motivating people.  Debian can't make a decision on the basis
> of not abandoning users.  We have to make a decision on the basis that
> someone is volunteering to do the work.  Perhaps they're volunteering to
> do that work so that we're not abandoning users, and that's great, but
> that additional step is important.
> 
> I think it's therefore useful in this sort of thread to be very clear
> whether your conclusion is "and therefore I am volunteering to do the work
> to keep i386 alive" or whether your conclusion is "and therefore I am
> asking other people to volunteer to keep i386 alive," and be aware that
> the latter may not be successful because volunteer time is a limited
> resource and there are many worthy things that we could all be working on
> to make the lives of users better.
> 
> If we can confirm that the volunteer resources are there, we can ask what
> they need from the rest of the project to be successful.

I cannot donate my time (I'm also lacking the skill) but I'm willing to put my
money somewhere if it means to keep i386 alive for longer. I privately own
perfectly working old hardware that I would hate to send to the landfill just
because software support is ending.  Should i386 be discontinued I should
probably only keep using the hardware in a separate network disconnected from
the internet but that would make the hardware much less useful.

If somebody could direct me to organizations or people who just need financial
support to keep i386 alive, that would be great. In the light of a planet with
limited resources, I think it's worth my money to help keeping old hardware
around and avoid the waste of resources and energy that manufacturing new
equipment incurs.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Help required to determine why some packages are being installed

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Jonas Smedegaard (2021-01-27 16:15:17)
> I suspect that's not really the case - that instead apt tools might pick at
> random.

no, apt does not pick at random. The apt solver prefers the first alternative.

> It is my understanding that build daemons _ignore_ secondary entries
> exactly to avoid such ambiguity.

More precisely: not only build daemons but sbuild (even if you run it locally)
strips out all but the first alternative for Build-Depends, Build-Depends-Indep
and Build-Depends-Arch, with the exception of those alternatives that name the
same package as the first alternative. Thus, this does *not* have an effect on
alternatives in binary packages.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Need help with Multi-Arch in systemd

[Johannes Schauer Marin Rodrigues]

Quoting Helmut Grohne (2021-07-09 14:24:01)
> Another possibility (kudos to David Kalnischkies) would be exploiting
> something I might call a loophole in the Multi-Arch spec. While we don't
> currently use arch-qualified dependencies in the archive, the spec considers
> them and dpkg and apt support them (somewhat). So in theory, we could say
> that systemd-container Depends: systemd:${DEB_HOST_ARCH}.  I'm not sure
> whether all resolvers agree on this, but I'd expect this dependency to pull
> the right instance (despite being marked Multi-Arch: foreign). Before jumping
> onto this, please consider David's warning: here be dragons.

No, not all solvers agree on this. Imagine this situation:

Package: pkga
Architecture: amd64
Depends: pkgb:i386
Multi-Arch: no

Package: pkgb
Architecture: i386
Multi-Arch: foreign

dose3, apt and dpkg agree that pkga can be installed because its dependency is
satisfied by pkgb. But if we change pkgb:i386 to pkgb:amd64 to end up with
this:

Package: pkga
Architecture: amd64
Depends: pkgb:amd64
Multi-Arch: no

Package: pkgb
Architecture: i386
Multi-Arch: foreign

Then dose3 can satisfy the dependency of pkg while apt and dpkg cannot. This is
because when building the cudf representation of above two packages, dose3
considers pkg to provide pkgb for all architectures, including amd64, because
it is marked as m-a:foreign.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Huh?? sbuild fails and pbuilder succeeds in building a Fortran-containing Python package

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Julian Gilbey (2021-02-10 22:51:39)
> I wonder if anyone has an idea about this thorny problem.  I'm trying
> to create a package and thought I'd got something wrong, but have
> found the same behaviour happens with the "amp" package (version
> 0.6.1-1) and the "spherepack" package (version 3.3~a1-4), which
> suggests to me that the problem lies outside of the package itself.
> But I have very little idea how to track this bug down.
>
> [...]
> 
> They are so different, yet supposedly using essentially identical
> build environments.  (They are also both using sid repositories.)
> 
> If you have any idea why I might be seeing this behaviour, and what I
> might be able to do about it, that would be fantastic!

I would actually not be surprised if we find reproducibility problems between
packages built on pbuilder versus packages build with sbuild. For example
sbuild sets HOME=/sbuild-nonexistent which will make packages FTBFS that try to
put anything in $HOME. This is not the problem here, though. If I wrap the call
to dh in debian/rules in strace -p I get:

[pid   609] statfs("/dev/shm/", {f_type=TMPFS_MAGIC, f_bsize=4096, 
f_blocks=2097152, f_bfree=861837, f_bavail=861837, f_files=1495318, 
f_ffree=1426033, f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096, 
f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_RELATIME}) = 0
[pid   609] futex(0x7f82bf2a7410, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid   609] getpid()= 609
[pid   609] lstat("/dev/shm/o5kYxf", 0x7ffcfec00740) = -1 ENOENT (No such file 
or directory)
[pid   609] openat(AT_FDCWD, "/dev/shm/o5kYxf", O_RDWR|O_CREAT|O_EXCL, 0600) = 
-1 EACCES (Permission denied)
[pid   609] close(3)= 0
[pid   609] close(4)= 0
[pid   609] write(2, "error: [Errno 13] Permission den"..., 36error: [Errno 13] 
Permission denied

A workaround for this problem is indeed to add

--chroot-setup-commands="chmod 777 /dev/shm"

to your sbuild invocation. A cursory glance into the pbuilder codebase reveals
that pbuilder will mount a tmpfs into /dev/shm:

https://sources.debian.org/src/pbuilder/0.231/pbuilder-modules/?hl=417#L417

For sbuild, whether this is done, this depends on your chroot backend. The
package builds fine on the buildds though, so I guess they have a schroot setup
that sets /dev/shm up correctly?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Huh?? sbuild fails and pbuilder succeeds in building a Fortran-containing Python package

[Johannes Schauer Marin Rodrigues]

Quoting Sebastian Ramacher (2021-02-11 10:00:42)
> > > A workaround for this problem is indeed to add
> > > 
> > > --chroot-setup-commands="chmod 777 /dev/shm"
> > > 
> > > to your sbuild invocation. A cursory glance into the pbuilder codebase 
> > > reveals
> > > that pbuilder will mount a tmpfs into /dev/shm:
> > > 
> > > https://sources.debian.org/src/pbuilder/0.231/pbuilder-modules/?hl=417#L417
> > > 
> > > For sbuild, whether this is done, this depends on your chroot backend. The
> > > package builds fine on the buildds though, so I guess they have a schroot 
> > > setup
> > > that sets /dev/shm up correctly?
> > 
> > OK, so I'll submit this as a bug report / feature request to sbuild.
> 
> schroot already does the same in the sbuild and buildd profiles:
> 
> https://sources.debian.org/src/schroot/1.6.10-11/etc/profile-templates/buildd/linux/fstab/
> https://sources.debian.org/src/schroot/1.6.10-11/etc/profile-templates/sbuild/linux/fstab/
> 
> Is your setup using any of the two profiles?

It seems it is. See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=982518#13

Lets continue figuring this problem out via the BTS.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Tips for debugging/testing debian/control Depends/Breaks etc changes?

[Johannes Schauer Marin Rodrigues]

Hi Otto,

Quoting Otto Kekäläinen (2021-03-24 17:37:46)
> I've noticed I've spent quite a lot of time debugging various situations
> where the debian/control definitions for
> depends/breaks/replaces/conflicts/provides are not optimal.
> 
> The waste of time is two-fold:
> 
> 1) apt is not verbose enough
> 2) the cycle to rebuild/tests is too slow
> 
> As an example of 1, sometimes I see this:
> 
> apt install mariadb-client
>  The following packages have unmet dependencies:
>  mariadb-client : Depends: mariadb-client-10.5 (>= 1:10.5.10) but it
> is not going to be installed
> 
> apt install mariadb-client-10.5
>  Installing.. Done!
> 
> When this happens I have no idea why apt did not resolve the
> dependency by itself automatically, as there was no real conflict in
> installing it.
> 
> Do you have tips on how to debug the root cause of situations like these?
> 
> 
> For the problem 2, I hate to rebuild all of the packages (and
> binaries) just because there was a change in debian/control and go
> through the hassle of updating a test repo etc.
> 
> I wonder if there is some other "lighter" way to just edit the
> debian/control and produce new binary packages with them updated
> without having to actually build new binary packages (and no, editing
> the .deb files manually and repackaging them with 'ar' is not an option that
> would make life easier).

there is a way to throw arbitrary installation problems at the apt solver
without building any *.deb packages and without creating any repos with
Packages and Release files: by using /usr/lib/apt/solvers/apt directly. You do
it by going through the following steps:

1. Create an initial EDSP document that represents the current dependency
   situation. You can do this by running something like this:

   APT_EDSP_DUMP_FILENAME=/tmp/dump.edsp apt-get --simulate install 
--solver dump ghc

2. Open /tmp/dump.edsp in an editor and edit whatever you need. For example
   with the command above the first stanza will contain:

  Install: ghc:amd64

   And you probably want to change that into whatever you want to ask apt to
   install. Then go through the file and find the stanza for the package that
   you want to change the depends/breaks/replaces/conflicts/provides of and
   change whatever you like.

3. Evaluate the EDSP problem using the apt solver by calling:

  /usr/lib/apt/solvers/apt < /tmp/dump.edsp

   You can improve the output by throwing in options like:

  -oDebug::EDSP::WriteSolution=yes

   Or the options that David Kalnischkies mentioned in his mail.

The advantage is, that you only have to edit one file (the EDSP) and only have
to run one command (/usr/lib/apt/solvers/apt) to check what happens when you
change something. No need to build any actual Debian package or create a
repository with them.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Packages in contrib solely because they allow using non-free software

[Johannes Schauer Marin Rodrigues]

Quoting Simon McVittie (2021-04-04 13:55:21)
> On Sun, 04 Apr 2021 at 13:23:14 +0200, Joerg Jaspert wrote:
> > On 16093 March 1977, Dominik George wrote:
> > > That surprised me. If a package is free software, in ful laccordance
> > > with the DFSG, why is it put into contrib?
> > 
> > There is, as usual, no clear answer.
> > 
> > The policy for main is clear on that it needs to be self contained. So
> > software in main must not require something outside to work and do its job.
> > Contrib is the area where that is allowed. License wise its the same as
> > main, but it allows to depend on something not available for building or
> > working.
> 
> There was some discussion relevant to this on debian-devel-games earlier
> this year: the subthread starts at
> .
> 
> I don't think the rule can be as simple as "must not require something
> outside to work and do its job", because we have Free clients for non-Free
> network services (like all the instant messaging services that used to
> exist), and those were always in main. Similarly, it would be absurd to
> kick out email clients into contrib just because they are primarily used
> to read non-Free email messages like this one! :-)
> 
> As I mentioned on d-d-games, one of the major things I tend to ask myself
> when thinking about the borderline between main and contrib is: if the
> content that this package downloads was somehow in the Debian archive,
> would the downloader have a Depends or Recommends on it, or would it be
> a Suggests or no dependency at all?
> 
> Another factor in choosing main or contrib, for me, is whether the
> downloader is specifically hard-coded to work with particular content,
> or whether it generically works with any content in a particular
> format. game-data-packager and quakespasm are both close to the main/contrib
> borderline, and this factor is why I think they are in the correct archive
> areas: g-d-p downloads and repackages specific non-Free games, so it's
> in contrib, whereas quakespasm can play any Free or non-Free set of
> Quake-compatible levels, so it's in main.
> 
> I think winetricks is *probably* correctly in contrib, because it has
> hard-coded knowledge of how to best to download and install specific
> non-Free Windows DLLs? Like game-data-packager, it's quite close to the
> line between main and contrib, but I think it does make sense to consider
> it to be on the contrib side of that line (although I'm sure I could be
> convinced otherwise).
> 
> As Joerg says, this is all fairly subjective and unclear, but I think we
> do have an approximately coherent policy for what can and can't be in main,
> and that's realistically the best thing we are going to get.

another way to answer the question is to find some software similar to the one
that you want to package and see if that software is in Debian main or in
contrib. If it is in main, then at least one DD and FTP master already agreed
that packages of that nature are fit for main.

Maybe the comparison to winetricks is not very fitting because I think
winetricks can *only* download non-free software while lutris can also download
games that their developers distribute under a DFSG license?

Let me show you a package of mine that is still in Debian main even though it
is a "downloader [that is] specifically hard-coded to work with particular
[non-free] content" and thus fails Simon's test from above:

https://packages.debian.org/source/unstable/rss-bridge

I guess most people will use that package to access non-free services like
Facebook, Instagram or YouTube. After all, rss-bridge was specifically written
to work around the limitations of these non-free services:

https://github.com/RSS-Bridge/rss-bridge/#rant

Should anybody ever file an RC bug against the package I will point out that it
is also able to work with DFSG free software like Mediawiki. Maybe you can do
the same?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: sbuild and bind mounts

[Johannes Schauer Marin Rodrigues]

Hi Norbert,

Quoting Norbert Preining (2021-03-17 03:55:53)
> in the Qt/KDE Team we use a script for test-building our groups of packages.
> Those groups need to be build in tiers (levels) so that later packages have
> access to the build of earlier levels.  The script we currently use is based
> on cowbuilder with --bindmounts to get the package repo of hitherto built
> packages into the build process, and hook scripts to active the respective
> apt sources line.

with sbuild you can use --extra-package=package.deb or
--extra-package=/path/to/debs and then sbuild will use those debs or all debs
in the directory you specify, respectively, to satisfy dependencies and you
don't need to manage an apt repository yourself.

> Recently I started using btrfs snapshots with sbuild (and eatmydata)
> because it is sooo much faster than cowbuilder, so I would like to add
> sbuild support to our tier-build script.
> 
> My questions are how to achieve:
> - bind-mount directories into the sbuild chroot

This depends on the chroot backend you are using. The default backend is
schroot and you probably want a solution for that one and not for the
autopkgtest or unshare backend. With schroot you would add additional mount
points by editing /etc/schroot/sbuild/fstab

> - add lines to /etc/apt/sources.list
> before the build process starts?

You can either use this sbuild option:

--extra-repository="deb http://deb.debian.org/debian experimental main"

or you can use the option

--chroot-setup-commands="echo 'deb ...' >> /etc/apt/sources.list"

which runs arbitrary shell scripts inside your chroot.

All of the above is documented in the sbuild man page but there is lots of
stuff in there so I can understand why it's hard to find things without reading
the whole thing first. Anyways, if you see ways how to improve the man page,
I'd like to hear about them. Other than that there is
https://wiki.debian.org/sbuild which collects common recipes of how to use
sbuild in different scenarios, so feel free to add things you find useful to
it.

Thanks!

cheers, josch

signature.asc
Description: signature

Unsolicited internet access in default installs (was: New service: https://debuginfod.debian.net)

[Johannes Schauer Marin Rodrigues]

Quoting Steinar H. Gunderson (2021-02-27 13:46:27)
> On Sat, Feb 27, 2021 at 12:29:34PM +, Thaddeus H. Black wrote:
> > I would prefer Kurt's option.  Network silence is important.  Network
> > noise would probably be a bug.  A sysadmin should not be made to take
> > special precautions to avoid the inadvertent disclosure of the user's
> > presence on the network.
> 
> It's 2021; machines are not silent on the network. That ship sailed long ago.

I was about to write a rant, stating that it's unacceptable for any requests to
remote servers to be made by a Debian default installation without my explicit
consent. So I ran:

qemu-system-x86_64 -enable-kvm -m 2G -netdev user,id=u1 -device e1000,netdev=u1 
-object filter-dump,id=f1,netdev=u1,file=install.pcap -cdrom 
debian-testing-amd64-DVD-1.iso -hdd disk.img

And found out that I was wrong and you are right. Even though I answered "No"
to all questions related to mirrors and popcon during the installation, the
above command still recorded a DNS query for debian.map.fastlydns.net and a
subsequent download of /debian-security/dists/bullseye-security/InRelease.

Later on, after the installation had finished and gnome started, I see requests
to cdn.fwupd.org where something got downloaded from and then some
communication with 0.debian.pool.ntp.org.

No idea why we are still asking whether popcon should be enabled or not because
apparently it's 2021 and it's okay to tell others out there that I just
installed Debian.

signature.asc
Description: signature

Re: New service: https://debuginfod.debian.net

[Johannes Schauer Marin Rodrigues]

Quoting Ian Campbell (2021-02-24 18:50:39)
> What are the security implications for users/clients of using this or more
> importantly enabling it by default?
> 
> Presumably clients have to trust that the server is not going to feed
> them malicious debug info. Are the tools which consume this information
> written to operate on completely untrusted inputs? It seems like many
> of them could have been written historically with the assumption that
> their inputs are mostly to be trusted. I suppose the use https helps
> mitigate this at least a bit when it comes to a debian.{org,net}
> service.
> 
> What about information leakage? apart from debugids does this leak
> anything else to the server? On a quick look it seems like it might
> potentially leak source code paths (at least the leaf bits) to things
> being debugged -- does this mean that if a user is debugging private
> software (perhaps unpublished or perhaps proprietary software for
> $work) on a Debian system they are at risk of leaking the source
> filenames if they run gdb on one of their binaries while debugging?  This
> might be a problem if it comes to enabling this transparently.

This sounds like it should be treated in a similar way as we treat submissions
to popcon.debian.org where we ask the user explicitly and which is not getting
enabled unless with explicit consent by the user.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Split Packages files based on new section "buildlibs"

[Johannes Schauer Marin Rodrigues]

Quoting Andrej Shadura (2020-11-10 10:27:44)
> On Tue, 10 Nov 2020, at 08:28, Paul Wise wrote:
> > > The current proposal is to reduce the main Packages.xz files size by
> > > splitting[4] out all of the packages that are not intended for users,
> > > writing those into an own file. Those packages would have a section of
> > > "buildlibs", independent of their other properties.
>  
> > Should (almost?) everything in the existing libdevel section move to
> > the new buildlibs section?
> 
> I wouldn’t say so.
> 
> If I install, say, libftdi-dev, I expect to be able to do actual development
> with it, for Debian or not. In fact, installing libftdi-dev would be the
> first thing I do if I were to develop with the library.
> 
> On the contrary, if I’m going to do some development with, say, clap (Rust
> command-line arguments parser), I wouldn’t install librust-clap-dev; more
> than that, if I actually did, I’d be very difficult for me to actually use it
> to develop an app.

Aha. Would it?

I have the following in my ~/.cargo/config.toml:

[source.deb]
directory = "/usr/share/cargo/registry"

[source.crates-io]
replace-with = "deb"

[net]
offline = true

Then I clone some upstream git repo that uses clap:

git clone https://github.com/dotenv-rs/dotenv.git

And then I run "cargo build". Every time I get a message like:

error: no matching package named `foo` found

I install librust-foo-dev until finally:

Parent pid 535147, child pid 535148
Child process initialized in 30.93 ms
   Compiling proc-macro2 v1.0.18
   Compiling unicode-xid v0.2.0
   Compiling syn v1.0.12
   Compiling dotenv v0.15.0 (/tmp/dotenv/dotenv)
   Compiling quote v1.0.7
   Compiling proc-macro-hack v0.5.9
   Compiling dotenv_codegen_implementation v0.15.0 
(/tmp/dotenv/dotenv_codegen_implementation)
   Compiling dotenv_codegen v0.15.0 (/tmp/dotenv/dotenv_codegen)
Finished dev [unoptimized + debuginfo] target(s) in 9.93s

Parent is shutting down, bye...

So how is this "very difficult"? The steps are the same as when I clone a
Python upstream git repo and I get the message "ModuleNotFoundError: No module
named 'foo'" -- I just install python3-foo and it will work. Same here with
rust and the librust-foo-dev packages.

I do not deny that many upstreams will tell developers to use cargo, pip, go
get... whatever to manage their software. Personally, I rather avoid using
these package managers and use the packages provided by Debian instead. There
are real advantages over using the packages from Debian. Instead of relying on
another 3rd party repository where anybody can upload anything, I benefit from
the additional work put in by DDs to make sure that no garbage ends up in the
archive. Rather than the "fast-paced" development style that seems to be modern
these days, I prefer the stability and security that I get by sourcing all my
code and libraries from the Debian archive.  With Debian packages, I cannot
really get typosquatted, I know that all software is DFSG-free and I know that
I will receive security updates.

This is not an argument against splitting "main" into several archives. I'm no
big fan of this solution but maybe it should be done. What I want to make an
argument for in this email though is, that I do not believe that our packages
have no value outside compiling other Debian packages even for languages where
upstream prefers its users to use their own package manager. There are many
reasons to prefer the trusted Debian sources for quality, security and software
freedom and if any split is done, it should not be made in a way that makes it
harder for our users to install those packages. They have real value.

At this point let me also give a big thank you to all the nodejs, python, rust
etc package maintainers for taking on the tedious task of making Debian
packages for software that is already present in another repo. It's really
great stuff -- thanks a lot!

cheers, josch

signature.asc
Description: signature

Re: base-passwd marked as Essential: yes but other packages depend on it being configured.

[Johannes Schauer Marin Rodrigues]

Quoting Tim Woodall (2021-02-21 20:07:17)
> What I'm trying to do is build in a fakechroot. My idea was to unpack the
> core packages, fakeroot fakechroot chroot image apt-get install apt and have
> a base system to install build-deps etc.
> 
> However, apt does not configure packages in the needed order due to this
> passwd/group issue.
> 
> (debootstrap does not support fakechroot minimal system and I thought
> debootstrap was perl so I didn't want to try to modify it)

Are you by any chance trying to do the same as:

mmdebstrap --variant=apt --mode=fakechroot unstable /path/to/chroot

Thanks!

cheers, josch

signature.asc
Description: signature

Re: base-passwd marked as Essential: yes but other packages depend on it being configured.

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Tim Woodall (2021-02-21 18:22:19)
> base-passwd is marked as Essential: yes
> 
> However, it actually creates the initial passwd and group files in the preinst
> script.

this reminds me of:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=963788

Would this problem not also be solved by dpkg taking care of managing
/etc/passwd and /etc/group? If it could do that, we could solve the
reproducibility issue in the bug above *and* base-passwd would not anymore have
to do anything in its preinst script but could rely on dpkg doing the right
thing, maybe helped by some new declarative statements in the package?

Since dpkg 1.20.0, software like debootstrap or mmdebstrap do not anymore need
to create /var/lib/dpkg/* or /etc/dpkg/dpkg.cfg.d/ because dpkg will take care
of it. So maybe another approach would be to also let dpkg take care of the
initial bootstrapping stuff like making sure /etc/passwd has some sensible
content so that software like debootstrap and mmdebstrap can rely on it?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Debian Policy 4.6.0.0 released

[Johannes Schauer Marin Rodrigues]

Quoting Sean Whitton (2021-08-18 22:21:15)
> On Wed 18 Aug 2021 at 11:10AM +02, Aurelien Jarno wrote:
> 
> >> 9.1.1
> >> No package is allowed to install files in ``/usr/lib64/``. Previously,
> >> this prohibition only applied to packages for 64-bit architectures.
> >
> > This path is used by the multilib 64-bit toolchain on 32-bit
> > architectures, for instance libc6-amd64:i386, or a few essential
> > libraries like ncurses.
> >
> > What kind of fix do you expect on the packages? Is it finally the time
> > to get rid of multilib and use pure multiarch instead?
> 
> We did not intend to make any packages buggy with this change.  It was
> thought to be just a clarification.
> 
> Russ, perhaps we should revert this?

Or maybe it is finally the time to get rid of multilib and use pure multiarch
instead?

signature.asc
Description: signature

Re: Q: Use https for {deb,security}.debian.org by default

[Johannes Schauer Marin Rodrigues]

Quoting Jeremy Stanley (2021-08-20 18:34:22)
> > If so, I think the next step would be to open a bug with a summary of this
> > discussion.  I'm happy to do that, but I'm not sure what package owns this
> > configuration.
> It's not owned directly by a particular package, I think D-I and various
> bootstrapping tools independently write it at installation, so the "fixes"
> for this are likely to be in a variety of places.

As the author of one of these "bootstrapping tools" I would appreciate if we
had the information about which distro and release would use which mirror
address stored somewhere in a machine readable format. The switch of the
security mirror line for bullseye or later would've been much easier if we
wouldn't have to fix all tools hardcoding it...

Maybe distro-info-data could carry such information?

signature.asc
Description: signature

Re: Require packages to build without any configured DNS

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Mattia Rizzolo (2021-09-06 16:39:39)
> As the pbuilder maintainer, I've been asked to make it serve a non-working
> /etc/resolv.conf just to make that bug above moot, so I'm quite biased on the
> matter myself :)

sbuild already disables network access for all chroot backends that support it.

Schroot, the default chroot backend, currently doesn't allow this. See #802849.

The only chroot backend that allows disabling the network is the unshare
backend. It does so, by unsharing the network namespace, only bringing up the
loopback interface and writing an empty /etc/resolv.conf.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Require packages to build without any configured DNS

[Johannes Schauer Marin Rodrigues]

Quoting Mattia Rizzolo (2021-09-14 15:34:36)
> On Tue, Sep 14, 2021 at 10:05:01AM +0200, Johannes Schauer Marin Rodrigues 
> wrote:
> > Hi,
> > 
> > Quoting Mattia Rizzolo (2021-09-06 16:39:39)
> > > As the pbuilder maintainer, I've been asked to make it serve a non-working
> > > /etc/resolv.conf just to make that bug above moot, so I'm quite biased on 
> > > the
> > > matter myself :)
> > 
> > sbuild already disables network access for all chroot backends that support 
> > it.
> 
> As several people already stated, this is *not* about network access.

Yes, I mention it for context.

> > Schroot, the default chroot backend, currently doesn't allow this. See
> > #802849.
> 
> Likewise pbuilder, yes.
> 
> > The only chroot backend that allows disabling the network is the unshare
> > backend. It does so, by unsharing the network namespace, only bringing up 
> > the
> > loopback interface and writing an empty /etc/resolv.conf.
> 
> So you ship an *empty* /etc/resolv.conf?  Then I suppose you also can't build
> packages using dnspython in their tests with your configuration?

Correct. This currently fails:

sbuild -d unstable --chroot-mode=unshare python-oslo.rootwrap

The error message is the same as for the package mentioned in #989171, namely:

https://tests.reproducible-builds.org/debian/rb-pkg/bullseye/amd64/python-oslo.rootwrap.html

This is why I'm writing about sbuild. I wonder if it's a bug for sbuild to
write out an empty /etc/resolv.conf.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: dirsearch: A CLI tool designed to brute force directories and files in webservers

[Johannes Schauer Marin Rodrigues]

Hi,

On Thu, 23 Sep 2021 16:29:27 +0800 clay stan  wrote:
> Package: wnpp
> Severity: wishlist
> Owner: clay stan 
> X-Debbugs-Cc: debian-devel@lists.debian.org
> 
> * Package name:  dirsearch
>   Version : 0.4.2
>   Upstream Author : maurosoria
> * URL : https://github.com/maurosoria/dirsearch
>   License : GPL-2

please improve your ITP bugs. You have already been told in #987335 and #987336
that you should add descriptions to your ITPs. But still you keep filing ITP
bugs without them: #986783, #990386, #991934, #992067, #993914

This is not okay. Please read up on how to file a proper ITP here:

https://www.debian.org/doc/manuals/developers-reference/pkgs.en.html#new-packages

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#1002590: ITP: wayvnc -- VNC server for wlroots-based Wayland compositors

[Johannes Schauer Marin Rodrigues]

Package: wnpp
Severity: wishlist
Owner: Johannes Schauer Marin Rodrigues 
X-Debbugs-Cc: debian-devel@lists.debian.org, jo...@debian.org

* Package name: wayvnc
  Version : 0.4.1
  Upstream Author : Andri Yngvason
* URL : https://github.com/any1/wayvnc
* License : ISC
  Programming Lang: C
  Description : VNC server for wlroots-based Wayland compositors

This is a VNC server for wlroots-based Wayland compositors. It attaches
to a running Wayland session, creates virtual input devices, and exposes
a single display via the RFB protocol. The Wayland session may be a
headless one, so it is also possible to run wayvnc without a physical
display attached.

Re: dash and hidden bashisms in configure scripts

[Johannes Schauer Marin Rodrigues]

Quoting Richard Laager (2021-11-10 20:16:31)
> Can we just enable LINENO in dash then, let the other packages FTBFS, and
> people who care about the packages that FTBFS can either:

in the best case, a package will FTBFS but it can also happen that the
configure script will just do something different without failing. The right
way to approach this if we want LINENO support in dash would be to rebuild all
packages affected by this with and without dash with LINENO support and check
if those that do not FTBFS still build bit-by-bit reproducibly.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: merged-/usr transition: debconf or not?

[Johannes Schauer Marin Rodrigues]

Quoting Russ Allbery (2021-11-20 18:22:27)
> * Create a new essential package that contains these symlinks and that
>   needs to be unpacked before any binaries are executed in the target file
>   system.  This has many of the advantages and drawbacks of the approach
>   of putting the symlinks in base-files, but may make it easier to handle
>   the upgrade problem.  Ideally an upgrade would then involve installing
>   usrmerge, letting it run, and then installing this new essential package
>   so that it takes over ownership of those symlinks.
> 
>   This still feels kind of complex and messy to me, but maybe less so.
> 
> * Create the symlinks directly in the bootstrapping script.  In other
>   words, after unpacking base-files, the bootstrapping script would
>   directly create the required symlinks in the target file system, before
>   unpacking any other package.
> 
>   This has the obvious drawback of moving things outside the packaging
>   system and creating a new special case that every bootstrapping script
>   needs to be aware of (and I know we have at least four or five that
>   would need modifications).  It has the advantage that the usrmerge
>   symlinks are now not in the dpkg database and thus not subject to
>   rewriting, and therefore won't need to be special-cased.  However, that
>   comes with the obvious disadvantage that they're not in the dpkg
>   database, so for instance dpkg -S /lib wouldn't find that symlink unless
>   it was added as some sort of dpkg-query special case (which doesn't seem
>   like a great idea).
> 
>   The advantage of this approach is that it closely mimicks what's already
>   happening now with the usrmerge package, and for which we therefore have
>   a lot of existing experience.

Please don't. Since 2014 it is possible to create a Debian chroot without
taking care of the unpack order and without running any special setup
beforehand that cannot be put into apt or dpkg itself (like creating
/var/lib/dpkg/info which is not needed anymore since dpkg 1.20.0). One of my
biggest motivations in writing mmdebstrap was to get rid of all the quirks that
currently live in debootstrap, move them into apt and dpkg so that we end up
with a declarative bootstrap process that is driven by the package contents,
their metadata and maintainer scripts but does not need magic from the outside
other than standardized interfaces. Requiring the merged-/usr setup to be done
by the bootstrap script would be a step backwards. Think about the following
points:

 * you create a Debian unstable chroot using snapshot.d.o -- now you have to
   somewhere put code that identifies that this is a chroot from the past and
   that has to decide whether that chroot should be merged-/usr or not. If
   merged-/usr setup was living somewhere in the Essential:yes package set
   the chroots would be created the right way automatically
 * same for derivatives -- now every bootstrap script has to learn which
   derivative defaults to merged-/usr, which ones do not and when they switched
 * we somehow need to convert installations of users that only run testing
   or unstable but never do full stable installation to merged-/usr -- one way
   to do that is to let one package in the Essential:yes set drive the
   conversion at which point the bootstrap script doesn't need to know about
   merged-/usr because the Essential:yes set knows how to set it up
 * think about it from a software engineering point of view: Debian as a
   component based software distribution should move specific functionality
   into its packages and global functionality into declarative metadata.

Just as we are currently trying to reduce maintainer scripts and replace them
by declarative approaches, we should also try to reduce the magic in our
bootstrap scripts and move it either to our package managers or replace them by
declarative solutions. We have been moving into the right direction during the
last few years as apt and dpkg have picked up functionality I had earlier put
into mmdebstrap (thanks a lot to guillem, DonKult and juliank for picking up my
patches or writing the functionality themselves). It would mean a step
backwards if all bootstrap scripts would have to carry the setup_merged_usr()
function from debootstrap and then decide by some $magic when this function
should be executed or not.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Mapping Reproducibility Bug Reports to Commits

[Johannes Schauer Marin Rodrigues]

Hi Muhammad,

others already explained how packaging VCS are (sadly) basically a free-for-all
in Debian and that you will probably not get anything better than some
heuristics.  I wanted to add some more ideas to the ones that were already
presented. So in addition to what was already said you can also try any of the
following:

1. If the packaging is on salsa and the commit contains a "closes: " line,
   then the bug will contain a message like this one which will let you
   directly identify the commit that fixed the bug:
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=907352#39

2. If the changelog entry only closes the reproducible bug and nothing else,
   then you can use snapshot.d.o and then debdiff the version that closed the
   bug with the version before that. This method will work even for packages
   that are not using any VCS.

3. If the changelog closes multiple bugs but also points out *who* closed the
   reproducible bug and that person changed nothing else according to
   d/changelog then it's also easy to find the commit. This of course only
   works if the package does use a VCS and if your tools can detect and
   understand the specific packaging style that was used.

4. There were GSoC projects involving reproducible builds. For example Maria
   Valentina Marin Rodrigues contributed back in 2015 and if you find a commit
   of her in packaging repos it will be fixing a reproducible builds bug. There
   might be more GSoC students for which you can apply a similar approach.

Just my 2c.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#999868: ITP: openstreetmap-carto -- standard OpenStreetMap Mapnik stylesheet

[Johannes Schauer Marin Rodrigues]

Package: wnpp
Severity: wishlist
Owner: Johannes Schauer Marin Rodrigues 
X-Debbugs-Cc: debian-devel@lists.debian.org, jo...@debian.org

* Package name: openstreetmap-carto
  Version : 5.4.0
  Upstream Author : Andy Allan 
* URL : https://github.com/gravitystorm/openstreetmap-carto
* License : CC0
  Programming Lang: CartoCSS
  Description : standard OpenStreetMap Mapnik stylesheet

This package provides the standard OpenStreetMap stylesheet for Mapnik,
built from the CartoCSS source. It also provides the necessary icons,
and the script to download the necessary shapefiles.

This package does not provide a tileserver, or perform tile rendering.

openstreetmap-carto got removed this March because a new version could
not be packaged as nodejs dependencies carto-css, chroma-js and hsluv
were missing.  I now packaged carto-css, chroma-js and hsluv and all
three are already in NEW. I packaged the new upstream version of
openstreetmap-carto and verified that it is working correctly:
https://salsa.debian.org/debian-gis-team/openstreetmap-carto

Re: Remove packages from NEW queue?

[Johannes Schauer Marin Rodrigues]

Quoting Tobias Frost (2021-11-18 10:38:40)
> (speculatinng on the why you want it rejected: if you want to replace it with 
> e.g. a newer version, you can just upload the new version)

slightly related question: if I upload a new version to NEW, will the Age of
the package be reset? I'm asking because my package has been in NEW for four
months already and I'd like to avoid loosing that place by an upload of a new
upstream version.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: sbuild/schroot: need to get .deb files internally downloaded as package dependencies

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Uladzimir Bely (2021-10-28 10:58:26)
> When building a package with sbuild, some packages (dependencies of package 
> being built) are internally downloaded and installed by apt. After the build 
> finished, schroot is cleaned again (while everything is done in overlay)
> 
> I need to cache outside of schroot these .deb files that were downloaded by 
> apt. They are supposed to be used for creaing a local partial debian 
> repository, so that the second build will use this local repo instead of 
> internet one.
> 
> Currently it's done by parsing dependencies and downloading them 
> independently, but I suppose there should be some more straightforward way to 
> get them using sbuild.
> 
> I investigated and tried several options for sbuild (like 
> --purge-build=never, 
> --purge-deps=never, --purge-session=never), tried to find .deb files during 
> the build using at some stages (--pre-build-commands="ls -la /var/cache/apt/
> archives/", --starting-build-commands="ls -la /var/cache/apt/archives/", --
> finished-build-commands="ls -la /var/cache/apt/archives/") - but I couldn't 
> see any .deb files inside the schroot.
> 
> I previously asked in debian-user maillist 
> (https://lists.debian.org/debian-user/2021/10/msg01044.html), but the tips I 
> got (like using caching proxy) are 
> not what I really need. While sbuild already does the thing (downloading/
> installing dependencies), I suppose there should be a way to (at least) see
> the *.deb files downloaded in '/var/cache/apt/archives' inside schroot.

sbuild maintainer here. You have not explained why you don't want to use a
caching proxy like apt-cacher-ng. But using a caching proxy is also what I
would suggest you use.

If you don't want to use that, here is how you download only the packages you
need to satisfy the build depenencies of a particular source package:

cat << END > /tmp/apt.conf
Apt::Architecture "amd64";
Dir "/tmp/apt";
Dir::Etc::trusted "/etc/apt/trusted.gpg";
Dir::Etc::trustedparts "/etc/apt/trusted.gpg.d/";
Apt::Get::Download-Only true;
Apt::Install-Recommends false;
END
mkdir -p /tmp/apt/etc/apt/apt.conf.d/ /tmp/apt/etc/apt/sources.list.d/ 
/tmp/apt/var/lib/dpkg /tmp/apt/var/cache/apt/archives/partial 
/tmp/apt/etc/apt/preferences.d/
touch /tmp/apt/var/lib/dpkg/status
cat << END > /tmp/apt/etc/apt/sources.list
deb http://deb.debian.org/debian/ unstable main
deb-src http://deb.debian.org/debian/ unstable main
END
APT_CONFIG=/tmp/apt.conf apt update
APT_CONFIG=/tmp/apt.conf apt-get dist-upgrade
APT_CONFIG=/tmp/apt.conf apt-get build-dep hello

Now all your packages to build src:hello are in
/tmp/apt/var/cache/apt/archives. No need to waste CPU cycles on a full package
build if all you want is to download a certain set of binary packages.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Lintian and Dpkg's :any multiarch qualifier

[Johannes Schauer Marin Rodrigues]

Hi Felix,

Quoting Felix Lechner (2021-11-04 20:25:28)
> On Thu, Nov 4, 2021 at 2:35 AM Johannes Schauer Marin Rodrigues 
>  wrote:
> > it seems the only tool that calls :any an "acceptor" is lintian
> 
> I think the multiarch spec is confusing because of its terminology.
> It's been a hurdle for many people.

I think inventing new terminology that is not used anywhere else just adds
confusion. I agree that terminology can be confusing but this can be fixed by
either

 * adding better documentation for the existing terminology, for example to
   wiki pages, Debian policy or the upcoming Multiarch Specification shipped by
   dpkg [1] or by

 * changing the terminology such that all the involved tools use the same
   terminology

I don't think just Lintian starting to use new terminology that is not used by
dpkg, apt or the wiki pages is helpful at all. Quite the opposite. I think this
just makes the status quo worse.

Thanks!

cheers, josch

[1] 
https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/tree/doc/multiarch.txt?h=pu/doc-multiarch-spec

signature.asc
Description: signature

Re: d/watch for Gitea's new dynamic pages

[Johannes Schauer Marin Rodrigues]

Quoting Felix Lechner (2021-11-04 20:17:49)
> For anyone having issues with Gitea's new dynamic download links [1] here is
> a debian/watch file that works.
> 
> * * *
> 
> version=4
> opts=\
>  downloadurlmangle=s/\/releases\/tag\/(\d\S+)$/\/archive\/$1\.tar\.gz/,\
>  filenamemangle=s/.+\/tag\/(\d\S+)$/foot-$1\.tar\.gz/ \
> https://codeberg.org/dnkl/foot/releases .*/releases/tag/(\d\S+)

Not all projects use releases. Here is a watch file that works with git tags
which also use data-url attribute instead of href:

version=4
opts=downloadurlmangle=s{/src/tag/(@ANY_VERSION@)}{/archive/$1.tar.gz},\
filenamemangle=s{.*/(.*)}{@PACKAGE@-$1.tar.gz} \
https://gitlab.mister-muffin.de/josch/@PACKAGE@/tags 
/josch/@PACKAGE@/src/tag/([0-9.]+)

signature.asc
Description: signature

Re: Lintian and Dpkg's :any multiarch qualifier

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Felix Lechner (2021-11-03 23:34:17)
> For a brief time between October 1 and October 15, Lintian gave potentially
> confusing advice on some build prerequisites. [1]
> 
> The :any multiarch acceptor—a rarely used feature some other tools call the
> "muliarch qualifier"—

it seems the only tool that calls :any an "acceptor" is lintian while the rest
of the world calls it a qualifier. I filed a MR to fix this:

https://salsa.debian.org/lintian/lintian/-/merge_requests/378

> was originally not implemented at all [2] and then implemented incorrectly.
> [3] Many people do not even know about the feature.  To my knowledge it works
> now.
> 
> Here are two questions:
> 
> 1. Did anyone find the latest Lintian versions (2.109.0 and up)
> confusing as to whether the :any should be included? The material you
> would have encountered includes both the context offered by Lintian
> (the extra information after the tag) and any relevant tag
> descriptions.
> 
> 2. Should Lintian issue any advice when it sees the :any multiarch
> acceptor? If so, for which packages? It might allow maintainers to
> undo erroneous advice they may have been given, although many folks use the
> feature legitimately, as well.

which wording do you need advice on? I grepped the Lintian git for tags
containing the string :any and still wasn't sure what you are exactly asking
about.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: merged /usr considered harmful (was Re: Bits from the Technical Committee)

[Johannes Schauer Marin Rodrigues]

Quoting Michael Biebl (2021-07-19 15:10:42)
> Am 19.07.21 um 03:36 schrieb Guillem Jover:
> > What I've also said multiple times, is that
> > merged-usr-via-moves-and-symlink-farms could have been implemented in
> > a fully automated way, by debhelper, w/o requiring any maintainer scripts,
> > all with full cooperation and managed by dpkg, with .debs shipping
> > actual tracked pathnames
> I'm convinced this view is way too naive and not implementable in 
> practice (and yes, openSUSE is a data point that confirms that)
> 
> What you propose is, that each and every package does its /usr-merge 
> transition on its own. This only works, if packages are independent 
> (enough) so this actually works.
> Unfortunately this is not the case. Take PAM for example. You can't just 
> recompile src:pam and have debhelper automatically move all files to 
> /usr. This would break all packages that install a PAM plugin. You have 
> a transition here, involving many packages.
> Same is true for udev rules, systemd service files, basically every 
> package that provides interfaces/hooks to other packages is affected.
> So it's not that simple unfortunately. You can't fully automate that.
> 
> According to
> apt-file search -x '^/(lib|bin|sbin)'
> on my Debian sid/amd64 system, we have 1747 packages shipping 24583 
> files in those directories.

more precisely, on amd64 unstable:

/bin 247 files
/lib{32,64,x32} 83 files
/lib/firmware 2379 files
/lib/live 115 files
/lib/modules 17500 files
/lib/systemd 2221 files
/lib/udev 614 files
/lib/x86_64-linux-gnu 343 files
/lib/* 441 files
/sbin 547 files

So most files come from /lib/modules, where only 14 packages are involved,
/lib/systemd which will be fixed by an update to dh_installsystemd, and
/lib/firmware where only 36 packages are involved. The remainder then doesn't
sound so scary anymore as it only involves 656 unique packages and not 1747.
And again many of those remaining packages will be fixed by an update to
debhelper, correct? Given that 90% of source packages use dh, that would reduce
the number to a very manageable size.

> There are *many* such entangled transitions 
> hidden in there, so I fear this is not manageable.
> As Luca pointed out, even distros with a much stricter governance model 
> were not able to do that.
> 
> The /usr-merge transition as described and decided on in the TC bug, 
> seems to me is the only viable way forward.
> 
> Yes, it does break dpkg -S, but your idea of using a list of mapped 
> paths as in [1] seems like an entirely reasonable approach to solve this.
> 
> Once we have this global switch to merged-usr, packages can bit by bit, 
> completely independent, update their debian/rules to use --prefix=/usr 
> and after a few years, we don't have any packages anymore installing any 
> files in /. We could aid this process with a lintian check that flags
> packages that install files in /(sbin|bin|lib)/.

So what what is actually the roadmap after the bullseye release? What is the
way forward? Should I rather file bugs with patches against individual packages
to move their files from /(sbin|bin|lib)/ to /usr/(sbin|bin|lib)/ or do we
already have a debhelper patch to do that move for us?

This would mean that we only have to bear with the problems mentioned by
guillem until that move is complete, correct?

And another question: once there are no more files shipped by any package in
/(sbin|bin|lib)/ we can let base-file create the symlinks?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: proposed MBF: packages still using source format 1.0

[Johannes Schauer Marin Rodrigues]

Hi Adrian,

Quoting Adrian Bunk (2022-03-07 22:42:42)
> On Sun, Mar 06, 2022 at 09:25:45PM +0100, Lucas Nussbaum wrote:
> >...
> > I think that we should reduce the number of packages using the 1.0 format, 
> > as
> > (1) format 3.0 has many advantages, as documented in
> > https://wiki.debian.org/Projects/DebSrc3.0 ; (2) this contributes to
> > standardization of packaging practices, lowering the bar for contributors to
> > contribute to those packages.
> >...
> 
> You are not making a compelling case that these benefits clearly 
> outweight the substantial costs.
> 
> Such a MBF also:
> (1) causes a lot of extra work, and
> (2) causes a lot of breakage because such larger packaging changes
> are rarely done as careful as would be necessary
> 
> When people are making invasive packaging changes like a dh compat bump 
> or change the packaging due to such a MBF we often end up with bug 
> reports like #1000229 where something broke due to that (empty binary 
> packages are among the more typical breakages).
> 
> Unless a compelling case is made that the benefits of a MBF clearly 
> outweight these drawbacks, such MBFs usually have a negative benefit.

you are absolutely right! So what happens if we address both of these issues
by:

 (1) providing a patch and thus remove "a lot of extra work" being necessary
 (2) show that the package produces bit-by-bit identical binary packages after
 the change compared to before by  using reproducible builds and thus
 prevent "a lot of breakage"

I did exactly that and rebuilt all the packages found by Lucas with the
following changes:

$ mkdir -p debian/source
$ echo '3.0 (quilt)' >debian/source/format

141 source packages produce bit-by-bit reproducible binary packages after
applying this change:

 abootimg, acpi, amideco, asmix, aspell-el, avfs, aview, avrp, aylet,
 bfbtester, bindfs, cconv, cdde, cereal, citadel-client, cl-log, coco-cpp,
 cycfx2prog, daemontools, dbix-easy-perl, debaux, dictconv, ding-libs, dirdiff,
 dns323-firmware-tools, dv4l, famfamfam-flag, festlex-poslex,
 festvox-kallpc16k, festvox-kallpc8k, festvox-kdlpc16k, festvox-kdlpc8k,
 flvstreamer, fortunes-bofh-excuses, gav-themes, gcc-3.3, glbsp, glktermw,
 glslang, glurp, gnomediaicons, gtkterm, imaprowl, imgvtopgm, jdresolve,
 jtex-base, judy, kawari8, lakai, libalgorithm-dependency-perl,
 libanyevent-serialize-perl, libapache2-mod-log-slow, libaudio-scrobbler-perl,
 libbiblio-isis-perl, libcitadel, libclass-csv-perl, libclass-pluggable-perl,
 libcrypt-smbhash-perl, libdansguardian-perl, libdata-javascript-anon-perl,
 libdatapager-perl, libdata-validate-domain-perl, libdbix-dr-perl,
 libebook-tools-perl, libemail-foldertype-perl, libfile-searchpath-perl,
 libhtml-element-extended-perl, libhtml-popuptreeselect-perl,
 libimage-metadata-jpeg-perl, libjcode-pm-perl, libjlayer-java, libjmac-java,
 liblog-dispatch-filerotate-perl, libmodem-vgetty-perl, libmp4-info-perl,
 libnbcompat, libnet-finger-perl, libnet-proxy-perl, libropkg-perl,
 libtemplate-plugin-cycle-perl, libtemplate-plugin-utf8decode-perl,
 libtext-aligner-perl, libtext-table-perl, libtie-shadowhash-perl,
 libtimezonemap, libtree-multinode-perl, libunibreak, libuser-perl,
 libyaml-shell-perl, m16c-flash, manpages-tr, mapivi, midge, moblin-gtk-engine,
 mpclib3, ng-utils, nomarch, ocamlcreal, ocaml-getopt, ocaml-magic,
 ocaml-shout, osspsa, pandora-build, pcaputils, pdf2svg, pfqueue, phnxdeco,
 pidgin-awayonlock, pngmeta, pngnq, powerman, pscan, qprint, randtype, redet,
 ruby-bsearch, sbox-dtc, scriptaculous, sipcalc, speex, spirv-headers, src2tex,
 ssh-askpass-fullscreen, stx2any, sylseg-sk, tcsh, tor, ufiformat,
 uzbek-wordlist, vanessa-adt, vanessa-logger, watchdog, wayland, weston, xinit,
 xserver-xorg-video-nouveau, xserver-xorg-video-qxl, xsettings-kde,
 xtide-coastline, yaret, zmakebas

Then for the remaining packages we apply these changes and rebuild again:

$ mkdir -p debian/source
$ echo '3.0 (quilt)' >debian/source/format
$ echo single-debian-patch >debian/source/options

An additional 223 source packages produce bit-by-bit reproducible binary
packages after applying this change:

 4g8, aconnectgui, aft, alsamixergui, appconfig, ascdc, asmail, awardeco,
 baycomusb, beav, binstats, blop, catdvi, cbmplugs, cd-circleprint, chase,
 chrpath, ciphersaber, cl-regex, coco-cs, coco-java, code2html, codegroup,
 compartment, console-cyrillic, coolmail, cpipe, crack-attack, crypt++el, cvs,
 dbus-sharp-glib, debfoster, dictem, dist, dmitry, dnsmasq, docbook-website,
 dutch, dvdtape, edict-el, eldav, fake, fortunes-it, freebirth, freetable,
 freetds, funnelweb-doc, fuse-umfuse-ext2, fwanalog, g2p-sk, gkrellm-reminder,
 gkrellm-thinkbat, gkrellm-xkb, glw, gmemusage, gplaycli, gss-ntlmssp,
 gwaterfall, gworldclock, hsqldb1.8.0, ifrench-gut, imgsizer, impose+, inn,
 intel2gas, ipv6calc, its-playback-time, jargon, jgraph, kelbt, knews,
 libapache-gallery-perl, libdbd-sybase-perl,
 

Re: Is removing smell from packages OK? (Was: Why? "Marked for autoremoval on 24 March due to xdelta3: #965883")

[Johannes Schauer Marin Rodrigues]

Quoting Andreas Tille (2022-02-25 09:22:38)
> I had to decide between a "proper NMU" and an "upload that fits the packaging
> standards I apply to what I upload" (which includes maintained on Salsa,
> usage of dh, DEP5 copyright ... basically removing the smell from the
> package).  I decided for the latter but at the same time I was aware that I
> violated the rules we gave given each other.
> 
> Given the fact that there was a nearly 4 year old patch (#895957) made
> me feel that I'm not alone with this but on the other hand the creator
> of the patch (thanks Jeremy for doing at least half of the necessary
> work) hesitated to upload his work.  This brings up again the discussion
> about how much changes are allowed to simply remove smell from packages
> is accepted.

Is this not something that can be solved by salvaging [1] the package in
question? Do a tiny NMU fixing an RC bug (and only that) first but then after
waiting 21 days you can get the packaging into shape without your changes being
classified as a NMU and thus without the restrictions we put upon the changes
allowed in a NMU.

Thanks!

cheers, josch

[1] 
https://www.debian.org/doc/manuals/developers-reference/pkgs.en.html#package-salvaging

signature.asc
Description: signature

Bug#1004120: ITP: debianutils-tempfile -- tempfile from debianutils

[Johannes Schauer Marin Rodrigues]

Package: wnpp
Severity: wishlist
Owner: Johannes Schauer Marin Rodrigues 
X-Debbugs-Cc: debian-devel@lists.debian.org, jo...@debian.org

* Package name: debianutils-tempfile
  Version : 4.11
  Upstream Author : Johannes Schauer Marin Rodrigues 
* URL : https://salsa.debian.org/debian/debianutils-tempfile
* License : GPL2+
  Programming Lang: C
  Description : tempfile from debianutils

debianutils is currently blocked from migrating to testing [1] due to
its removal of tempfile from the Essential:yes set [2].

The debianutils maintainer would like to avoid reintroducing tempfile in
debianutils but is willing to add a Depends relationship to another
package that implements tempfile.

I thus just packaged the last version of tempfile from debianutils in
this new source package. The packaging can be found at [3].

This version of tempfile is the same as the last version of tempfile
shipped by debianutils and thus includes the deprecation warning.

The package is supposed to be temporary until all tools moved away from
tempfile (famous last words).

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=996747
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=992399#20
[3] https://salsa.debian.org/debian/debianutils-tempfile

Re: Bug#885698: What licenses should be included in /usr/share/common-licenses?

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Bill Allombert (2023-09-10 18:29:36)
> On Sun, Sep 10, 2023 at 09:00:22AM -0700, Russ Allbery wrote:
> > Jonas Smedegaard  writes:
> > > Quoting Hideki Yamane (2023-09-10 11:00:07)
> > >>  Hmm, how about providing license-common package and that depends on
> > >>  "license-common-list", and ISO image provides both, then? It would be
> > >>  no regressions.
> > 
> > I do wonder why we've never done this.  Does anyone know?  common-licenses
> > is in an essential package so it doesn't require a dependency and is
> > always present, and we've leaned on that in the past in justifying not
> > including those licenses in the binary packages themselves, but I'm not
> > sure why a package dependency wouldn't be legally equivalent.  We allow
> > symlinking the /usr/share/doc directory in some cases where there is a
> > dependency, so we don't strictly require every binary package have a
> > copyright file.
> 
> Or we could generate DEBIAN/copyright from debian/copyright using data in
> license-common-list at build time. So maintainers would not need to manage
> the copying themselves.

I very much like this idea. The main reason maintainers want more licenses in
/usr/share/common-licenses/ is so that they do not anymore have humongous
d/copyright files with all license texts copypasted over and over again. If
long texts could be reduced to a reference that get expanded by a machine it
would make debian/copyright look much nicer and would make it easier to
maintain while at the same time shipping the full license text in the binary
package.

Does anybody know why such an approach would be a bad idea?

I have zero legal training so the only potential problem with this approach
that I was able to come up with is, that then the source package itself would
not anymore contain the license text and thus we would be shipping code covered
by a license that states that the code may only be distributed with the license
text alongside it without that text. So while auto-generating this would
probably create compliant binary packages, it would leave the source package
without the license text. Is that a problem?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: [idea]: Switch default compression from "xz" to "zstd" for .deb packages

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Thomas Goirand (2023-09-19 09:50:45)
> I'm not sure if we should switch to zstd, or if xz will do the work, though
> I'd be delighted if the dpkg performances could be improved. I'm spending all
> of my days installing server, sometimes with 1.5 TB of RAM and 128 core (AMD
> Epyc...), on last gen NVMe, using a local mirror, it's really painful to see
> how slow the debootstrap process is, compared to what it could be. Unpacking
> multiple .deb at the same time seems a very good idea to me, as well as
> parallelizing everything we can.

I was about to say that zdebootstrap by Adam Borowski used to be a thing four
years ago but now I see another commit from two days ago so maybe it's still
alive and usable?

https://git.sr.ht/~kilobyte/zdebootstrap

There is also my package mmdebstrap which gives you a Debian chroot a few times
faster than debootstrap does. Here are some benchmarks from my laptop:

| variant   | mmdebstrap | debootstrap  |
| - | -- |  |
| essential | 9.52 s | n.a  |
| apt   | 10.98 s| n.a  |
| minbase   | 13.54 s| 26.37 s  |
| buildd| 21.31 s| 34.85 s  |
| standard  | 23.01 s| 48.83 s  |

Depending on your use-case you might be interested in the essential or apt
variants which are even faster because they install less stuff than "minbase".

Thanks!

cheers, josch

signature.asc
Description: signature

Re: debvm for autopkgtests with multiple host?

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Helmut Grohne (2023-09-23 20:18:21)
> Quoting Ian Jackson (2023-09-23 12:19:27)
> > To summarise that discussion: at that time the best available solution that
> > worked in ci.d.n seemed to be to write an ad-hoc script to run the tests in
> > qemu; three packes had done that, each separately, with complex scripts
> > with many moving parts.

the three packages are probably sbuild, dropbear and cryptsetup?

> > I saw debvm, and wondered if it was suitable for this purpose.  But, then I
> > looked at its debian/test/control and I see that the tests are marked as
> > flaky.[2]  So maybe it isn't reliable enough :-/.
> 
> The reliability of tests is ok. The reason for marking them flaky is that
> they currently test the "wrong" packages. ci.d.n sets up chroots in a
> delicate way to combine particular packages to see which combinations cause
> breakage. Then debvm just creates an unstable system and tests that. In
> effect, it currently tests unstable (inside those virtual machines) rather
> than what it is supposed to be testing.
> 
> Johannes solved this problem on the mmdebstrap side and mmdebstrap's
> tests no longer are flaky in this way. Therefore this should be solvable
> on the debvm side. I just haven't gotten do figuring out the right runes
> thus far. Roughly speaking, the hosts' apt configuration, pinning and
> sources.lists should be used inside the created virtual machine.

right now the mmdebstrap autopkgtest only mimics sources and pinning of the
outside system. I had not considered that apt configuration on salsaci or debci
was set to something that influenced the tests. Is the apt configuration on
those systems set to something that is not the default and should be considered
as well?

There is really not much magic. The core of it is to pass this to your
mmdebstrap or debvm-create invocation:

--setup-hook='for f in /etc/apt/sources.list /etc/apt/sources.list.d/* 
/etc/apt/preferences.d/*;
  do [ -e "$f" ] && { echo; sed "s| file://| copy://|" "$f"; } 
| tee "$1/$f" >&2; done'
--hook-dir=/usr/share/mmdebstrap/hooks/file-mirror-automount

The first will hook will make sure that the chroot receives the sources and
pinning values of the outside system. The second will do some bind-mount magic
which allows the chrooted processes to access even file:// repositories from
outside the chroot. Full script here:

https://sources.debian.org/src/sbuild/0.85.3/debian/tests/unshare-qemuwrapper/

> There is another practical problem. None of the autopkgtest nodes support
> kvm. Emulation will always use tcg. For one thing, tcg is slow.  It can be so
> slow on some architectures that RCU becomes unhappy as its grace periods
> become too long. For another, tcg is buggy. It has emulation bugs even on
> release architectures that make some expected functionality fail. For
> instance, gdb reliably segfaults when run in s390x tcg emulation.

Architectures that are not amd64 or arm64 are a common source of problems. The
current sbuild autopkgtest hits another issue when running qemu on s390x:

514s ^M[   28.399829] illegal operation: 0001 ilc:1 [#1] SMP 
514s ^M[   28.400490] Modules linked in: chacha_s390(+) libchacha virtio_pci 
virtio_pci_legacy_dev virtio_pci_modern_dev virtio_blk
514s ^M[   28.402977] CPU: 4 PID: 163 Comm: cryptomgr_test Not tainted 
6.5.0-1-s390x #1  Debian 6.5.3-1
514s ^M[   28.403150] Hardware name: QEMU 8561 QEMU (KVM/Linux)
514s ^M[   28.403285] Krnl PSW : 0704c0018000 0042 (0x42)
514s ^M[   28.403964]R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 
PM:0 RI:0 EA:3
[...]
514s ^M[   28.408739] Last Breaking-Event-Address:
514s ^M[   28.408758]  [<03ff800fb084>] chacha_crypt_generic+0x54/0xfd0 
[libchacha]
514s ^M[   28.409610] ---[ end trace  ]---

https://ci.debian.net/data/autopkgtest/testing/s390x/s/sbuild/38123402/log.gz

In addition to debvm, these tests should maybe directly depend on qemu so that
they get run on new qemu uploads that trigger these kinds of regressions.
Assuming this is really a qemu problem and not something a new kernel version
introduced...

Thanks!

cheers, josch

signature.asc
Description: signature

Re: allow missing description fields and empty long descriptions for Rust/etc packages?

[Johannes Schauer Marin Rodrigues]

Hi Gard,

Quoting Gard Spreemann (2023-09-20 09:26:58)
> Paul Wise  writes:
> > […] since the Rust packages are basically only used as build-deps and
> > therefore have no human users.
> I just wanted to raise awareness that some of us humans do use librust-*-dev
> packages directly, having put cargo in permanent offline mode and having
> swapped out its cargo.io package source for Debian's packages on the local
> filesystem.

do you have more details on how you achieve this?

I'm using Debian's rust packages instead of creates.io using this wrapper
script to cargo:

https://salsa.debian.org/josch/nocrates.io/-/blob/master/cargowrapper.sh

How do you do it?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Bug#1053165: ITS: nunit

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting David Bremner (2023-09-28 16:40:13)
> Bastian Germann  writes:
> > Source: nunit
> >
> > I intend to salvage nunit with the plan to orphan it in three weeks.
> > Please notify me if you object.
> 
> In my opinion, your repeated "salvaging" of packages in order to orphan
> them is an abuse of the ITS process. Yes, it's a clever procedural hack,
> but Debian is not (supposed to be) about tricking our fellow
> developers. I don't know what best process to do the QA work you want to
> do is; I suspect you should consult the MIA team for suggestions.
> 
> David
> 
> - who co-wrote the ITS process, and knows what it says.

I don't think it's a clever hack. It would be a hack if it at least works
within the given set of rules. But devref specifically says:

"Note that the process is only intended for actively taking over
maintainership.  Do not start a package salvaging process when you do not
intend to maintain the package for a prolonged time."

So salvaging a package with the intention to orphan it is clearly violating the
explicit rules of the salvaging process.

So I do not think whether or not this approach is wrong is a matter of opinion.
It is clearly written down that you should only salvage if you "intend to
maintain the package for a prolonged time".

I like a lot that we have the salvaging process in place. Please do not taint
it by breaking its rules.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: sbuild can't find piuparts even when it's listed in build dependencies

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Jonathan Kamens (2023-09-26 15:20:49)
> I'm trying to use sbuild to build my package, and it's failing to find 
> piuparts:
> 
> | Post Build  
>  |
> +--+
> 
> 
> piuparts
> 
> 
> sudo: piuparts: command not found
> 
> E: Piuparts run failed.
> 
> I tried adding it to Build-Depends but that didn't help.

piuparts is run outside the build chroot, not inside of it.

> My best guess is that the issue here is that piuparts is installed in /sbin
> and /sbin isn't in the default sudo path, but that would imply that there's a
> bug in the build tools rather than that I'm doing something wrong, and I
> think the latter is more likely. ;-)
> 
> I'm on Bookworm.
> 
> Any tips?

do you have piuparts installed outside the chroot?

If you are having problems with sbuild, you can also always file a bug against
sbuild in the BTS.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#1054404: ITP: reform-firedecor -- MNT Reform specific window decoration for Wayfire

[Johannes Schauer Marin Rodrigues]

Package: wnpp
Severity: wishlist
Owner: Johannes Schauer Marin Rodrigues 
X-Debbugs-Cc: debian-devel@lists.debian.org

* Package name: reform-firedecor
  Version : 
  Upstream Contact: Lukas F. Hartmann 
* URL : https://github.com/mntmn/Firedecor
* License : Expat
  Programming Lang: C++
  Description : MNT Reform specific window decoration for Wayfire

An advanced window decoration plugin for the Wayfire window manager.  Provides
pretty standard rectangular frames and title bars with an icon and familiar
minimize/maximize/close buttons.

This is a fork of the original project by AhoyISki which seems abandoned and as
a result, was never ported to wayfire 0.8.0 (currently in experimental). This
fork is maintained by MNT Research GmbH for the MNT Reform laptop (hence the
"reform" prefix in the package name) but is otherwise just a normal wayfire
window decorator.

Re: Hyphens in man pages

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Gioele Barabucci (2023-10-15 17:59:32)
> On 15/10/23 17:33, Iustin Pop wrote:
> > At least you're not lazy. I am, so what I did many times is add a
> > build-depends on pandoc, and write the man page in rst or md. I think
> > that's a worse solution (pandoc is really heavy), but at least, I don't
> > have to go back to *roff.
> 
> Another option for the members of the lazy club is `podlators-perl`.

that is a virtual package provided by perl. What mechanism exactly are you
referring to when mentioning podlators-perl?

> The `.pod` syntax is OK, and it is not as heavy a dependency as pandoc.

When I have to write a new man page I always write POD instead of troff and
then run pod2man on it. I have yet to find something I wanted to put in a man
page that I was unable to express via the POD format.

Another reason I'm a fan of pod2man is, that it's possible to embed POD
documentation into scripts that are not Perl. For example debvm does this and
even though the debvm tools are written in POSIX shell, pod2man is doing the
right thing:

pod2man /usr/bin/debvm-run | man -l -

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Potential MBF: packages failing to build twice in a row

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting John Goerzen (2023-08-13 23:32:03)
> On Sat, Aug 05 2023, Lucas Nussbaum wrote:
> > I wonder what we should do, because 5000+ failing packages is a lot...
> Let's think about the level of trouble we cause trying to tackle something
> that has clearly not bothered anyone for years.

this is not the first time that is has been said in this thread that "this
hasn't bothered anybody for years". I wanted to come out and say that it has
bothered me. It just hasn't bothered me enough to investigate what the proper
way to solve it is. It hasn't bothered me enough to bother other people with
this issue. After all, I can just run "git clean -fdx" to "solve" the problem
whenever it happens.

Lucas filed bugs against three of my packages. Since I didn't know how to
properly solve this issue and since this thread exists I finally felt it was
time to ask my fellow DDs how to fix this and I'll be happy to upload packages
with this problem fixed.

Thank you Lucas for doing this work!

> Personally, I'm a volunteer.  I have X amount of time to devote to Debian.
> I work on Debian because I enjoy it.  It is satisfying!

I'm also just a volunteer doing this in my free time as my hobby because I find
it fun. I can totally understand how fixing this can be anything but fun for
others and I do not want to invalidate your opinion on this. But as this thread
has a number of people saying how they don't find this useful and how they
don't find fixing this fun, I wanted to write a message with my very different
view on the subject. I cannot tell you exactly why but my *feeling* is that I'm
very happy that Lucas filed bugs against three of my packages and I will
personally find it fun to upload a version that fixes them. I understand that
there are many who do not think like that and I understand that it's worth
discussing the practical benefit of it all. But I for one am happy that these
bugs were filed and I will have fun closing them with a fix.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Compiler runs out of memory when building a package

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Dima Kogan (2023-08-13 22:54:48)
> I'm looking for a suggestion to fix a problem.
> 
> I uploaded a package, and it cleared NEW a few days ago. I now see that
> it fails to build on most 32-bit arches becaues the compiler runs out of
> memory.

I ran into the same problem with my package vcmi. The fix I've been using was
to add this to my d/rules:

export DEB_CXXFLAGS_MAINT_APPEND+=--param ggc-min-expand=5

https://sources.debian.org/src/vcmi/1.1.0%2Bdfsg-1/debian/rules/#L20

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Using a build profile on a buildd build

[Johannes Schauer Marin Rodrigues]

Quoting Scott Talbert (2022-06-15 22:30:13)
> Is it possible to instruct the buildds to use a build profile when performing
> an official build (e.g., using nocheck to break a dependency loop)?  If so,
> how?

No, it is not. Main reason is probably that nobody has found the time to
implement it yet.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Bug#1014908: ITP: gender-guesser -- Guess the gender from first name

[Johannes Schauer Marin Rodrigues]

Quoting Steve McIntyre (2022-07-14 13:54:52)
> And I see you uploaded ~immediately - why even bother with an ITP?

I did that quite a few times in the past as well. Is there a rule of how long I
have to wait with my upload to NEW after filing the ITP?

signature.asc
Description: signature

Re: How to get rid of unused packages (Was: proposed MBF: packages still using source format 1.0)

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Andreas Tille (2022-03-21 11:55:09)
> Am Sat, Mar 19, 2022 at 08:37:28PM +0100 schrieb Erik Schanze:
> > Am 16.03.22 um 14:11 schrieb Andreas Tille:
> > > was not uploaded by its maintainer for >10 years.
> > 
> > Yes, because upstream development was finished and packaging was working
> > so far. No need for new uploads IMO.
> 
> My point was that there are teams inside Debian (like reprocucible
> builds or crossbuilding like bug #989953) who file bugs with patches to
> a lot of packages.  I personally think we should somehow help them to spent
> their energy on packages that are worth it.

personally, when doing work that affects multiple packages, I concentrate on
those that are in unstable *and* in testing. In my experience this weeds out
99% of those source packages that I really shouldn't bother with. I heard from
others that they use the same heuristic to spend their QA time on packages that
will likely make it into the next stable release.

During my last round of mass-rebuilds I unfortunately didn't apply this
heuristic and stumbled across src:ants. In contrast to Andreas, I think that
even packages without a maintainer upload for >10 years should *not* be kicked
out of the archive even if their popcon numbers are down to zero.

However, I do not understand why we do not have a mechanism to kick out source
packages like src:ants automatically for good. Not because of its low (and
decreasing) popcon number but because:

 - the last stable release the source package was part of, was stretch
 - its binary package was last installable during the development of buster and
   uninstallable since then
 - the source package has four open RC bugs with the *youngest* from four years
   ago

Why do we carry essentially useless weight around for so many years?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Firmware: Scope of non-free-firmware

[Johannes Schauer Marin Rodrigues]

Quoting Thomas Goirand (2022-05-11 17:14:57)
> > For backwards compatibility, I think that the firmware component is
> > going to need to be a subset of non-free; i.e. packages are going to
> > need to be *copied* not moved from non-free to the firmware component,
> > which means they would be available from both non-free components.

I think that's a good idea as it wouldn't break any setups. The number of
packages in non-free-firmware is probably very small and the package data would
not be duplicated on the mirrors anyways because non-free and non-free-firmware
would both reference the same deb archives in the /pool directory, right?

> A work around would be to have some automation to check if non-free is
> activated, and (propose to) update the sources.list automatically to add
> non-free-firmware. I'd prefer doing this, as having copies of the same
> package in both non-free and non-free-firmware is (IMO) a mess.

Maybe I'm lacking imagination but which approach would you take to do this
reliably? If you go that route, then a heuristic is not enough. You must not
break existing setups and you must also make sure never to get into a situation
where that automation has to bail out or otherwise a system will end up without
non-free-firmware even though it had non-free enabled.

What do you propose?

I'm also curious because I would like to do arbitrary machine-edits of
user-supplied apt sources.list files but so far nothing worked reliably enough.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: autopkgtest/sbuild environment variables: LC_ALL, HOME, XDG_RUNTIME_DIR etc

[Johannes Schauer Marin Rodrigues]

Quoting Simon McVittie (2022-04-29 10:58:34)
> On Fri, 29 Apr 2022 at 08:34:48 +0100, Julian Gilbey wrote:
> > So can I suggest that
> > sbuild-setup(7) explains this a bit more and discusses setting up a
> > meaningful HOME directory?
> 
> I'm sure patches are accepted, but the problem with this is that what you
> want for sbuild does not match what you want for autopkgtest. For sbuild,
> the environment that most closely resembles our real, production buildds
> involves the /etc/schroot/buildd profile, and a uid whose home directory
> is /nonexistent. For autopkgtest, one of the more permissive profiles
> like /etc/schroot/desktop is more realistic, and tests are allowed to
> assume that they run as a user with a real home directory.
> 
> I would personally recommend one of the better-isolated autopkgtest
> backends like -lxc or -qemu for running autopkgtest tests. -qemu doesn't
> need root, and there are proposed patches adding backends that use
> unprivileged user namespaces (-unshare and Podman), which I should
> probably be reviewing instead of replying to this email.

Alas, https://salsa.debian.org/ci-team/autopkgtest/-/merge_requests/138 was
merged 14 hours ago. :)

Thanks!

cheers, josch

signature.asc
Description: signature

Re: …/doc …/log: .gz → .zst

[Johannes Schauer Marin Rodrigues]

Quoting Hakan Bayındır (2022-08-19 17:47:42)
> I’d object that, because after we rotate the logs, we use a lot of z
> commands, namely zcat, zgrep, zless. Which allows us process many gigabytes
> of gzip files without extracting them first.
> 
> We have a big cluster at office and a central logging system. That system
> handles close to a thousand machines at the same time, and we neither have
> disk space, nor processing power problems.
> 
> Sorry for not agreeing with your idea, but operations needs this kind of
> interoperability and composability.

$ apt-file search /bin/zcat
gzip: /bin/zcat
zutils: /bin/zcat

It sounds like you are using zcat from gzip which indeed can only handle gzip
compression. Why don't you use zcat from zutils?

$ apt-file show zutils
zutils: /bin/zcat
zutils: /bin/zcmp
zutils: /bin/zdiff
zutils: /bin/zegrep
zutils: /bin/zfgrep
zutils: /bin/zgrep
zutils: /bin/ztest
zutils: /bin/zupdate
zutils: /etc/zutilsrc

$ apt-cache show zutils
[...]
Description-en: utilities for dealing with compressed files transparently
 Zutils is a collection of utilities for dealing with any combination of
 compressed and non-compressed files transparently. Currently the supported
 compressors are gzip, bzip2, lzip, xz, and zstd.

Problem solved?

signature.asc
Description: signature

fontconfig RC bugs (was: Re: key packages RC bugs of the month September)

[Johannes Schauer Marin Rodrigues]

Hi Paul,

Quoting Paul Gevers (2022-09-01 13:53:41)
> I am asking for help with investigating RC bug reports, judging 
> severity, reproducing the issue, clarifying the problem, i.e. bug 
> triaging of all RC bugs that haven't seen activity for a while and that 
> are still affecting bookworm. Of course ideally the bug gets fixed. To 
> give examples, I mention 5 bugs below, next month hope I'll mail 5 other 
> ones.
> 
> The full list I use to check for RC bugs in key packages can be found at 
> [2].

looking at the full list, 5 bugs per mail don't seem like much. On the other
hand, if you had listed more then 5 I might not've looked through the list at
all and had not spotted fontconfig in it.

> #960679 src:fontconfig
> strict dependency of arch:any libfontconfig1 on arch:all 
> fontconfig-config going wrong
> https://bugs.debian.org/960679

fontconfig also has a second RC bug: #909750

The last maintainer upload of fontconfig was more than two years ago. Since
then it has been NMU-ed by me and Julien Cristau.

Since there is no maintainer action on #960679 I wanted to ask the d-devel
crowd if you see any problem with making fontconfig-config arch:any to fix it?

There is a patch for #909750 which I can apply in my next fontconfig NMU as
well.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Transition: pkg-config to pkgconf: next steps

[Johannes Schauer Marin Rodrigues]

Quoting Andrej Shadura (2022-10-20 12:25:13)
> I’ve been rebuilding packages with pkgconf for the past couple of weeks, and
> it looks very good so far:
> 
> http://pkgconf-migration.debian.net/

Thank you! Attached is a dd-list of those packages listed in the "Failures
only" page in case somebody wants to have a quick glance whether their package
is affected.

Thanks!

cheers, joschA Mennucc1 
   gpr
   xmorph (U)

A. Maitland Bottoms 
   gr-fcdproplus (U)
   gr-iio
   gr-radar
   uhd

Abou Al Montacir 
   view3dscene (U)

Adrian Bunk 
   gconf-editor

Adrian Knoth 
   ardour (U)
   liblrdf (U)

Adrien Cunin 
   libdjconsole

Aggelos Avgerinos 
   xmobar (U)

Alberto Garcia 
   webkit2gtk (U)
   wpewebkit (U)

Alessandro Ghedini 
   kcov

Alessio Treglia 
   gigedit (U)
   lives (U)

Alexander GQ Gerasiov 
   clickhouse
   xneur

Alexander Wirt 
   icinga2 (U)

Ana Custura 
   chkservice

Andreas Beckmann 
   beignet (U)

Andreas Metzler 
   enblend-enfuse (U)

Andreas Tille 
   anfo (U)
   biobambam2 (U)
   camitk (U)
   iqtree (U)
   jellyfish (U)
   libmaus2 (U)
   wise (U)

Andrej Shadura 
   golang-github-containers-toolbox (U)

Andrew Lee (李健秋) 
   kwin-effect-xrdesktop
   xrdesktop

Andrew Shadura 
   guessnet

Anton Gladky 
   ceres-solver (U)
   eigen3 (U)
   vtk6 (U)
   vtk9 (U)
   yade (U)

Apollon Oikonomopoulos 
   xmobar (U)

Arnout Engelen 
   jack-tools (U)

Aurelien Jarno 
   freebsd-libs (U)
   kfreebsd-10 (U)

Barry deFreese 
   xnee (U)

Barry deFreese 
   xboard (U)

Bastian Blank 
   linux (U)

Ben Hutchings 
   linux (U)

Bernd Zeimetz 
   ceph (U)

Birger Schacht 
   foot
   usbguard

Brandon Barnes 
   dolphin-emu (U)

Bruno "Fuddl" Kleinert 
   faumachine (U)

Camm Maguire 
   xmpi

Carles Fernandez 
   gnss-sdr (U)

Carsten Schoenert 
   kopano-webapp (U)
   kopanocore (U)

Ceph Packaging Team 
   ceph

Changwoo Ryu 
   gnome-hwp-support (U)

ChangZhuo Chen (陳昌倬) 
   lunar-date (U)

Charles Plessy 
   wise (U)

Chiara Marmo 
   freeture (U)

Chow Loong Jin 
   zeitgeist-sharp (U)

Chris Halls 
   libreoffice (U)

Chris Hofstaedtler 
   dnsdist (U)

Chris Lamb 
   zoneminder (U)

Chris Leishman 
   libcypher-parser
   libneo4j-client

Christian Dreihsig 
   boinc-app-eah-brp (U)

Christian T. Steigies 
   gle-graphics (U)
   luola

Christoph Berg 
   wsjtx (U)

Christoph Egger 
   clisp (U)
   fife (U)
   freebsd-libs (U)
   kfreebsd-10 (U)

Christoph Ender 
   fizmo

Christoph Haas 
   zabbix (U)

Christoph Martin 
   vdr-plugin-infosatepg (U)

Christoph Martin 
   xapp (U)

Chrysostomos Nanakos 
   accelio

Clint Adams 
   ghc (U)
   haskell-gi-gio (U)
   haskell-gi-gtk (U)
   haskell-hledger-web (U)
   haskell-hopenpgp-tools (U)
   haskell-pandoc-citeproc (U)
   haskell-yesod-bin (U)

CrossWire Packaging Team 
   xiphos

Cyril Brulebois 
   xfonts-scalable (U)

Damien Caliste 
   v-sim (U)

Daniel Echeverri 
   guake

Daniel Glassey 
   xiphos (U)

Daniel Hughes 
   widemargin (U)

Daniel Leidert 
   drawxtl (U)

Daniel Schaal 
   bino

Dave Hibberd 
   wsjtx (U)
   xnec2c (U)

David Nusinow 
   xfonts-scalable (U)

David Paleino 
   bist

Debian Astro Maintainers 
   gravit

Debian Astro Team 
   healpy

Debian Astronomy Team 
   freeture

Debian Bitcoin Packaging Team 
   cgminer

Debian BOINC Maintainers 
   boinc-app-eah-brp

Debian Chinese Team 
   lunar-calendar
   lunar-date

Debian Cinnamon Team 
   xapp

Debian CLI Applications Team 
   cowbell
   graphmonkey
   widemargin
   xsddiagram
   yahtzeesharp

Debian CLI Libraries Team 
   gtk-sharp2
   log4net
   zeitgeist-sharp

Debian Common Lisp Team 
   clisp
   cmucl

Debian D Language Group 
   vibe.d

Debian Electronics Team 
   drawtiming
   irsim
   klayout

Debian FreeIPA Team 
   jss

Debian Games Team 
   0ad
   dolphin-emu
   fife
   freeorion
   lugaru
   wesnoth-1.14
   wesnoth-1.16
   widelands
   xboard

Debian Games Team 
   lix

Debian GCC Maintainers 
   gcc-10
   gcc-11
   gcc-11-cross-mipsen
   gcc-12
   gcc-12-cross-mipsen
   gcc-9
   gcc-9-cross-mipsen

Debian GNOME Maintainers 
   endeavour
   gnome-books
   gnome-recipes
   xdg-user-dirs-gtk

Debian Go Packaging Team 
   acmetool
   golang-github-twstrike-gotk3adapter

Debian Go Packaging Team 
   cadvisor
   gitlab-ci-multi-runner (U)
   golang-github-containers-toolbox

Debian Hamradio Maintainers 
   gnss-sdr
   gr-fcdproplus
   wsjtx
   xnec2c

Debian Haskell Group 
   darcs
   ghc
   haskell-blogliterately
   haskell-gi-gio
   haskell-gi-gtk
   haskell-gi-harfbuzz
   haskell-hledger-web
   haskell-hopenpgp-tools
   haskell-pandoc-citeproc
   haskell-stack
   haskell-yesod-bin
   haskell-yi-frontend-pango
   xmobar
   xmonad-contrib
   xmonad-extras

Debian HPC Team 
   libvma

Debian Java Maintainers 
   zookeeper

Debian KDE Extras Team 
   kmplayer

Debian Kernel Team 
   linux

Debian Korean L10N 
   gnome-hwp-support

Debian LibreOffice Maintainers 
   libreoffice

Debian Math Team 
   fplll

Debian Med Packaging Team 
   anfo
   

Re: epoch for tss2 package

[Johannes Schauer Marin Rodrigues]

Quoting Andreas Henriksson (2022-10-20 12:13:24)
> Hello Debora Babb,
> 
> On Wed, Oct 19, 2022 at 11:04:35PM -0700, Debora Velarde Babb wrote:
> > Greetings,
> > 
> > The upstream package for tss2 has been renamed ibmtss.  When the name
> > was changed upstream, the version number convention also changed. 
> > Upstream tss2-1470 was updated to ibmtss-1.3.0.  The current version of
> > ibmtss is now 1.6.0.  I believe I need to use an epoch number to handle
> > the version change correctly.  
> 
> Upstream renaming their project is one of the very few chances you get
> to GET RID OF an epoch in a debian package!
> 
> > 
> > Initially I attempted to create the package with the new name ibmtss. 
> > There was some discussion on debian-mentors list and the response was
> > that I should NOT change the name to ibmtss and instructed to instead
> > use an epoch number.
> 
> This seems like very bad advice to me.
> 
> IMHO you should:
> 
> 1. package the new project under the new name.
>(i.e. rename both source and binary packages.)
> 2. Provide additional empty binary (transitional) packages under the old
>name which depends on the respective new binary packages, so old
>installations gets upgraded to the new package.
> 3. Use a debian/rules override to alter version number *only* for the
>empty transitional packages, so that they get a higher version number
>than was previously provided. eg. last-tss2-version+really1.3.0-1
> 
> 
> eg.
> 
> override_dh_gencontrol:
> # Make transitional packages have a higher version number
> # than the old binary packages built from src:tss2 (1045-1.2)
> dh_gencontrol --package=tss2 -- \
> -v1045+really1.6.0-1
> dh_gencontrol --package=libtss0 -- \
> -v1045+really1.6.0-1
> dh_gencontrol --package=libtss-dev -- \
> -v1045+really1.6.0-1
> # Use the correct version number for real binary packages
> dh_gencontrol --remaining-packages
> 
> 
> Once your transitional packages has shipped in a stable release,
> you can then remove them from debian/control and also drop the
> debian/rules override of dh_gencontrol in your next upload to unstable.

The transitional package should be Priority:optional and Section:oldlibs
according to devref 6.8.7.:

https://www.debian.org/doc/manuals/developers-reference/best-pkging-practices.en.html#make-transition-packages-deborphan-compliant

What is the reason for using a transitional package instead of using
Provides/Conflicts/Replaces? I found

https://wiki.debian.org/RenamingPackages

But that lists:

> Cannot be used for packages that are used in build dependencies, as several
> build tools (like sbuild) do not support virtual packages in those
> dependencies by design, to guarantee deterministic builds.

Wait what? If sbuild doesn't support virtual packages I'd like to hear about
that. Can I just remove this reason from the wiki page? It is obviously wrong.
If it is not, please file a bug against sbuild.

> Most package managers (including APT, as of 2011-02-21) do not know to
> replace the old with the new one and will only use the new package if it is
> installed for some other reason.

That is a rather old timestamp. And I also do not understand what the author
tries to say here. Isn't using the new package exactly what we want here?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: epoch for tss2 package

[Johannes Schauer Marin Rodrigues]

Quoting Philipp Kern (2022-10-20 14:29:13)
> On 20.10.22 13:40, Johannes Schauer Marin Rodrigues wrote:
> > Quoting Andreas Henriksson (2022-10-20 12:13:24)
> >> Cannot be used for packages that are used in build dependencies, as several
> >> build tools (like sbuild) do not support virtual packages in those
> >> dependencies by design, to guarantee deterministic builds.
> > Wait what? If sbuild doesn't support virtual packages I'd like to hear about
> > that. Can I just remove this reason from the wiki page? It is obviously 
> > wrong.
> > If it is not, please file a bug against sbuild.
> 
> The correct statement here is that you ought to pick a default choice 
> first[1] before a virtual alternative. We don't want to leave it up to 
> the resolver to pick an arbitrary available build-dependency. So this is 
> more of a policy rather than a technical question. Behavior for 
> experimental might currently differ due to a different resolver choice 
> that's more flexible by design - to get newer versions from experimental 
> if necessary.
> 
> Kind regards
> Philipp Kern
> 
> [1] This might require an overall agreement across Debian at times. But that
> seems to be more relevant for dependencies than build-dependencies.

Is that really the correct statement? Even after excluding all virtual packages
with a single provider, there are literally thousands of source packages for
which their first alternative is a virtual package. Is this "policy" documented
somewhere? Because if it is, then it either should change or the archive has to
be changed to match that policy.

In any case, this was not my original question. Andreas presented a way to use
a transitional package to rename a package which will work fine I guess except
that we have to carry an empty package for a release and that empty package has
to be cleaned up at some point, for example by deborphan.

My original question was why using a virtual package for the same purpose is a
bad idea. The wiki page https://wiki.debian.org/RenamingPackages lists reasons
against it that are incorrect. So why is it really a bad idea?

Is there any reason not to delete the first reason (the sbuild one) completely?

And either I misunderstand the second reason or I implemented my POC
incorrectly or apt (as of 2022-10-21) is perfectly capable of replacing the old
with the new one.

Can somebody shed some light on this?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: epoch for tss2 package

[Johannes Schauer Marin Rodrigues]

Hi Debora,

Quoting Debora Velarde Babb (2022-10-20 08:04:35)
> The upstream package for tss2 has been renamed ibmtss.  When the name was
> changed upstream, the version number convention also changed.  Upstream
> tss2-1470 was updated to ibmtss-1.3.0.  The current version of ibmtss is now
> 1.6.0.  I believe I need to use an epoch number to handle the version change
> correctly.
> 
> Initially I attempted to create the package with the new name ibmtss. 
> There was some discussion on debian-mentors list and the response was
> that I should NOT change the name to ibmtss and instructed to instead
> use an epoch number.

which discussion was that? I searched the last four months of the
debian-mentors list archive for your name as well as for the string "tss" and
couldn't find the advice to use an epoch instead renaming the source package.
Just like pabs (and for the same reason) I would argue for a one-time migration
rather than carrying a wrong name as well as the epoch number forever.

> I recently posted a packaging question to the debian-mentors list and someone
> kindly explained that use of an epoch number needed to be discussed on
> debian-devel first.

That is correct. Thanks for reaching out to us! :)

cheers, josch

signature.asc
Description: signature

Re: ppc64el porterbox replacement: plummer.d.o -> platti.d.o

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting David Bremner (2022-10-22 18:16:12)
> Aurelien Jarno  writes:
> > We lost access to the Power9 machine hosted at Unicamp, which was
> > hosting the ppc64el porterbox called plummer.d.o. A new porterbox called
> > platti.d.o has been setup as a replacement.
> 
> It would be nifty if someone (TM) would update where
> ppc64el-porterbox.debian.net points to.

this is the first time I hear about $arch-porterbox.debian.net. This is super
cool! When was that announced and who maintains it? Why is it only on
debian.net and not on debian.org?

I always use https://db.debian.org/machines.cgi to obtain the mapping from
debian architecture to porterbox machine. Maybe that website could inform me
that $arch-porterbox.debian.net also exists?

Whoever maintains this mapping (thank you!!) should also add it to
https://wiki.debian.org/DebianNetDomains and then it would be easy to figure
out whom to trigger once an update is necessary. :)

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Presenting DPKG_ROOT

[Johannes Schauer Marin Rodrigues]

Hi Russ,

Quoting Russ Allbery (2022-09-20 00:05:23)
> Johannes Schauer Marin Rodrigues  writes:
> > in 2016 we filed our first DPKG_ROOT patch #824594 against
> > base-files. The dpkg version at the time just had included support for
> > the DPKG_ROOT variable being set for maintainer scripts and we were
> > excited to try out this new feature for creating foreign architecture
> > chroots. At the time we thought that no discussion on d-devel was
> > necessary before filing the bug because we knew only 10 source packages
> > had to add DPKG_ROOT to their maintainer scripts and because doing so
> > would not affect any normal installation.
> 
> [...]
> 
> Thank you for this excellent write-up!
> 
> This is exactly the type of fairly obscure Debian lore that, although it
> only affects a small number of packages, is worth documenting because it
> can be very difficult to understand otherwise why it's present or to debug
> problems caused by accidentally breaking it.
> 
> I would therefore love to see this documented in Policy.  The
> documentation doesn't have to be long, but even though this only affects a
> small handful of packages used during early bootstrapping, I think we
> should write it down somewhere official so that we have a record of what
> we're doing and how it's supposed to work (and what packages need to care
> about it).
> 
> If possible, could you write up a brief description along those lines and
> open a bug against debian-policy with that description?  We can then
> figure out where to put it in the document.

sure thing!

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1020323

Lets continue the discussion about how to best include the relevant information
into policy over there.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: 'The' timestamp of a snapshot of deb.debian.org

[Johannes Schauer Marin Rodrigues]

Hi Roland,

Quoting Roland Clobus (2022-09-18 10:58:37)
> I'm looking for 'the' timestamp of the Debian Archive, which will allow me to
> virtually travel through time to re-generate a specific state of Debian.

Holger just suggested on IRC that I reply to your mail -- probably with my
metasnap.debian.net and debbisect hats on -- even though there are probably
other people who understand snapshot.d.o better than me. I am wondering though:
what is it exactly you want to achieve? If you want to go back in time, then
snapshot.d.o already does what you want, no?

What is it that you want to do in the end?

Thanks!

cheers, josch

signature.asc
Description: signature

Presenting DPKG_ROOT

[Johannes Schauer Marin Rodrigues]

Hi,

in 2016 we filed our first DPKG_ROOT patch #824594 against base-files. The dpkg
version at the time just had included support for the DPKG_ROOT variable being
set for maintainer scripts and we were excited to try out this new feature for
creating foreign architecture chroots. At the time we thought that no
discussion on d-devel was necessary before filing the bug because we knew only
10 source packages had to add DPKG_ROOT to their maintainer scripts and because
doing so would not affect any normal installation.

In hindsight, this was a mistake. We should've brought it up on d-devel and
explained our idea first before starting to file our bugs with patches. Luckily
in this very first bug, Santiago pushed back and requested to first see that
our concept really works before applying anything. This was excellent advice
and resulted in a CI job on salsa that regularly checks that Debian unstable
with our patches produces bit-by-bit identical chroots created with DPKG_ROOT
compared to chroots created the normal way. This proof of our method working
exactly as intended convinced Santiago and probably many other maintainers that
applied our patches to their source packages.

Today, six years later, all but two of our patches have been applied. The
transition is nearly complete.

So why are we (Helmut and me) writing you now that things are already done?

Firstly to apologize for having misjudged the situation in the past years. We
should've communicated DPKG_ROOT to all of d-devel at the very beginning to
allow for project wide discussion but decided not to do so. For that mistake we
are sorry.

Secondly, while we of course are hoping for a blessing of our contributions in
hindsight, we wanted to ask whether we maybe missed anything that makes our
DPKG_ROOT approach inferior to other ways to solve this problem. We also wanted
to ask about the scope of DPKG_ROOT. So what exactly is the problem we are
trying to solve?

In the very early days of a new architecture, emulation support is either not
available at all or too buggy to be useful for any practical purposes.  After
having cross-built the initial package set, these packages need to be installed
to create a chroot that can then be used to continue building packages natively
on the new architecture, completing the early bootstrap process. But creating
that chroot requires package maintainer scripts to be run but we cannot emulate
the new architecture to run any of its binaries. So how was this done in the
past? By performing the tasks that are usually carried out by package
maintainer scripts manually. We wanted to find a way that would allow for an
automatic creation of a foreign architecture chroot without being able to run
any of the binaries in it.

To solve this problem, since dpkg 1.18.10 (old-old-stable) the variable
`DPKG_ROOT` is set to the empty string in all maintainer script invocations.
All, except when dpkg is run with `--force-script-chrootless` and `--root` set
to a chroot directory path which will be stored in the `DPKG_ROOT` environment
variable. This is a “force” option because if set, dpkg will not do a chroot
call into the new chroot before calling the maintainer script, thus causing
undesired changes in the outer system instead. By installing packages in a way
that avoids the chroot call, maintainer scripts will run the tools from outside
the chroot instead of the foreign architecture tools inside the chroot.  These
tools need to know where they need to operate on and they use the value of the
`DPKG_ROOT` variable to get this information.

As of today, tools like dpkg-divert, dpkg-maintscript-helper,
deb-systemd-helper, or update-shells understand the `DPKG_ROOT` variable and
will do the right thing.  Maintainer script snippets as they are generated by
debhelper also already respect `DPKG_ROOT`. Where we need package-specific
patches is when maintainer scripts call general purpose tools like mv, cp,
chown or chmod, where it doesn’t make sense to let them support `DPKG_ROOT`
because those tools are not limited to be used in maintainer scripts. Source
packages in the Essential:yes and build-essential set that require changes
involving the `DPKG_ROOT` variable in their maintainer scripts are:

base-files, base-passwd, coreutils, dash, debianutils, dpkg, gcc-defaults,
glibc, pam, shadow

Usually, patches look like this:

https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=983565;filename=coreutils_8.32-4.1.debdiff;msg=5

So if before the maintainer script ran `rm /usr/bin/touch` then it now runs `rm
"$DPKG_ROOT/usr/bin/touch"`. Since the `DPKG_ROOT` variable is usually empty,
this change will be a NO-OP in normal installations and only affects setups
that explicitly called dpkg in the way described above. Another way to support
`DPKG_ROOT` is to remove maintainer scripts altogether and replace them by
declarative methods, which was done in case of the transition from add-shell
and remove-shell to update-shells:

Re: Presenting DPKG_ROOT

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Josh Triplett (2022-09-11 15:26:52)
> Johannes Schauer Marin Rodrigues wrote:
> > What do you think? Is this the right solution to the problem? A few more 
> > bits
> > about DPKG_ROOT as well as alternative solution proposals (including 
> > rejected
> > ones) can be found on this wiki page:
> > 
> > https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap#Proposal:_chrootless_maintscripts
> > 
> > So lets come back to our question of scope: Right now, our DPKG_ROOT patches
> > are limited to Essential:yes, build-essential, apt and systemd. We also
> > restrict the mechanism to initial installations only and upgrades are not
> > supported. We also currently require that the system (and its tools) on the
> > outside must be the same as the chroot that is being created with 
> > DPKG_ROOT. As
> > far as enabling very early architecture bootstrap goes, this solves the
> > problem.
> > 
> > So what do you think? Is this okay? Is there a better solution?
> 
> So, first of all, I'm thrilled to see work on improving bootstrapping
> and cross-bootstrapping. I'm also thrilled to see DPKG_ROOT being
> discussed more broadly.
> 
> Regarding the specific solution: the DPKG_ROOT approach has concerned me
> since it was introduced, because maintainer scripts run as root, so a
> bug in one package's DPKG_ROOT support (let alone the absence of
> DPKG_ROOT support) would result in modifying the host system rather than the
> target chroot.

That is correct.

> I would love to see the DPKG_ROOT support augmented with `unshare`, a mount
> namespace, a UID namespace, and a chroot, such that:
> 
> - The host filesystem is bind-mounted read-only.
> - Host devices are not available (only a minimal /dev); this will also
>   help ensure bootstraps don't depend on any aspect of the host system.
> - The maintainer scripts run as container-root, but not as host-root, so
>   that they can't accidentally change any aspect of the host.
> 
> This could still make use of the existing DPKG_ROOT support; this would
> be completely transparent to the maintainer scripts and tools ported
> thus far.
> 
> This would also allow bootstrapping as non-root, on systems that allow
> creating UID namespaces as non-root.
> 
> As a bonus, testsuites could use an overlayfs instead of a read-only
> bind mount, and then check afterwards if *any* changes occurred in the
> overlay, which would be a test failure.
> 
> Does that seem like a reasonable addition to the DPKG_ROOT support?

Bind-mounting the host system read-only is an interesting idea that we haven't
tried out yet. To avoid unintended modifications of the host system we are
currently testing two different approaches in our CI system:

 1. we run the whole chroot creation inside fakeroot. That way, the maintainer
scripts think they are run as root but they actually cannot modify anything
important.

 2. we run the chroot creation as root but inside mmdebstrap in unshare mode.
We create a tarball of the outer system before and after running the
DPKG_ROOT method and then compare these two tarballs to make sure that no
changes were done in the outer system.

I think making the outer system read-only via bind-mounts is another valuable
test that we should implement.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Presenting DPKG_ROOT

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Josh Triplett (2022-09-11 15:26:52)
> Johannes Schauer Marin Rodrigues wrote:
> > What do you think? Is this the right solution to the problem? A few more 
> > bits
> > about DPKG_ROOT as well as alternative solution proposals (including 
> > rejected
> > ones) can be found on this wiki page:
> > 
> > https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap#Proposal:_chrootless_maintscripts
> > 
> > So lets come back to our question of scope: Right now, our DPKG_ROOT patches
> > are limited to Essential:yes, build-essential, apt and systemd. We also
> > restrict the mechanism to initial installations only and upgrades are not
> > supported. We also currently require that the system (and its tools) on the
> > outside must be the same as the chroot that is being created with 
> > DPKG_ROOT. As
> > far as enabling very early architecture bootstrap goes, this solves the
> > problem.
> > 
> > So what do you think? Is this okay? Is there a better solution?
> 
> So, first of all, I'm thrilled to see work on improving bootstrapping
> and cross-bootstrapping. I'm also thrilled to see DPKG_ROOT being
> discussed more broadly.
> 
> Regarding the specific solution: the DPKG_ROOT approach has concerned me
> since it was introduced, because maintainer scripts run as root, so a
> bug in one package's DPKG_ROOT support (let alone the absence of
> DPKG_ROOT support) would result in modifying the host system rather than
> the target chroot.
> 
> I would love to see the DPKG_ROOT support augmented with `unshare`, a
> mount namespace, a UID namespace, and a chroot, such that:
> 
> - The host filesystem is bind-mounted read-only.
> - Host devices are not available (only a minimal /dev); this will also
>   help ensure bootstraps don't depend on any aspect of the host system.
> - The maintainer scripts run as container-root, but not as host-root, so
>   that they can't accidentally change any aspect of the host.
> 
> This could still make use of the existing DPKG_ROOT support; this would
> be completely transparent to the maintainer scripts and tools ported
> thus far.
> 
> This would also allow bootstrapping as non-root, on systems that allow
> creating UID namespaces as non-root.
> 
> As a bonus, testsuites could use an overlayfs instead of a read-only
> bind mount, and then check afterwards if *any* changes occurred in the
> overlay, which would be a test failure.
> 
> Does that seem like a reasonable addition to the DPKG_ROOT support?

re-mounting the host filesystem read-only is a good idea for the machinery that
in the end will create chroots using DPKG_ROOT (for example rebootstrap). Note,
that as the normal user, you probably will never need to create a chroot with
DPKG_ROOT because even if you want to create a foreign architecture chroot,
qemu support is available for all our release arches. But if you are involved
in early architecture bootstrapping and want to make sure that chroot creation
does not modify your outer system, then you can create your chroot using either
of the following techniques.

DISCLAIMER: two of our patches have not been applied yet so DO NOT run any of
the below commands unless you really know what you are doing.

1. as the normal user using fakeroot

fakeroot mmdebstrap --variant=essential --mode=chrootless unstable 
chroot.tar

2. inside mmdebstrap in unshare mode as a customize-hook

mmdebstrap --mode=unshare --skip=setup,update,cleanup --variant=custom \
--setup-hook='mount -o remount,bind,ro /' \
--customize-hook='mmdebstrap --variant=essential --mode=chrootless unstable 
"$1"' \
'' chroot.tar

3. using the (undocumented) mmdebstrap --unshare-helper

mmdebstrap --unshare-helper mmdebstrap \
--setup-hook='mount -o remount,bind,ro /' \
--variant=essential --mode=chrootless unstable chroot.tar

4. using lxc unshare tools

lxc-usernsexec -- lxc-unshare -s 'MOUNT|PID|UTSNAME|IPC' -- mmdebstrap \
--setup-hook='mount -o remount,bind,ro /' \
--variant=essential --mode=chrootless unstable chroot.tar

If you set SOURCE_DATE_EPOCH and add a repo with the last two remaining patches
applied to the packages in the Essential:yes set, then the tarballs created by
either of the commands above will be bit-by-bit identical to tarballs created
like this:

mmdebstrap --variant=essential unstable chroot.tar

Re-mounting / as read-only with an unshared mount namespace is unfortunately
nothing we can test in our CI system because those do not support any unshare
operation. But as a test, re-mounting / read-only is also not useful because
there are scripts which will not error-out if they cannot write something and
we would thus never detect these unintended changes. So instead, in our CI, we
let the maintainer systems write whatever they want, but compare the whole
system state before and after, to make sure that no file on the outside was
modified (even in its metadata).

signature.asc
Description: signature

Re: 'The' timestamp of a snapshot of deb.debian.org

[Johannes Schauer Marin Rodrigues]

Quoting Roland Clobus (2022-09-21 22:03:33)
> As to the origin of my question: The snapshot-mirror has been offline for a
> while, and I've used deb.debian.org to generate my test images (for the
> reproducible live-build-based live-ISO images). I've compared the timestamps
> of the InRelease file at deb.d.o with the timestamp in the URL available on
> snapshot.d.o and I've noticed a difference.  That means that all images that
> I've generated using deb.d.o cannot be verified at any later moment than
> within the same dak-round (which last for 6 hours).

Why not? Just check which package versions your image contains and find the
correct snapshot.debian.org timestamp(s) containing all these packages via
metasnap.debian.net.

signature.asc
Description: signature

Re: bits from the release team: are you ready to skate yet?

[Johannes Schauer Marin Rodrigues]

Hi Paul,

Quoting Paul Gevers (2022-10-13 17:25:36)
> On 13-10-2022 14:20, Johannes Schauer Marin Rodrigues wrote:
> > Quoting Paul Gevers (2022-10-13 10:00:42)
> >> Please also consider supporting the nodoc build profile. We are aware
> >> that nodoc is regularly used in a non-reproducible way (as intended,
> >> but with this consequence), so checking for correctness of this
> >> profile may be a bit harder. Ideally, using the profile would just
> >> make documentation binaries virtually empty.
> > 
> > No. Ideally, using the nodoc profile would make documentation binaries not 
> > be
> > emitted at all. This then also makes checking for correctness a lot easier
> > because then all binary packages built with the nodoc profile will be
> > bit-by-bit identical if your source package builds reproducibly.
> 
> Policy [1] says something else:
> """
> This option does not change the set of binary packages generated by the 
> source package, but documentation-only binary packages may be nearly 
> empty when built with this option.
> """
> I suggest you try and get policy updated.
> 
> [1]
> https://www.debian.org/doc/debian-policy/ch-source.html#debian-rules-and-deb-build-options

hrm... maybe I misunderstand but I thought your initial mail talked about build
profiles (aka DEB_BUILD_PROFILES) and not build options (aka
DEB_BUILD_OPTIONS). The policy section you cite is about DEB_BUILD_OPTIONS and
not about DEB_BUILD_PROFILES.

As far as I know, build profiles are not documented in policy at all yet. The
bug for that is https://bugs.debian.org/757760

Am I missing something?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: bits from the release team: are you ready to skate yet?

[Johannes Schauer Marin Rodrigues]

Quoting Paul Gevers (2022-10-13 10:00:42)
> Please also consider supporting the nodoc build profile. We are aware
> that nodoc is regularly used in a non-reproducible way (as intended,
> but with this consequence), so checking for correctness of this
> profile may be a bit harder. Ideally, using the profile would just
> make documentation binaries virtually empty.

No. Ideally, using the nodoc profile would make documentation binaries not be
emitted at all. This then also makes checking for correctness a lot easier
because then all binary packages built with the nodoc profile will be
bit-by-bit identical if your source package builds reproducibly.

This can be achieved by adding this to the binary package stanza in d/control:

Package: foo-doc
Architecture: all
Build-Profiles: 

Then, in d/rules you can surround code that creates the foo-doc package with a
conditional like this one:

ifneq (,$(filter foo-doc,$(shell dh_listpackages)))
# do stuff needed to build foo-doc
endif

Using dh_listpackages you also automatically catch other cases in which foo-doc
might not get built other than the nodoc build profile being active, for
example for an arch:any-only build.

Also, do not forget to add the build dependencies necessary to build foo-doc to
Build-Depends-Indep instead of keeping them in Build-Depends if your foo-doc
package is Architecture:all.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Welcome to your new installation of Debian GNU/Linux bookworm/sid

[Johannes Schauer Marin Rodrigues]

Quoting Bastian Blank (2022-10-09 10:24:26)
> On Sun, Oct 09, 2022 at 09:41:29AM +0200, Johannes Schauer Marin Rodrigues 
> wrote:
> > This breaks a number of setups like:
> > 
> >  - the sbuild autopkgtest
> >https://salsa.debian.org/debian/sbuild/-/jobs/3353627/raw
> >  - the dropbear autopkgtest
> >
> > https://ci.debian.net/data/autopkgtest/unstable/amd64/d/dropbear/26716581/log.gz
> >  - autopkgtest-virt-qemu image builders
> >  - the MNT reform image builder
> >  - the mmdebstrap testsuite which builds a qemu system image for its
> >local tests
> >  - the mmdebstrap jenkins job
> 
> This is a bug in mmdebstrap:
> 
> | open my $fh, '>', "$options->{root}/etc/machine-id"
> |   or error "failed to open(): $!";
> | print $fh "uninitialized\n";
> | close $fh;

Yes, maybe. I saw that you filed #1021478 against mmdebstrap --thanks for that!

If this is a bug in mmdebstrap, then mmdebstrap should do the same thing as
debootstrap which is currently being discussed in #1018740 which I see you also
commented on.

I do not understand enough about systemd to be able to say whether an empty
value or "uninitialized" is the correct default value for tools like
debootstrap or mmdebstrap to set. If nobody else chimes in, I'll change
mmdebstrap to write the empty string as suggested by Bastian.

Thanks!

cheers, josch

signature.asc
Description: signature

Welcome to your new installation of Debian GNU/Linux bookworm/sid

[Johannes Schauer Marin Rodrigues]

Hi,

the last upload of src:systemd (251.5-1) enabled firstboot by default on
Debian. From debian/changelog:

  * Enable firstboot, disabled by default on Debian.
Currently the first-boot conditions are not met by any Debian
image (/etc/machine-id with content uninitialized, so we can
just enable the build and ship it in the main package.
This lets image builders (eg: cloud images) tinker with it.

https://www.freedesktop.org/software/systemd/man/machine-id.html#First%20Boot%20Semantics
(Closes: #844528)

This breaks a number of setups like:

 - the sbuild autopkgtest
   https://salsa.debian.org/debian/sbuild/-/jobs/3353627/raw
 - the dropbear autopkgtest
   
https://ci.debian.net/data/autopkgtest/unstable/amd64/d/dropbear/26716581/log.gz
 - autopkgtest-virt-qemu image builders
 - the MNT reform image builder
 - the mmdebstrap testsuite which builds a qemu system image for its
   local tests
 - the mmdebstrap jenkins job

The scripts I found that broke will fail because they will idle forever waiting
for user input with this message in the boot log:

Welcome to your new installation of Debian GNU/Linux bookworm/sid
Please configure your system!
-- Press any key to proceed --

One possible workaround is to write out an empty /etc/machine-id.

As you can see from the package selection above, this is just the part of the
archive I'm interested in. I'm sharing this here so that others doing similar
things can get a heads-up hopefully before they sink hours into figuring out
why their qemu virtual machine suddenly stalls forever...

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Q: uscan with GitHub

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Paul Wise (2022-09-20 02:38:30)
> On Mon, 2022-09-19 at 20:50 +0900, Hideki Yamane wrote:
> > Recent changes in GitHub releases pages, I cannot check upstream
> > version with uscan. How do you deal with it?
> 
> If you are using the automatically generated tarballs, then
> just switching to the uscan git mode seems like the way to go.

the only reason I'm not using git mode for all my upstream projects that use
git is this paragraph from the uscan man page:

If the upstream publishes the released tarball via its web
interface, please use it instead of using this mode. This mode is
the last resort method.

This sounds like I should rather use other methods like rewriting my d/watch
files to use api.github.com before using this "last resort" method.

Is this paragraph still accurate?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: FTBS bugs -- MBF?

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Lucas Nussbaum (2022-10-02 21:51:52)
> On 02/10/22 at 04:23 +0200, Adam Borowski wrote:
> > Nǐmen hǎo!
> > I did another _source_ rebuild of the archive -- checking if every package
> > is capable of repacking its source.  Ie, if you can unpack it, (possibly
> > modify), and pack again.
> > 
> > Putting aside packages that are broken in other ways as well (B-Depends
> > non-installable, FTBFS or a RC bug), there seems to be no new fancy types
> > of breakage that haven't been fixed in 2020.
> > 
> > This leaves one big set: packages that fail the clean step due to
> > undeclared B-Depends.  According to the Policy:
> > 
> > # "clean"
> > #Only the "Build-Depends" and "Build-Conflicts" fields must be
> > #satisfied when this target is invoked.
> > 
> > ... which makes sense as you might be interested in only an arch:all or
> > arch:any build, and we have no clean-indep/clean-arch targets.
> > 
> > For sbuild, the incantation is:
> > alias sbuild-source='sbuild -s --source-only-changes --no-arch-all 
> > --no-arch-any --no-run-autopkgtest'
> > 
> > As 291 packages fail this requirement, I'm posting here before (instead?)
> > filing bugs.  There's also a question of severity.
> > 
> > Raw list and dd-list attached.
> 
> All those source packages are Architecture: all.
> 
> To make this easier to detect (and avoid regressions in the long term),
> I wonder if sbuild should have an option that would make it do, for a
> source+all build:

please do not abuse sbuild to produce source packages. Source packages are the
input to sbuild and not its output. Sbuild has some convenience features that
let it create the source package for you from an unpacked directory so that you
do not have to do that step manually but that doesn't change the fact that to
operate it still needs to create a dsc first.

Instead of trying to use the -s or --source option, use the
--source-only-changes option instead which will not re-create the source
package but gives you a .changes file ready to a source-only upload anyways in
addition to the arch-all or arch-any .changes file.

> - install B-D
> - run clean
> - install B-D-I
> - build the binary packages

This will be tricky to implement because sbuild doesn't run the clean target.
Instead it runs dpkg-buildpackage which then runs the clean target. But feel
free to try and implement it and file a merge request on salsa. Maybe it's not
as bad as I fear.

Changing salsa-ci.yml to test for this would not be easy either, because
"apt-get build-dep" only exposes the --arch-only and --indep-only options. So
there is no way to tell apt "only the dependencies for the clean target,
please".

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Q: checking with piuparts before getting into repository

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Hideki Yamane (2022-10-07 03:39:39)
> On Thu, 6 Oct 2022 17:45:06 +0200 Michael Biebl  wrote:
> > If you are using salsa, you can utilize all the features gitlab 
> > provides. E.g. in src:systemd, we use
> > https://salsa.debian.org/systemd-team/systemd/-/blob/debian/master/debian/gitlab-ci.yml
> 
>  Yes, I'm using salsa-ci pipeline and quite happy with that :)
> 
>  However, it is just a workflow and best practice, not the infrastructure
>  that all of us should use for releasing packages, and you know we are
>  all human so sometimes make mistakes.
> 
>  I think if we can integrate piuparts infra for releasing packages, 
>  we can avoid this kind of disaster and make sid users much happy, IMHO.
>   
>  dput -> queue -> build -> "piuparts test pass" -> repo
> 
> 
>  Is there any blocker to implement such thing?

it would be really cool to have a suite only containing packages that had their
piuparts and autopkgtests pass. Oh wait, we already have that and it's called
"testing". Why was #1021336 a problem? Because (for good reasons) many places
use the "unstable" suite. These places (including the build infrastructure) use
"unstable" instead of "testing" for good reasons. If you introduce yet another
suite before packages land in unstable, then these places would likely want to
use that suite instead, because they need to consume the most up-to-date
packages. At which point you are back where you started. So shifting the
problem to another suite doesn't help you at all because those pieces of our
machinery that want to consume the most up-to-date packages will then use that
suite and then still be affected by the problems you wanted to protect them
from.

So no, I do not think this makes sense. If some software is fine with older,
but tested packages, then these software pieces can use "testing" today. Others
have to live with the fact that the newest software is "unstable" for a reason.

I think the right way to fix this is to find most of these problems that can be
found by a machine before the upload happens. This can happen on the
developer's machine but as our distro grows and we are testing more and more
things (lintian, upgrades, arch:any builds, arch:all builds, reproducibility,
cross building...) I do not think we can expect every developer to run all the
tests locally for all packages they maintain. This is why I think using the
salsa CI pipelines are a great solution to situations like this.

In the spirit of this, i filed
https://salsa.debian.org/vorlon/pam/-/merge_requests/14

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Bug#1016575: sbuild aborting when dh-python isn't installed locally

[Johannes Schauer Marin Rodrigues]

Hi Matthias,

no idea why you CC-ed debian-devel@lists.debian.org with your bug. This is a
bug about using Debian so the correct mailing list would be
debian-u...@lists.debian.org. The mailing list you chose is for the development
of Debian.

I'm closing this bug because what you experienced is actually a feature.
Explanation below.

Quoting matthias.geiger1...@tutanota.de (2022-08-03 10:04:32)
> I just did a clean testing install and set up sbuild as usual. When trying to
> build a package the Build-depends: on dh-python sbuild aborts:
>  
> sbuild
> dpkg-source: info: using options from secrets-6.2/debian/source/options: 
> --extend-diff-ignore=^[^/]*[.]egg-info/
> dh clean
> dh: error: unable to load addon python3: Can't locate 
> Debian/Debhelper/Sequence/python3.pm in @INC (you may need to install the 
> Debian::Debhelper::Sequence::python3 module) (@INC contains: /etc/perl 
> /usr/local/lib/x86_64-linux-gnu/perl/5.34.0 /usr/local/share/perl/5.34.0 
> /usr/lib/x86_64-linux-gnu/perl5/5.34 /usr/share/perl5 
> /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.34 
> /usr/share/perl/5.34 /usr/local/lib/site_perl) at (eval 13) line 1.
> BEGIN failed--compilation aborted at (eval 13) line 1.
> 
> make: *** [debian/rules:4: clean] Error 255
> E: Failed to clean source directory /home/werdahias/Packaging/secrets-6.2 
> (/home/werdahias/Packaging/secrets_6.2-1.dsc)
> 
> This shouldn't be an issue as sbuild *should* just install dh-python itself.
> It worked once I installed dh-python on the host machine.

No, sbuild should *not* just install dh-python on your host system. There are
multiple reasons for that but one is, that when you run sbuild, the expectation
is, that this will not change your host system. That's a big part of the reason
why many people use sbuild: so that the things that are needed to build a
source package do not clutter their host system but are instead installed into
a fresh buildd chroot environment. Another reason why sbuild cannot even do
this in principle is, because sbuild is not run as root. So it doesn't even
have the necessary privileges to install anything on your host system -- which
is a good thing!

To go a bit further into why what you experience is a feature and not a bug:
The usual input to sbuild is a source package (a *.dsc). From what I can see,
you are running sbuild without any arguments from within an unpacked source
package directory (I assume that's what /home/werdahias/Packaging/secrets-6.2
is). This would usually not be possible because an unpacked source package is
not a source package. Because the input to sbuild is a source package, you
first have to build said source package from your unpacked source package tree.
To make this as convenient as possible for sbuild users, sbuild has a feature
that allows it to build the source package automatically if you run it from
within an unpackaged source package tree. As part of building that source
package it is running the "clean" target of your debian/rules and in your case,
the clean target needs dh-python installed which you do not have installed and
thus results in the failure you observe. I can think of two ways to fix this:

 a) give up on this convenience feature of sbuild and build the *.dsc yourself
 in any way you want and then feed that to sbuild

 b) run sbuild with --no-clean-source which will not run the "clean" target and
 thus not require dh-python in your case. But beware that this means that you
 are responsible for making sure that your unpacked source directory is clean

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Help trying to debug an sbuild failure?

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Theodore Ts'o (2022-12-27 05:19:45)
> On Mon, Dec 26, 2022 at 08:45:53PM +0100, Santiago Vila wrote:
> > El 26/12/22 a las 20:29, Theodore Ts'o escribió:
> > > I: The directory does not exist inside the chroot.
> > 
> > This is really a problem with schroot. I guess that this will not work 
> > either:
> > 
> > schroot -c the-chroot-name
> > 
> > This usually works when you are in your $HOME because this file:
> > 
> > /etc/schroot/default/fstab
> 
> Nope, that's not the issue; yes, /tmp and /home are missing form
> /etc/schroot/sbuild/fstab, but that was true on the *working* setup as
> well, and from what I can tell; that's deliberate.  It looks like the
> goal is to keep things hermetic when building with sbuild, so it's a
> feature that there aren't random directories leaking through from the
> host to the sbuild enviroment.

that is correct. :)

> I think I've figured out the issue.  The problem is that the user and group
> id's for sbuild are different on my desktop and my laptop, and I did an rsync
> to copy the /chroot directories from my desktop to my laptop --- and it
> appears that sbuild is very sensisitve about the user id's being the same
> across the host and chroot environments.
> 
> Apparently sbuild copies the files for the package it is building over
> a directory in /var/lib/sbuild/build, with the permissions being mode
> 770, and that is what sbuild bind mounts into the chroot.  If my
> theory is correct, if the user/group id's are different from between
> the base /etc/passwd and chroot, then things go bad in a hurry.

I think your analysis makes sense.

> >From my working system (while gbp buildpackage was running sbuild):
> 
> % ls -al /var/lib/sbuild/build/
> total 12
> 4 drwxrws--- 3 sbuild sbuild 4096 Dec 26 23:05 ./
> 4 drwxrws--- 4 sbuild sbuild 4096 Dec 19  2020 ../
> 4 drwxrwx--- 3 tytso  sbuild 4096 Dec 26 23:05 f2fs-tools-oT4KHz/
> 
> Amd on the broken (laptop) system:
> 
> # ls -al /var/lib/sbuild/build/
> total 32
> 4 drwxrws--- 8 fwupd-refresh Debian-exim 4096 Dec 26 22:48 ./
> 4 drwxrws--- 4 sbuildsbuild  4096 Dec 25 14:45 ../
> 4 drwxrwx--- 2 tytso Debian-exim 4096 Dec 26 14:01 f2fs-tools-9QfprK/
> 4 drwxrwx--- 2 tytso Debian-exim 4096 Dec 26 16:01 f2fs-tools-btkVPv/
> 4 drwxrwx--- 2 tytso Debian-exim 4096 Dec 26 14:20 f2fs-tools-cVTRAh/
> 4 drwxrwx--- 2 tytso Debian-exim 4096 Dec 26 22:48 f2fs-tools-Myld8j/
> ...
> 
> Each of were created from a failed sbuild invocation...  And
> "Debian-exim" on my laptop has the same group id as "sbuild" on my
> desktop (and which is the group id in my chroots).
> 
> This also explains the error message:
> 
> E: Failed to change to directory ‘/<>’: Permission denied
> 
> Oops.
> 
> So I guess I need to either manually juggle group id's in the chroots
> (and/or my laptop's root directory --- but it's probably easier to do
> it in the chroots, since there are fewer filees to chgrp in the
> chroots), or I could recreate the sbuild chroots from scratch using
> sbuild-createchroot (but then I would need to recreate all of the
> custom hacks so that ccache,eatmydata, apt-auto-proxy, etc. are working
> correctly in the chroot).
> 
> What fun

Note, that if you keep upgrading a Debian unstable chroot across multiple
releases, it will end up looking slightly different than a freshly
debootstrapped Debian unstable chroot. So I think there is value in
semi-regularly re-creating the build chroot from scratch. Maybe write a script
that does what you need?

That being said, I think the biggest problem is, that the main sbuild
contributors these days (me and Jochen) do not use the schroot backend anymore
but the unshare backend instead which doesn't suffer from this kind of problem
(but of course has different quirks that one needs to be aware of). So I must
admit that in the past few years, the schroot backend hasn't gotten the love
that it deserves considering that it's the default backend used on the buildd
machinery.

Finally, I think this is something that could be solved in sbuild. Ultimately,
schroot is able to do things as the root user, so it should have sufficient
permissions to fix up a chroot that carries incorrect permissions. Could you
quickly note in a bug against sbuild on the Debian BTS which steps exactly you
carried out so that I am able to reproduce your problem?

I'm making no promises that I'll find the time to improve the schroot backend
of sbuild to survive the kind of chroot-rsync that you have performed but if
this is important to you, then maybe we can make a trade and I implement this
sbuild functionality and you have a look at pull requests
https://github.com/tytso/e2fsprogs/pull/118 or #107 and leave some comments in
return? :)

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#1027471: ITP: box64 -- run amd64 binaries on arm64 without emulating library calls

[Johannes Schauer Marin Rodrigues]

Package: wnpp
Severity: wishlist
Owner: Johannes Schauer Marin Rodrigues 
X-Debbugs-Cc: debian-devel@lists.debian.org

* Package name: box64
  Version : 0.2.0
  Upstream Contact: Sebastien Chevalier 
* URL : https://github.com/ptitSeb/box64
* License : Expat
  Programming Lang: C
  Description : run amd64 binaries on arm64 without emulating library calls

 Using DynaRec on arm64, box64 will dynamically recompile x64_64 code to
 aarch64 but will also translate calls to foreign library functions to
 native library calls. This usually makes amd64 executables run via
 box64 much faster than emulating amd64 on arm64 via qemu which will
 emulate everything down to the C library. For example OpenArena emulated
 with box64 will be at about 85% of its native speed.

Re: Multi-host networking software, autopkgtests

[Johannes Schauer Marin Rodrigues]

Quoting Ian Jackson (2023-01-07 16:35:17)
> Thanks.  I considered this but it seemed overkill

maybe

> (and it won't inherit the dependency versions selected by autopkgtest).

Yes it will. When creating the chroot for the virtual machine, you use the apt
sources configured by the autopkgtest runner:

https://sources.debian.org/src/sbuild/0.85.0/debian/tests/unshare-qemuwrapper/#L62

or here:

https://sources.debian.org/src/dropbear/2022.83-1/debian/tests/remote-unlocking/?hl=80#L80

Re: Multi-host networking software, autopkgtests

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Ian Jackson (2023-01-06 17:59:58)
> This all seems very complex.  I definitely want to have something working
> before something like that could exist.  Also, I think it would be a good
> idea to do something ad-hoc, ideally in a number of packages, to gain
> experience so we know what shape the "official" thing ought to be.
> 
> I think therefore that I need to pursue some kind of within-testbed
> nesting, as an interim approach at the very least.  I was hoping that
> someone else had solved (part of) this problem already...

there already exist a number of autopkgtest scripts that start a qemu virtual
machine and then run some script inside of it by connecting to it via ssh.
Maybe this can be helpful to you for now:

https://sources.debian.org/src/dropbear/2022.83-1/debian/tests/remote-unlocking/?hl=112#L137
https://sources.debian.org/src/sbuild/0.85.0/debian/tests/unshare-qemuwrapper/?hl=206#L206
https://sources.debian.org/src/cryptsetup/2%3A2.6.0-2/debian/tests/utils/cryptroot-common/#L505

Thanks!

cheers, josch

Re: Please, minimize your build chroots

[Johannes Schauer Marin Rodrigues]

Quoting Santiago Vila (2022-12-16 02:15:13)
> I've just filed 21 bugs with subject "Missing build-depends on tzdata"
> in bookworm (as tzdata is not build-essential).

thank you for that!

> I can think of two solutions for this:
> 
> A) Either debootstrap, when using buildd profile, installs only
> essential and build-essential packages.
> 
> or
> 
> B) debootstrap keeps installing all required packages in the buildd profile,
> no matter if they are really build-essential or not, but those who
> are not build-essential should have their priority downgraded to "important"
> by ftpmaster.
> 
> This problem was already reported here:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=837060

Thank you for finding my bug from back then. :)

> and apparently we have not decided yet if we are going to do A or B,
> or maybe some other thing. I don't really care how it's fixed, but I
> believe it's about time that we sync practice with policy, because
> currently we are doing this in a quite suboptimal way.

I think truly fixing this problem is a bit more tricky because most build tools
like the sbuild schroot backend require apt being installed in the chroot. As
of today, the sbuild schroot backend is unable to function with a chroot that
doesn't contain apt. I don't think it's conceptually possible to fix the
schroot backend to work with chroots without apt because schroot (for good
reason) doesn't give you root anywhere but inside the chroot.

To be able to install build dependencies in chroots without apt, the sbuild
unshare backend could be used. Also helmut's mdbp can easily be changed to
build packages in chroots without apt when using its mmdebstrap backend.

But of course changing debootstrap to only install essential, build-essential
and apt (but not prio:required) would already fix a large part of the problem.

> In the meantime: If you want to help QA and have any kind of chroot used
> for any kind of QA (say, ci.debian.org or reproducible-builds, or even
> your personal chroots), please try to minimize the packages there,
> do not merely accept debootstrap default behaviour.

You can use mmdebstrap to create such a chroot:

$ mmdebstrap --variant=apt --include=build-essential unstable chroot.tar

This will also install apt because most build tools need it. The mmdebstrap
package mimics debootstrap behaviour. As soon as debootstrap --variant=buildd
is fixed, I'll let mmdebstrap do the same thing.

Thanks!

cheers, josch

Re: Please, minimize your build chroots

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting David Kalnischkies (2022-12-18 17:18:28)
> On Fri, Dec 16, 2022 at 03:38:17PM +0100, Santiago Vila wrote:
> > Then there is "e2fsprogs", which apt seems to treat as if it were
> > an essential package:
> > 
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=826587
> 
> As Julian explained, it is considered "essential" because the maintainer
> said so. If you don't think e2fsprogs should be considered "essential"
> for a system it is installed on please talk to the maintainer.
> 
> Sure, the package is not (anymore) really "Essential:yes", but 'apt'
> isn't either and will print the same message anyhow. I don't think it
> would be a net-benefit for a user to invent words for different types of
> essentialness in apt because in the end you either know what you are
> doing while removing a somewhat essential package and continue or you don't
> know what you are doing and (hopefully) get the hell out.

would it be so difficult to cater to both kind of users? For those who do not
know the terminology, using the word "essential" is probably fine. But for
those who do it's confusing. Why can apt not say something like:

WARNING: The following packages will be removed. Apt considers them essential
because they are marked as Priority:required. This should NOT be done unless
you know exactly what you are doing!

> > This sort-of breaks sbuild when using an ordinary chroot (not overlayfs),
> > because after building a package needing e2fsprogs, it may not be removed
> > and it's kept in the chroot.
> 
> "It may". Well, certainly apt won't autoremove it. Like a lot of other
> packages it wont even through they aren't essential or protected or
> whatever… ("just" prio:required is enough for example). They are not not
> irremovable through. It might just be a little harder to remove them
> than to install them. Heck, some people believe its far easier to start
> vim than to exit it.

Note also, that you really shouldn't be using sbuild with an ordinary chroot,
that is without overlayfs or without the chroot being a tarball unpacked to a
tmpfs. The system will not only be different from before after removal of
packages at the end, if you add --allow-remove-essential to the removal
commandline in sbuild, the chroot might even be completely unusable. What is
the use-case of using sbuild with non-emphimeral chroots?

So personally, I'll not invest my own time in making sbuild better at package
removal at the end of the build process.

> > build packages. Here we would need some interface like
> > SUDO_FORCE_REMOVE=yes, or maybe just stop doing anything at all with the
> > Important:yes flag.
> 
> Ironically, one of the selling points for Protected:yes (that is how the
> field ended up named which dpkg is supporting by now) was that it allows
> to shrink the essential set (e2fsprogs even being an example) as there
> is a non-empty set of people who believe users do incredibly dumb things
> if you give them the option to.
> 
> I mean, we got practically bullied into replacing the "Do as I say!"
> prompt with a semi-hidden cmndline flag (--allow-remove-essential)
> because some wannabe YT star yolo'ed the prompt ending in misery and
> somehow that was framed as our fault by everyone as we didn't show the
> appropriate meme-gif (rendered with caca) to make them understand
> without reading [sorry, not sorry. I am not even exaggerating that much].

After this operation, 195 MS disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!'

https://youtu.be/0506yDSgU7M?t=637

"I have to type 'Yes, do as I say!' in order to install it."

Sigh...

> Due to that, you are now presented with:
> | E: Removing essential system-critical packages is not permitted. This might 
> break the system.
> 
> See? "essential" again and even "system-critical" at that.
> It is all a lie of course. Nobody really needs an init system, much less
> some silly metapackage for it, as long as there is /bin/sh and a keyboard.
> I should make a video about it to – essentially – become famous & rich…

Dammit, and now you wrote that one can use in a public forum that it's possible
to add --allow-remove-essential! Think of the children!

> Btw, apt also has behaviour specifically for sbuild: 'apt-cache show
> mail-transport-agent' has a zero exitcode even through that makes no
> sense at all apart from not making (some?) sbuild versions explode.
> You are welcome. I hate it.

E... lets change that. What's the problem in sbuild?

Thanks!

cheers, josch

Re: dh_auto_test fails and I do not understand why

[Johannes Schauer Marin Rodrigues]

Quoting Niels Thykier (2022-12-15 10:59:10)
> Long story short:
> 
>   * Bug in fakeroot (#1023286 + #1024544)
>   * Me thinking it was a bug in debhelper so I tried to fix it
> (which did not work and broke on the way in)
>   * Me realizing it was a bug in fakeroot and my change did not
> even function as a work around, so I undid it (it broke on
> the way out as well).
> 
> And the winners are: All the people that have (and are) able to use 
> "Rules-Requires-Root: no" as packages with that flag would have been
> completely unaffected by this entire ordeal!
> 
> Your options include:
> 
>   * Migrate to "Rules-Requires-Root: no" if you can
>   * Ensure you have debhelper (>= 13.11.3~) ("just uploaded") or
> debhelper/13.11.1 (or debhelper << 13.11)
> 
> I will now go back to looking more at my prototype for getting even more 
> packages buildable with "Rules-Requires-Root: no".

the original cause of all of this is the fakeroot bug since glibc 2.34,
specifically since coreutils was rebuilt with glibc 2.34.

This is a call for help. Please have a look at #1023286 and #1024544.

Back in August, glib 2.34 reached unstable. In October, coreutils 9.1 was
uploaded and used the new glibc functionality. Since then, calls to chown and
other utilities use some new glibc functions on armel, armhf and i386, namely
__stat64_time64, __fstatat64_time64 and __lstat64_time64.

Clint quickly fixed the problem and I submitted a test case so that this kind
of issue does not fly under the radar again in the future. The test case seems
to be working because now we get a build failure on mipsel because the test is
not passing.

So something else is missing. It's likely another glibc function that is
mipsel-specific. I tried to run ltrace on a mipsel porterbox to figure out what
library calls are done by coreutils chown on mipsel but that failed with
"unexpected breakpoint" errors, see #1023436.

Maybe somebody can find out what is wrong with fakeroot on mipsel? Since
fakeroot is a central part of Debian package builds, it would be a shame to
have this bug open for much longer...

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Please, minimize your build chroots

[Johannes Schauer Marin Rodrigues]

Quoting Holger Levsen (2023-01-28 14:53:37)
> On Sat, Jan 28, 2023 at 02:28:30PM +0100, Johannes Schauer Marin Rodrigues 
> wrote:
> > could we decouple the policy and bug severity question from the question of
> > what a buildd chroot should contain, please?
> [...]
> > Why do people call just accepting that Priority:required packages have to be
> > part of the buildd chroot the easier solution? [...]
> 
> because people get upset if they receive bug reports with severity serious
> when they don't perceive these bugs as serious.
> 
> happened more than 9000 times already.

And I asked in my mail to please "decouple the policy and bug severity question
from the question of what a buildd chroot should contain" for a reason.

I have no problem with downgrading the severity of these bugs to non-RC and
release bookworm without explicit build dependencies on e2fsprogs.

Discussing policy questions and bug severity distracts from the actual question
that I find interesting.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Please, minimize your build chroots

[Johannes Schauer Marin Rodrigues]

Quoting Ansgar (2023-01-28 14:41:31)
> Johannes Schauer Marin Rodrigues writes:
> > I think the much more interesting question is in what environment we want to
> > build our packages in. Currently, on buildds, we build them in a chroot that
> > has Priority:required and build-essential because of (what I think is) a 
> > bug in
> > debootstrap: #837060
> 
> I would rather say: The build-essential packages are those installed by
> debootstrap's buildd profile. At least that seems to be current practice
> for a long time.

Do you agree that the software in our archive should agree on which packages
are needed to build a source package? Currently, they do not. dose3 requires
only Essential:yes and build-essential being installed. Who is buggy? If you
think that dose3 is buggy, who is writing the dose3 patch? Is it not easier to
just fix debootstrap so that debootstraps buildd profile does the same thing as
most other pieces of software resolving build dependencies in the archive? I
know of no other build dependency resolving software in Debian that considers
Priority:required packages to be required for building source packages other
than debootstrap. Do you?

> > So there are two ways forward:
> >
> >  1. accept that Priority:required is needed for building source packages
> >
> >  - adapt Debian policy accordingly
> 
> AFAIU it would not need changes? Policy doesn't seem to explicitly state
> what packages are actually build-essential...

I'm not sure. But I also find the policy question less interesting. I find it
more interesting to first agree how the final solution should look like and
then change policy accordingly (if needed).

> >  - revert the changes to packages made due to Santiago's bugs - change
> >  all tools that do build dependency resolution to now also consider
> >  Priority:required packages
> 
> Why would they need changes? Do they explicitly include essential
> packages too?

Of course they do not do so explicitly, because all Essential:yes packages are
dependencies (and build dependencies) of all packages already *implicitly*.

> >  2. make sure that packages are built without Priority:required packages
> 
> > To me it seems that we nearly are already at (2).
> 
> I think we are already at (1) given everything works already?

Aha. Try running "apt-get build-dep" on a system without tzdata nor e2fsprogs
and witness how apt will not install tzdata nor e2fsprogs. The only reason this
usually works right now is because debootstrap installs Priority:required by
default in all situations including the buildd variant.

My proposal is to fix debootstrap #837060 (patch is in the bug report) so that
it only installs Essential:yes, build-essential and apt and discuss if it makes
sense to have packages like tzdata or e2fsprogs in a buildd chroot or not and
if yes, add those packages as dependencies of the build-essential package.

I do not propose to do this before bookworm release.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Please, minimize your build chroots

[Johannes Schauer Marin Rodrigues]

Quoting Timo Röhling (2023-01-28 13:30:42)
> Hi Andreas,
> 
> * Andreas Henriksson  [2023-01-28 12:50]:
> >Policy is not a religion. Policy has many bugs. Policy is very outdated.
> >[...]
> >Here's an example you could follow:
> >https://lists.debian.org/debian-policy/2022/12/msg00023.html
> Your argument cuts both ways. Right now, Policy says that
> the bugs are RC, and if you believe that should be different, why don't you
> propose such a change and seek consensus yourself?

could we decouple the policy and bug severity question from the question of
what a buildd chroot should contain, please?

Santiago did the work, filed bugs and the fix is to add one more line to
d/control. If you want to reproduce it, use `mmdebstrap --variant=apt
--include=build-essential unstable chroot.tar` to create your chroot tarball.

I think the much more interesting question is in what environment we want to
build our packages in. Currently, on buildds, we build them in a chroot that
has Priority:required and build-essential because of (what I think is) a bug in
debootstrap: #837060

So there are two ways forward:

 1. accept that Priority:required is needed for building source packages

 - adapt Debian policy accordingly
 - revert the changes to packages made due to Santiago's bugs
 - change all tools that do build dependency resolution to now also
   consider Priority:required packages

 2. make sure that packages are built without Priority:required packages

 - fix debootstrap #837060 or use mmdebstrap to create buildd chroots
 - Santiago already did a mass-rebuild and submitted bugs to make sure that
   packages do not FTBFS

The last time I changed all the tools involved in build dependency resolution I
had to submit patches to dpkg, sbuild, apt, dose3, debhelper, cdbs, pbuilder,
lintian, wanna-build, devscripts and others. Of those who prefer option (1)
over option (2) who is going to investigate and potentially change all the
tools who currently just add the "build-essential" package? Or is the solution
to add a dependency to build-essential on all Priority:required packages?

To me it seems that we nearly are already at (2). The debootstrap bug #837060
has a working patch and mmdebstrap exists that can do what is needed today.
Santiago did an archive rebuild to make sure our source package compile in a
chroot without Priority:required.

Why do people call just accepting that Priority:required packages have to be
part of the buildd chroot the easier solution? We just need to fix debootstrap
bug #837060 and we are done, no?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: SONAME bumps (transitions) always via experimental

[Johannes Schauer Marin Rodrigues]

Quoting Paul Gevers (2023-01-05 12:26:09)
> Once accepted, the proposed workflow should also become documented in Debian
> policy.

I think how transitions are done is not even documented in the dev-ref right
now, no?

Last time I was uploading a package for a transition I followed

https://wiki.debian.org/Teams/ReleaseTeam/Transitions

which was very helpful but it would've been nice to find this in Debian policy
and/or the developer reference manual.

> The benefits of all SONAME bump NEW uploads going through experimental 
> are at least:
> * disentangle the start of transitions from NEW acceptance by ftp-master
> * auto transition trackers [1] are setup before the start of transitions
> * reduce the chance of uploads accidentally interfering with ongoing
>transitions (by more awareness and exposure via tracker.d.o).

 * easier testing for all interested parties of reverse dependencies without
   potentially interrupting unstable

> Are there objections against this workflow? (Or voices from people who 
> like this?)

+1

Re: rebootstrap status update for Loongarch

[Johannes Schauer Marin Rodrigues]

Quoting Zhang Ning (2022-11-03 11:45:17)
> 6, gcc, binutils: need help, I don't know how to submit a patch to Debian 
> toolchain-team, don't accept a poll request.
> 
> 7, m4, diffutils: need help, I don't know where is correct upstream[1][2], 
> the patch[3][4] is for generated files, but Debian source has these files, 
> thus how can I submit these patch to Debian source? these two packages don't 
> have VCS.
> 
> 8, newt: need help, build failed without python. cdebconf and base-passwd are 
> skipped.
> 
> 9, gmp: pass with manuall cross_build action, need help.
> 
> 10, libffi: need help, I don't know how to submit a patch to debian libffi, 
> no VCS.

In Debian, for packages having a VCS is still optional in 2022. And for those
packages that do have a VCS, it up to them whether or not they enable receiving
merge requests.

What works for all packages in Debian is to send patches as a bug to the Debian
bugtracking system. So just file a bug against the source packages in question
and attach your patch to the bug.

So you can either send all your patches to the bugtracker in the first place
(because all source packages accept that way to submit patches) or (and that's
what I do) first check if the source package is on salsa and receiving merge
requests and submit the patch there and fall back to the bugtracker if there is
no reaction to my merge request on salsa.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: DEB_BUILD_OPTIONS=nowerror

[Johannes Schauer Marin Rodrigues]

Quoting Helmut Grohne (2023-02-24 08:27:53)
> shellcheck:
>  * grml-debootstrap

not run with DEB_BUILD_OPTIONS=nocheck
https://sources.debian.org/src/grml-debootstrap/0.103/debian/rules/?hl=13#L13

>  * josm-installer

not run with DEB_BUILD_OPTIONS=nocheck
https://sources.debian.org/src/josm-installer/0.0.2+svn18515/debian/rules/?hl=25#L25

>  * kdump-tools

not run with DEB_BUILD_OPTIONS=nocheck
https://sources.debian.org/src/kdump-tools/1:1.8.1/debian/rules/?hl=64#L64

>  * python-sshoot

not run with DEB_BUILD_OPTIONS=nocheck (as dh_auto_test is skipped in that case)
https://sources.debian.org/src/python-sshoot/1.4.2-2/debian/rules/?hl=23#L23

>  * sshcommand

Indeed, that one seems to always run shellcheck:
https://sources.debian.org/src/sshcommand/0~20160110.1~2795f65-1.1/debian/rules/?hl=7#L7

>  * uwsgi

Also always run:
https://sources.debian.org/src/uwsgi/2.0.21-4/debian/rules/?hl=570#L570


I question the conflation of a hypothetical DEB_BUILD_OPTIONS=nowerror with
other linters like shellcheck (or other tools for other programming languages).

I believe that sshcommand and uwsgi should have shellcheck not run with
DEB_BUILD_OPTIONS=nocheck, so I filed:

https://salsa.debian.org/debian/sshcommand/-/merge_requests/1
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1031857

I'm surprised you are conflating shellcheck with the question about -Werror. I
don't yet understand why you would like to skip shellcheck by setting
DEB_BUILD_OPTIONS=nowerror and not by setting DEB_BUILD_OPTIONS=nocheck?

Yes, -Werror is also a kind of check but whether -Werror should be not be
passed when DEB_BUILD_OPTIONS=nocheck is given, is a slightly different
question I think.

Quoting Shengjing Zhu (2023-02-24 09:14:46)
> On Fri, Feb 24, 2023 at 2:26 PM Helmut Grohne  wrote:
> >
> > Hi,
> >
> > I observe a pattern repeated at least twice and probably more often in
> > packages.
> >
> >  * A package adds -Werror to the build. When a new toolchain version is
> >uploaded, it triggers a new warning and that makes the package FTBFS.
> >  * A package runs shellcheck during build. When a new shellcheck is
> >uploaded, it triggers a new warning and makes the package FTBFS.
> >
>█
> IMO, these are just linters. And shouldn't not run after the source is 
> released.
>█
> I'd like to propose the option name `nolint`.

I think that there is value to run linters at build time in a default build
because (as shown by Helmut's initial mail) new version of linters may show
different problems compared to the old version and those only get fixed if the
build breaks. So I think it's useful to have a package FTBFS if a new version
of a linter introduces a failure.

But of course there is also a use-case of disabling linters if one is not doing
a default build. So I think the more interesting questions are:

Should there be a new DEB_BUILD_OPTIONS=nowerror that disables -Werror?

Should DEB_BUILD_OPTIONS=nowerror also disable other linters?

Should other linters like shellcheck be disabled with
DEB_BUILD_OPTIONS=nocheck?

Should -Werror be disabled with DEB_BUILD_OPTIONS=nocheck?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: DEB_BUILD_OPTIONS=nowerror

[Johannes Schauer Marin Rodrigues]

Quoting Helmut Grohne (2023-02-24 07:19:41)
> As such, I propose a generic DEB_BUILD_OPTIONS=nowerror modelled after the
> original observation, but meant to also match other checkers such as
> shellcheck. The general idea should be that a warning should that can be
> non-fatal should be non-fatal if possible.

Are there packages that run shellcheck in other places than those disabled by
DEB_BUILD_OPTIONS=nocheck? If yes, should running shellcheck not be moved to
there? In contrast to -Werror, shellcheck is not involved in literally building
something.

Otherwise, other linter tools like black come to mind that also recently broke
my packages [1] when black got upgraded to 23.1.0-1 two weeks ago. But black
(like shellcheck I'd assume) is usually disabled with nocheck or by not running
autopkgtests.

In what places is shellcheck called such that it cannot or should not be moved
to a place where it could be disabled by DEB_BUILD_OPTIONS=nocheck?

Thanks!

cheers, josch

[1] Versions of mmdebstrap and diffoscope that work with the new black in their
autopkgtest have already been uploaded to unstable and are waiting to
transition to testing so that black can transition as well. Maybe the timing of
this upload of black was a bit unfortunate this late into the freeze because
other packages are using it as part of their build process and now FTBFS. Like
another package of mine (botch) which is also fixed now and waiting:
https://bugs.debian.org/1031469

Re: -ffile-prefix-map option and reproducibility

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Stéphane Glondu (2023-02-07 16:41:47)
> When building packages, a -ffile-prefix-map option is automatically injected
> into CFLAGS. Where does it come from? Since when?

probably due to
https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/commit/?id=b60c243ba99b8483202a6f6a814476275204fdff

which references:

https://lists.debian.org/debian-devel/2020/10/msg00222.html
https://bugs.debian.org/974087

> I suspect this was added to improve reproducibility. Ironically, it makes
> packages that capture this variable non reproducible, since the build path
> seems to be randomized (has it always been the case? since when?). It is the
> case of OCaml (see #1030785), and seemingly of R as well (found by grepping
> in my /etc). I wouldn't be surprised other packages are affected as well.
> 
> Is there a way to not get this option? More elegant than explicitly 
> filtering it out of CFLAGS in debian/rules...

See the man page of dpkg-buildflags -- this might do what you want:

export DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath

Thanks!

cheers, josch

32bit arch packages are built with wrong ownership due to fakeroot bug

[Johannes Schauer Marin Rodrigues]

Hi,

I wanted to bring fakeroot bugs #1023286 and #1030638 to the attention of a
wider audience because even though I filed these bugs, I do not see myself
finding the time to track down and correct their consequence for all the source
package builds that happened since their introduction with the upload of glibc
2.34.

Essentially, fakeroot uses LD_PRELOAD to "fake" being able to change file
ownership as if the process were run as the root user. If a new version of
glibc gets uploaded that uses different functions to do the same thing, this
mechanism breaks down and fakeroot needs to be adjusted to support these new
functions as well. Most of the breakage (except for mipsel) was done as part of
#1023286 but I discovered another problem and reported as #1030638 and there is
no fix for it yet.

The problem affects 32 bit architectures as glibc chooses different functions
to perform stat() functionality on those architectures, so the problem effects
armel, armhf, i386 and mipsel. Those architectures also not among the most used
once which explains why not much fuzz was created about this yet (at least not
as far as I've heard).

By now the fakeroot package carries tests that check for these problems and you
can see how fakeroot FTBFS on the architectures affected by this problem:

https://buildd.debian.org/status/package.php?p=fakeroot

For the affected architectures, package builds that change ownership of the
files shipped in *.deb packages will remain at root:root instead of being set
to the user and group that should own them. I found two examples for this
problem: mutt and bsdutils

$ dpkg-deb --fsys-tarfile mutt_2.2.9-1_amd64.deb | tar --numeric-owner-tv | 
grep -v 0/0
-rwxr-sr-x 0/8   14584 2022-11-13 18:01 ./usr/bin/mutt_dotlock
$ dpkg-deb --fsys-tarfile mutt_2.2.9-1_i386.deb | tar --numeric-owner -tv | 
grep /usr/bin/mutt_dotlock
-rwxr-sr-x 0/0   13804 2022-11-13 18:01 ./usr/bin/mutt_dotlock

$ dpkg-deb --fsys-tarfile bsdutils_1%3a2.38.1-4_amd64.deb | tar --numeric-owner 
-tv | grep -v 0/0
-rwxr-sr-x 0/5   39224 2022-11-25 16:19 ./usr/bin/wall
$ dpkg-deb --fsys-tarfile bsdutils_1%3a2.38.1-4_i386.deb| tar --numeric-owner 
-tv | grep /usr/bin/wall
-rwxr-sr-x 0/0   38436 2022-11-25 16:19 ./usr/bin/wall

/usr/bin/mutt_dotlock should be owned by group "mail" and /usr/bin/wall by
group "tty" but this is only the case for the amd64 version of those two
packages. On i386, they are owned by the root group.

So fakeroot should get fixed and the affected source packages should be
rebuilt. I lack the time to organize either, so here I am writing you this
mail.

Maybe somebody else finds this interesting enough to invest their time into
this. :)

Thanks!

cheers, josch

Re: 32bit arch packages are built with wrong ownership due to fakeroot bug

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Santiago Vila (2023-02-09 17:32:08)
> El 9/2/23 a las 15:37, Johannes Schauer Marin Rodrigues escribió:
> > I wanted to bring fakeroot bugs #1023286 and #1030638 to the attention of a
> > wider audience because even though I filed these bugs,
> 
> Thanks for bringing this up!
> 
> Can you confirm if all this is correct?
> 
> - Only packages uploaded after 2022-08-07 (when glibc 2.34 hit unstable) are
> potentially affected.

Correct. 2022-08-07 is a lower bound.

Or in a bit more detail: only packages rebuilt after toolchain packages have
been rebuilt with glibc 2.34 can be affected. For example all problems I
observed so far came from coreutils (cp, chown, chgrp) which was only rebuilt
with glibc 2.34 when 9.1-1 was uploaded in October 2022. But of course there
could be other software that modifies ownership information during builds that
was rebuilt with glibc 2.34 earlier than that and that I just do not know
about. So 2022-08-07 is a lower bound but not necessarily the largest lower
bound.

> - Packages with "Rules-Requires-Root: no" are never affected.

Only packages built inside fakeroot are affected. As far as I understand it,
fakeroot is not used by dpkg-buildpackage if Rules-Requires-Root is set to
"no".

> - No intervention from individual maintainers is required for fixing this, as
> we already have a binNMU mechanism which we already use for transitions.

Once fakeroot is fixed, binNMUs can be used to fix packages, yes. Without the
fakeroot fix in place, individual maintainers could do things to fix their
packages on the affected architectures but I do not think doing so is a good
idea.

> - A minor observation: Only packages which use dh_fixperms with -X (or
> --exclude) are apparently affected. Those which instead do chmod/chown after
> dh_fixperms are apparently not affected, at least after the first fakeroot
> bug (2022-11) was fixed.

I do not understand what makes you think that only packages using dh_fixperms
-X are affected? I think what makes the two packages that I found fail to have
correct permissions is that they both use dh_install which in turn uses 'cp -a'
which is broken under fakeroot on our 32bit architectures right now. I patched
mutt and added an 'ls -lha' in execute_before_dh_builddeb to show the problem:

$ ls -lha debian/tmp/usr/bin/mutt_dotlock debian/mutt/usr/bin/mutt_dotlock
-rwxr-sr-x 1 root root 9.6K Feb  9 22:06 debian/mutt/usr/bin/mutt_dotlock
-rwxr-sr-x 1 root mail  28K Feb  9 22:05 debian/tmp/usr/bin/mutt_dotlock

So the chgrp call in Makefile.am worked correctly and set the group owner to
"mail" but after dh_install moved mutt_dotlock from debian/tmp/ to debian/mutt/
using 'cp -a' (if I'm reading the code correctly) the group ownership
information is lost.

Thanks!

cheers, josch

Re: 32bit arch packages are built with wrong ownership due to fakeroot bug

[Johannes Schauer Marin Rodrigues]

Quoting Santiago Vila (2023-02-10 11:12:09)
> El 10/2/23 a las 3:18, Johannes Schauer Marin Rodrigues escribió:
> > I do not understand what makes you think that only packages using 
> > dh_fixperms
> > -X are affected? I think what makes the two packages that I found fail to 
> > have
> > correct permissions is that they both use dh_install which in turn uses 'cp 
> > -a'
> > which is broken under fakeroot on our 32bit architectures right now. I 
> > patched
> > mutt and added an 'ls -lha' in execute_before_dh_builddeb to show the 
> > problem:
> > 
> > $ ls -lha debian/tmp/usr/bin/mutt_dotlock debian/mutt/usr/bin/mutt_dotlock
> > -rwxr-sr-x 1 root root 9.6K Feb  9 22:06 debian/mutt/usr/bin/mutt_dotlock
> > -rwxr-sr-x 1 root mail  28K Feb  9 22:05 debian/tmp/usr/bin/mutt_dotlock
> > 
> > So the chgrp call in Makefile.am worked correctly and set the group owner to
> > "mail" but after dh_install moved mutt_dotlock from debian/tmp/ to 
> > debian/mutt/
> > using 'cp -a' (if I'm reading the code correctly) the group ownership
> > information is lost.
> 
> I mean: There are mainly two ways to ship files not root:root inside a 
> package.
> 
> One way is to accept the special permissions/ownerships resulting from 
> dh_install
> and then avoid resetting them in dh_fixperms using -X or --exclude.
> This is from mutt's debian/rules:
> 
> override_dh_fixperms:
>  dh_fixperms --exclude usr/bin/mutt_dotlock
> 
> The other way is not to care about the permissions from dh_install and set 
> them
> after dh_fixperms no matter what. This is from procmail's debian/rules:
> 
> override_dh_fixperms:
>  dh_fixperms
>  chgrp mail debian/procmail/usr/bin/procmail
>  chgrp mail debian/procmail/usr/bin/lockfile
>  chmod 6755 debian/procmail/usr/bin/procmail
>  chmod 2755 debian/procmail/usr/bin/lockfile
> 
> The minor observation is that packages which set permissions in the second
> way which were uploaded after the first fakeroot bug was fixed seem
> not to be affected:
> 
> dpkg -c procmail_3.24-1_i386.deb | grep -v root/root
> -rwxr-sr-x root/mail 26292 2023-01-05 22:35 ./usr/bin/lockfile
> -rwsr-sr-x root/mail112668 2023-01-05 22:35 ./usr/bin/procmail

Firstly, when talking about "after the first fakeroot bug was fixed", remember
that while it was fixed for i386 it was not fixed for mipsel. So all timestamps
we use in this discussion depend on the architecture we are talking about.

Secondly, this is still not about dh_fixperms so much as it is about dh_install
being called *before* the chgrp and chmod calls. You could add these calls in
other dh overrides as long as those are executed after dh_install has run. For
example you could place them into execute_before_dh_builddeb and achieve the
same effect. But yes, since the fundamental difference in these two styles of
how dh_fixperms is overridden makes the group owneership either depend on 'cp
-a' (which is broken) or on another manual chgrp (which works) you could call
it like that, I guess.

I didn't want to go into this in my last mail for similar reasons as mentioned
by smcv in their reply but since you are now explicitly asking about it: it's
true that yes, while not required, individual maintainers *could* do something
about this problem even without fakeroot being fixed: add a manual chgrp call
into their d/rules.  As long as that chgrp call happens *after* dh_install
(either in execute_after_dh_fixperms or in execute_before_dh_builddeb, for
example) and *on* the files installed by dh_install, it will correct the
missing ownership information due to 'cp -a' usage in dh_install.

This will of course just paper over the issue without fixing the underlying
cause. It's what I called "not a good idea" in my last mail.

Thanks!

cheers, josch

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Johannes Schauer Marin Rodrigues]

Quoting Marco d'Itri (2023-06-09 09:41:43)
> On Jun 08, Raphael Hertzog  wrote:
> > And creating the required symlinks would be done by those (standalone)
> > maintainer scripts...
> > 
> > I don't know if we already have some rule/invariant in the configuration
> > order of the unpacked packages, but I doubt so.
> Indeed, this would be very simple and it has already been proposed.
> But somebody then complained that special-casing a package would violate 
> the design contraints he self-imposed to his own image building tool, 
> and as we all know every Debian maintainer can veto any systemic changes that
> they do not like.

I definitely complained about special-casing a package because it would violate
the design contract for my own image building tool. You would not by any chance
be talking about me in your last message, would you?

Anyway, great communication style. This will totally help bringing us all
together to create a great operating system. Very productive. I already feel a
lot more motivated to discuss technical matters with you.

Now who do I have to talk to in case I'd really like to veto something?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Helmut Grohne (2023-06-09 15:22:39)
> Add a new package usrmerge-support (or whatever). It is a bit similar to
> multiarch-support: It must not have any dependencies or pre-dependencies. It
> will not have files, but maintainer scripts. Those scripts set up protective
> diversions on behalf of base-files for the symbolic links that cause
> aliasing. Then base-files will issue a Pre-Depends on usrmerge-support (but
> not yet ship symlinks). I initially thought, this could be part of
> usr-is-merged, but then base-files would pull that and standard mmdebstrap
> would no longer pull usrmerge and break. So it really needs to be a separate
> package. Anyway, once we have protective diversions, we can move files
> without risking that dpkg deletes the symbolic links.
> 
> Then we can actually perform that move of files to their canonical
> locations except for a small set of locations including dash, bash,
> libc6, and util-linux (maybe not exhaustive). [There is a lot of missing
> detail about non-bootstrap aspects here.]
> 
> Once all essential packages (but the exceptions) have no files left in
> aliased locations, we can upload base-files adding the symlinks together
> with the packages previously kept unmodified in one dinstall. Before
> that dinstall, things will continue to work normally. The protective
> diversions will not affect unpacking, because dpkg only performs exact
> matches on diversions. After that dinstall, base-files will create the
> symlinks and things will hopefully work (because the patched debootstrap only
> creates them after the initial unpack).

if I understand that plan correctly, the usrmerge-support package setting up
diversions is only necessary because you want to avoid having to do the move to
/usr of *all* affected packages in the essential set in a single dinstall? Is
that correct?

If yes, how many source packages are we have to be modified part from
base-files, dash, bash, libc6, and util-linux?

Is it just these? audit bzip2 coreutils debianutils dpkg gcc-13 grep gzip
hostname libcap2 libcap-ng libgpg-error libselinux libxcrypt ncurses pam sed
shadow sysvinit tar xz-utils zlib

Would it be too much to prepare patches for all of these, test that everything
works with some QA setup and then NMU all 22 source packages with pre-approved
patches in a single dinstall? Would that avoid having to temporarily go via a
usrmerge-support package?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: /usr-merge: continuous archive analysis

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Helmut Grohne (2023-07-12 15:34:38)
> This thread hopefully becomes more of a FYI than a discussion. I've turned
> those hacky scripts into some Python code that continuously (4 times a day)
> analyzes the archive for some of the problems summarized in DEP17. Interested
> parties may find the code at https://salsa.debian.org/helmutg/dumat. The
> results of the analysis[1] (around 2000 lines) are updated at
> https://subdivi.de/~helmut/dumat.yaml.

if you (like me) nearly didn't click that link because the number 2000 sounds
scarily large, don't be scared:

$ curl --silent https://subdivi.de/~helmut/dumat.yaml \
| python3 -c 'import 
yaml,sys;print("\n".join(yaml.safe_load(sys.stdin).keys()))' \
| wc -l
92

So these 2000 lines describe just 92 binary packages from 77 source packages.
Attached is the dd-list output for the above.

Thanks!

cheers, josch"Adam C. Powell, IV" 
   oce (U)

A. Maitland Bottoms 
   airspyhf (U)
   airspyone-host
   bladerf
   hackrf
   libiio
   libmirisdr
   libosmosdr
   rtl-sdr

Adrian Knoth 
   libffado (U)

Alessio Treglia 
   libnjb

Andrea Pappacoda 
   opensysusers

Andreas B. Mundt 
   libticables (U)

Andrej Shadura 
   pkgconf

Andrew Pollock 
   isc-dhcp (U)

Android Tools Maintainers 
   android-platform-tools

Anuradha Weeraman 
   ksh93u+m

Arne Bernin 
   libfreenect (U)

Bernhard Schmidt 
   mediastreamer2 (U)

CFEngine Team 
   cfengine3

Chris Halls 
   libreoffice (U)

Chris Hofstaedtler 
   util-linux (U)

Christoph Berg 
   hamlib (U)

Christoph Martin 
   cfengine3 (U)
   nfs-ganesha (U)

Damyan Ivanov 
   firebird3.0
   firebird4.0

Daniel Baumann 
   bfh-metapackages
   nvme-cli
   open-infrastructure-compute-tools
   open-infrastructure-storage-tools
   progress-linux-metapackages
   zutils

Debian BOINC Maintainers 
   boinc

Debian Ecosystem Init Diversity Team 

   libpam-elogind-compat

Debian EFI 
   fwupd

Debian Electronics Packaging Team 

   libsigrok

Debian GCC Maintainers 
   gcc-13
   gcc-snapshot

Debian Go Packaging Team 
   libpod

Debian Go Packaging Team 
   golang-github-blynn-nex

Debian Hamradio Maintainers 
   airspyhf
   hamlib

Debian ISC DHCP Maintainers 
   isc-dhcp

Debian LibreOffice Maintainers 
   libreoffice

Debian Med Packaging Team 
   eegdev

Debian Mono Group 
   mono

Debian Multimedia Maintainers 
   kodi
   libffado
   libmpeg3
   openni2

Debian OCaml Maintainers 
   ocaml-afl-persistent

Debian PhotoTools Maintainers 
   libgphoto2

Debian Printing Team 
   hplip

Debian Python Team 
   discodos
   jupyter-notebook

Debian QA Group 
   nvi
   powerman

Debian Science Maintainers 
   libticables
   oce
   opencascade

Debian Security Tools 
   cryptsetup-nuke-password

Debian systemd Maintainers 
   systemd

Debian sysvinit maintainers 
   orphan-sysvinit-scripts

Debian VoIP Team 
   mediastreamer2

Denis Barbier 
   oce (U)

Dimitri John Ledkov 
   what-is-python (U)

Dirk Eddelbuettel 
   gretl

Dmitry Smirnov 
   libpod (U)

Ervin Hegedus 
   hamlib (U)

Felipe Sateler 
   systemd (U)

Felix Lechner 
   mediastreamer2 (U)

Ferenc Wágner 
   libgphoto2 (U)

FingerForce Team 
   libfprint

Francois Marier 
   molly-guard (U)

Free Ekanayaka 
   libffado (U)

Frédéric Bonnard 
   rear

Georges Khaznadar 
   expeyes

Gianfranco Costamagna 
   boinc (U)
   llvm-toolchain-snapshot (U)

Gordon Ball 
   jupyter-notebook (U)

Guo Yixuan (郭溢譞) 
   boinc (U)

Gürkan Myczko 
   cadabra2

Ian Jackson 
   libpam-elogind-compat (U)

IOhannes m zmölnig (Debian/GNU) 
   libmpeg3 (U)

Jo Shields 
   mono (U)

Jochen Sprickerhof 
   openni2 (U)

Johannes Tiefenbacher 
   discodos (U)

John Scott 
   open-ath9k-htc-firmware

Jonas Meurer 
   cryptsetup-nuke-password (U)

Jonathan McDowell 
   libsigrok (U)

Josh Triplett 
   molly-guard (U)

Julien Puydt 
   ocaml-afl-persistent (U)

Jérôme Charaoui 
   puppet-agent (U)

Jörg Frings-Fürst 
   sane-backends

Ken McDonell 
   pcp (U)

Kilian Krause 
   mediastreamer2 (U)

Kurt Kremitzki 
   opencascade (U)

Laszlo Boszormenyi (GCS) 
   fuse3

LLVM Packaging Team 
   llvm-toolchain-snapshot

Lorenzo Puliti 
   runit

Luca Boccassi 
   systemd (U)

Ludovic Rousseau 
   libnfc (U)

Ludovico Gardenghi 
   molly-guard (U)

Marc Haber 
   molly-guard (U)

Marco d'Itri 
   systemd (U)

Marco Trevisan 
   libfprint (U)

Mario Limonciello 
   fwupd (U)

Mark Hindley 
   libpam-elogind-compat (U)

Mark Renouf 
   libfreenect (U)

Martin Pitt 
   systemd (U)

Matthew Vernon 
   orphan-sysvinit-scripts (U)

Matthias Klose  
   what-is-python

Matthias Klose 
   gcc-13 (U)
   gcc-snapshot (U)

Matthias Klumpp 
   fwupd (U)

Michael Biebl 
   systemd (U)

Michael Gilbert 
   isc-dhcp (U)

Mirco Bauer 
   mono (U)

Nathan Scott 
   pcp (U)

Nicolas Bourdaud 
   eegdev (U)
   libfreenect

Nobuhiro Iwamatsu 
   libnfc
   openni2 (U)

PCP Development Team 
   pcp

Petter Reinholdtsen 
   libmpeg3 (U)

Philippe Deniel 
   nfs-ganesha

Puppet Package Maintainers 
   

Re: pre-MBF: fail to build (repack) source

[Johannes Schauer Marin Rodrigues]

Quoting Simon McVittie (2023-07-17 12:10:52)
> On Mon, 17 Jul 2023 at 08:57:51 +, Holger Levsen wrote:
> > On Sun, Jul 16, 2023 at 11:41:36AM +0100, Simon McVittie wrote:
> > > On Sun, 16 Jul 2023 at 00:03:00 +0200, Adam Borowski wrote:
> > > > due to Build-Depends
> > > > being inadequate (per the Policy, B-D-Indep are _not_ necessarily
> > > > installed)
> > >
> > > For completeness, B-D-Arch are not guaranteed to be installed during
> > > clean either.
> >  
> > I wonder what's the point of B-D-Arch And B-D-Indep then?
> 
> The point is the same as it always was: primarily to exclude large
> dependencies from a `dpkg-buildpackage -B` build chroot (like the official
> buildd for each architecture) if they are only needed when building
> Architecture: all packages such as documentation, and secondarily to
> exclude large dependencies from a `dpkg-buildpackage -A` build chroot
> (like the official "all" buildd) if they are only needed when building
> Architecture: any packages.

additionally, moving build dependencies from Build-Depends to
Build-Depends-Indep is also one of the least disruptive ways to make a Debian
source package cross-compilable in cases where build dependencies that do not
work in a cross-context are only used to build the arch:all packages.

https://wiki.debian.org/CrossBuildPackagingGuidelines#Declare_.22Indep.22_Build-Dependencies

Moving stuff to Build-Depends-Indep also helps the architecture bootstrap
use-case as arch:all packages do not have to be bootstrapped for a new
architecture and the fewer build dependencies remain in Build-Depends or
Build-Depends-Arch the fewer build dependency cycles remain.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Re: i386 in the future (was Re: 64-bit time_t transition for 32-bit archs: a proposal)

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Diederik de Haas (2023-05-31 00:51:06)
> > If people have strong opinions about that plan, let us know please.
> 
> I have *strong* opinions about this.
> 
> https://lists.debian.org/debian-kernel/2023/01/msg00372.html was a message/
> plea to not forget about supporting OLD systems.
> 
> While it may be a no-brainer for a person with a $/€ 1000 a month residual 
> income to just buy new hardware whenever they feel like it, that is not the 
> case for everyone.

I think that depends on where you live. As Steve has said, if you live in a
place with tons of rich people around, so many "old" computers are discarded by
them that it's not a problem for anybody to get hold of one of those for near
to nothing. People with too much money just go through way too many computers
per year, thereby creating a vast amount of old but still usable computers. At
my university I recently saw a whole container filled with "old" desktop
machines to be discarded (systems from 10 years ago, so definitely 64bit
machines). This is just disturbing in my opinion but hey, those old systems
don't run the most recent MS Windows anymore...

This situation is probably very different around the world and I guess there
are many places where it is very hard to get hold of a machine younger than a
decade? Are you talking about those places?

> Besides people in 'third world countries' (I actually don't like such
> qualifications at all), there are also people in the '1st world' who work
> their asses off just to put food on the table, and thus also don't have the
> money to buy new equipment. But if you want to interact with your own
> government, you highly likely will need to have some PC (type) equipment.  It
> could also provide a way to learn/develop new skills.

In my own "1st world" country I know a number of people in that situation and
at least over here, a "computer" doesn't help them to do the daily life things.
They need a smartphone to stay connected with their employer via Whatsapp or
download the apps to participate in the things everybody else participates in.
In Germany they just rolled out a monthly ticket for trains and buses for the
whole country which will be smartphone-only starting next year -- will a person
with less income invest in a new/old desktop machine or in a smartphone new
enough to run such an app? Yes, this is another discussion but I also do not
think your argument applies well to "1st world" countries because of this
reason and the reasons I gave above.

> It's absolutely true that modern machines are more energy efficient. What is
> also true is that the production of new devices has a big environmental
> impact.
> 
> https://mastodon.green/@gerrymcgovern/110329331475328263 said:
> > The European Environmental Bureau has stated that extending the lifespan of
> > smartphones and other electronics by just one year would save the EU as
> > much carbon emissions as taking two million cars off the roads annually.
> 
> I would be VERY disappointed if Debian would abandon people who do NOT have
> the means to just buy new equipment whenever they feel like it.

That argument goes both ways. You could also say that for people with less
income, the electricity costs make using a more modern system cheaper for them.

This of course does not negate the environmental argument but the environmental
argument is a tricky one. A 20 year old machine bought 20 years ago will have
racked up a lot of electricity usage which pales against the energy required to
build and run small single-board machines that are similarly powerful built
today. There is a cut-off point where using an energy-hungry old system *does*
have a higher environmental impact than building a small new system.

> Especially when I see various proposals to make the 'life'/work of companies
> who make BILLIONS a YEAR, easier.  (I'll leave my moral objections to several
> of those aside this time)
> 
> Cheers,
>   Diederik
> 
> PS: Nothing in here should be taken as a personal attack, but as I said I 
> feel 
> rather strongly about this subject. (And communication isn't my strong suit)

There are probably many places in the world where your argument applies well.
Remember though, that this is also a person-power problem. If we can find many
more people interested to keep 20 year old systems alive by working on that in
their free-time, I do not think we would have this discussion. A lot of work is
required to keep an architecture and its installer alive. I suspect kibi would
*love* more hands helping maintain d-i. There will always be someone with real
reasons for using 20 year old systems and wanting to do a fresh installation on
one. The question is, is this worth our free-time or should we do other things
instead? Who is having fun doing that?  We can argue a lot about the social and
environmental reasons for supporting 20 year old systems but the sad fact of
the matter is: if there is nobody there doing the work then that discussion is

Re: debci / salsa ci: support for qemu runner

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Michael Biebl (2023-07-25 16:16:35)
> apparently, we in Debian struggle to find good opportunities where to spend
> our money.
> 
> I think support for qemu runners, i.e. supporting isolation-machine in 
> autopkgtest on both debci and salsa ci would be an excellent opportunity.

is this just about isolation-machine or is this about architectures where qemu
support exists but no real hardware?

If it's about isolation-machine I have "solved" this problem in the past by
creating a qemu virtual machine as part of the autopkgtest and then run my test
inside that. This is slow because kvm is not available but it works for small
tests.

As far as I'm concerned, having kvm available in debci and salsaci would be a
big plus.

No idea if that is easier or harder than what Michael proposed.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Potential MBF: packages failing to build twice in a row

[Johannes Schauer Marin Rodrigues]

Quoting Timo Röhling (2023-08-05 21:07:34)
> * Lucas Nussbaum  [2023-08-05 17:06]:
> >An example sbuild invocation to reproduce failures is:
> [omitted the command line equivalent of Tolstoy's War and Peace]
> 
> If we decide that this issue is important enough that people should
> care and mass bugs be filed, sbuild will need a more concise way to test
> this; something like pbuilder's --twice option.

if somebody feels strongly that a --twice option is needed for sbuild I'd be
happy to review and merge a MR against the sbuild packaging git on salsa:

https://salsa.debian.org/debian/sbuild/

Thanks!

cheers, josch

signature.asc
Description: signature

Re: autodep8 test for C/C++ header

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Benjamin Drung (2023-08-07 20:43:36)
> while working a whole week on fixing failing C/C++ header compilations for
> armhf time_t [1], I noticed a common pattern: The library -dev packages was
> missing one or more dependencies on another -dev package.  Over 200 -dev
> packages are affected.
> 
> I propose to add an autodep8 test for C/C++ header files that tries to
> compile the header file. This will catch issues like missing
> dependencies and other issues.
> 
> Here is a draft for the autopkgtest check script that takes a binary
> -dev package as parameter:
> https://github.com/bdrung/bdrung-scripts/blob/compile-header-check/compile-header-check
> 
> What do you think? Should I proceed developing and packaging this check
> script and submit support for it in autodep8?

yes please!

I've manually written similar scripts for my own packages in the past with the
same reason: find missing dependencies of -dev packages. Here is an example of
what I wrote for ros-* packages:

https://sources.debian.org/src/ros-image-pipeline/1.17.0-1/debian/tests/compilation/

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Potential MBF: packages failing to build twice in a row

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Guillem Jover (2023-08-09 20:55:17)
> On Wed, 2023-08-09 at 19:55:41 +0200, Johannes Schauer Marin Rodrigues wrote:
> > I would only consider switching the default if at the same time, some checks
> > were done that made sure that the result is bit-by-bit identical to the
> > original.
> > 
> > The source package is the *input* to sbuild not its output. If sbuild builds
> > the source package it can happen that the resulting source package is not 
> > what
> > was given to sbuild to get built before.
> > 
> > So if the source package gets rebuilt and checked whether it is bit-by-bit
> > identical to what was given to sbuild before, then essentially we would've
> > enforced reproducible source packages. If I remember correctly, reproducible
> > source packages are something that the reproducible builds team discarded 
> > as a
> > concept many years ago.
> 
> I think I've mentioned this before, but dpkg-source is supposed to be
> generating reproducible source packages since around the time dpkg-deb
> has been generating reproducible binary packages. If that's not the case
> in some circumstance I'd consider that a bug worth fixing (or at least
> pondering whether it makes sense to support :).

it has been a long time since I've analyzed this so things might've changed
indeed since then. But what I remember is that, depending on the source
package, running sbuild with --source would produce a different source package
than was originally passed to sbuild. I tried running this on a few source
packages to see if I can reproduce this problem today:

sbuild --source --arch-all --arch-any -d unstable --no-run-lintian \
--no-run-autopkgtest \
--starting-build-commands='grep -E "^ [a-f0-9]{64} " *_*.dsc > before' \
--finished-build-commands='grep -E "^ [a-f0-9]{64} " *_*.dsc | diff -u 
before -'

Which prints for src:hello this:

--- before  2023-08-09 19:46:05.092628335 +
+++ -   2023-08-09 19:46:25.873292249 +
@@ -1,3 +1,3 @@
  31e066137a962676e89f69d1b65382de95a7ef7d914b8cb956f41ea72e0f516b 
725946 hello_2.10.orig.tar.gz
  4ea69de913428a4034d30dcdcb34ab84f5c4a76acf9040f3091f0d3fac411b60 819 
hello_2.10.orig.tar.gz.asc
- 60ee7a466808301fbaa7fea2490b5e7a6d86f598956fb3e79c71b3295dc1f249 
12684 hello_2.10-3.debian.tar.xz
+ 84b14a8c49f9bca8d6c7a5550fed71790e147576c8eb716b2afbd49df4d5a7a9 
12692 hello_2.10-3.debian.tar.xz


I ran diffoscope on the differing debian.tar.xz files and got:

--- ../hello_2.10-3.debian.tar.xz.bak
+++ ../hello_2.10-3.debian.tar.xz
│┄ Format-specific differences are supported for XZ compressed files 
but no file-specific differences were detected; falling back to a binary diff. 
file(1) reports: XZ compressed data, checksum CRC64

I suspect that different versions of xz produce differently compressed
archives?

In any case, I have no time for a more thorough analysis right now but this was
an example that makes my point: passing --source to sbuild might overwrite your
existing source package with something different and thus it's not trivial
anymore to assure that what you built really came from those exact same sources
as the source that the built was done from was not bit-by-bit identical to the
source produced by sbuild --source. In fact the .buildinfo file produced by
sbuild will reference the *new* dsc and that was not the dsc that the built was
done from. So my point stands: avoid running sbuild with --source.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Potential MBF: packages failing to build twice in a row

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Stefano Rivera (2023-08-09 14:38:56)
> Personally, I have my sbuild configured to build a source package after the
> build, so that I can be sure that I don't regress my own packages' clean
> target. It would be nice if this was a default feature in sbuild, for most
> packages this is a very quick process.

I would only consider switching the default if at the same time, some checks
were done that made sure that the result is bit-by-bit identical to the
original.

The source package is the *input* to sbuild not its output. If sbuild builds
the source package it can happen that the resulting source package is not what
was given to sbuild to get built before.

So if the source package gets rebuilt and checked whether it is bit-by-bit
identical to what was given to sbuild before, then essentially we would've
enforced reproducible source packages. If I remember correctly, reproducible
source packages are something that the reproducible builds team discarded as a
concept many years ago.

So what should be the plan instead?

Thanks!

cheers, josch

signature.asc
Description: signature

snapshot.d.o has been in a bad state for several months

[Johannes Schauer Marin Rodrigues]

Hi,

snapshot.debian.org is getting worse again. There is not a single snapshot for
August yet and the last days of July are spotty:

http://snapshot.debian.org/archive/debian/?year=2023=7

None for the 29. and only a single timestamp for the 26., 27., 28. and 30.
There should be four per day. The situation is even worse for other archives.
For debian-ports, for the month of July, there are only 22 snapshots overall:

http://snapshot.debian.org/archive/debian-ports/?year=2023=7

This problem has been known for half a year already:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1031628

But that bug got closed in favor of #1029744 which was filed because
debian-ports had no snapshots at all for January and only three for February
this year but there is no reply to that bug.

In #1031628 Julien said that there is "not much we can do about it at the
moment".

What is the status of this problem? What is needed to fix it? Is this just a
problem of computational and/or storage resources which an be fixed by the
funds available to Debian?

I'd argue that snapshot.d.o is part of the central services Debian provides and
it should work better than it does right now.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Johannes Schauer Marin Rodrigues]

Hi,

mmdebstrap author here. This is the other bootstrapping tool which is currently
sitting at ~17% of the popcon value of debootstrap.

Quoting Helmut Grohne (2023-06-28 21:37:44)
> Once that is settled, the next big question is how to handle bootstrapping.
> We had a number of people arguing in favour of changing the bootstrap
> protocol. Such changes can be classified into generic changes and
> release-dependent changes. A release-dependent change enhances bootstrapping
> tools with knowledge of available releases and adapts behaviour in
> release-specific ways that are encoded into the bootstrapping tool. As it
> stands, the only bootstrapping tool that has been enhanced in this way is
> debootstrap and that support is limited to Debian and two derivatives. The
> category of generic changes includes imposing an ordering on initial unpacks
> (e.g. base-files first). Others are in favour of not changing the bootstrap
> protocol. In that view, some data.tar will have to ship the symbolic links
> and all other essential packages need to have their files canonicalized.
> 
> Among these options, the first has a working prototype (debootstrap),
> but it is unclear how that extends to use of snapshot.d.o and how to
> make it work with debootsnap and debbisect as those tend to use a suite
> name rather than a codename. The last option has a prototype and relies
> on uploading a number of essential packages in a short window of time.
> (What could possibly go wrong?)
> 
> It is not clear to me how we can get to a consensus on these, so the
> best I can do here is summarize options.
> 
> Option #3a:
> 
> The bootstrap protocol shall be changed to contain a task for
> merging /usr as is done in debootstrap in a release-dependent way.
> 
> This is an instance of M16 from DEP17.
> 
> Option #3b:
> 
> The bootstrap protocol shall be changed in unspecified ways to
> support the /usr-merged systems in a way that does not depend on
> matching the codename or suitename.
> 
> This is an instance of M16 from DEP17.
> 
> Option #3c:
> 
> The bootstrap protocol shall remain unchanged. Therefore, all
> essential packages need to move their files out of aliased locations
> and the aliasing symlinks are to be installed from a data.tar of a
> binary package such as base-files.
> 
> This is M2+M11 from DEP17.
> 
> While a few people including Marco d'Itri and Sam Hartman have argued in
> favour of exploring the space of #3b, no proposals have emerged in the
> interim. The proposal in #3a has three significant limitations:
>  * It creates compatibility issues when combining old a new suites
>unless changes to bootstrapping tools are backported to older
>releases.
>  * It becomes a whack-a-mole, since we need to add codenames of every
>derivative to every bootstrapping implementation.
>  * It breaks bootstrapping from snapshot.d.o and therefore breaks tools
>such as debbisect and debootsnap.
> 
> While the first of these limitations is shared with #3b, the others are
> not and that would make #3b more attractive to me if there was a
> concrete proposal to evaluate. The one about unpacking base-files first
> seemed the most concrete to me, but it has the downside of imposing a
> permanent cost on bootstrapping tools even though we only need that
> behaviour temporarily, which seems like too bad of a trade-off to start
> with in my opinion. Did I miss a relevant proposal for modifying the
> bootstrap protocol?
> 
> On the flip side, there is a demo for #3c showing that we can move most
> of the things except for a hand full of packages and then flip the
> switch (for bootstrapping) in unstable by uploading those packages
> simultaneously. The biggest downside of this probably is the inherent
> fragility of this approach. Even if this is extensively tested before
> uploading chances are good that we still break something unforeseen in
> the process.
> 
> Can I get more feedback from those who rather not have #3c implemented as to
> how you see things moving forward?

there was some feedback of people who'd rather not have #3c implemented. I
agree with them that changing bootstrapping tools is a quick and easy fix that
only requires a few lines of easily testable code, can be deployed quickly and
is unlikely to break unrelated things in a serious way.

In this email I'd like to explain why I think that #3a and #3b are not good
ways forward for the distribution. In summary, what I fear is, that choosing
either #3a or #3b would mean choosing a simple fix just because it's simple and
ignoring the fact that those solutions mean a permanent cost for the
distribution (even beyond debootstrap) for many years to come. What I'd like
Debian to choose would be the technically and architecturally proper solution
even if it means more work but results in multiple long-term benefits. To
expand further on why I think this way, let me go back in time a bit.

Around 7 years ago I started being