Bug#1023922: dpkg-checkbuilddeps: please add an option to print the list of packages

[Guillem Jover]

On Mon, 2022-11-14 at 13:38:30 +0100, Johannes Schauer Marin Rodrigues wrote:
> Quoting Guillem Jover (2022-11-14 13:17:34)
> > On Sun, 2022-11-13 at 11:22:50 +0100, Enrico Zini wrote:
> > > The way I chopped dpkg-checkbuilddeps was a first approximation. Given
> > > that now we have `apt-get satisfy`, my next step would be to have my
> > > hacked version print a list of arguments for it, which can include
> > > "Conflicts:", but which can already be preprocessed to reduce some noise
> > > like packages required or not required from the target architecture.
> > 
> > Ah, so from this I gather that in essence what we need is a way to map
> > from build-dependencies into run-time dependencies, removing/filtering
> > them from anything that is not accepted in the latter. That makes
> > sense and does not seem to have the concerns of the previously filed
> > bug request, as that required performing decisions in a layer too low
> > where that required information was not available.
> > 
> > I could see gathering any build-dependency fields as restricted by
> > (-A/-B/-I), remapping them based on current arch/profile, then
> > outputting them as a pair of Depends/Conflicts fields (perhaps even
> > an Architecture field if there was arch-restrictions applied?). For
> > «apt satisfy» you might need to trim the «Depends: » part though.
> > 
> > Would that work for you? I think it would work for pbuilder.
> > 
> > > More generally, I'd like Debian to have, as a standard, something
> > > similar to `rpmspec --parse filename.spec | grep BuildRequires`
> > > because I see it reimplemented so many times (pbuilder, sbuild, and so
> > > on) that my instincts screams invoking the rule of three and
> > > refactor[2].
> > 
> > I think the above would solve your problem, and potentially could
> > substantially reduce the code in pbuilder-satisfydepends. For sbuild
> > and mk-build-deps (devscripts), which are already in perl, that would
> > only potentially help if this is included as part of a public module.
> > So I'll be going that way in case they want to (eventually) switch.
> 
> As far as I can see, this additional feature will not break anything in 
> sbuild,
> so I think I was CC-ed because the question is whether sbuild can use this? In
> an earlier mail, Enrico writes:

Sorry, probably the phrasing and placement of my reply made this
unclear, with above I meant my previous reply, where I was referring to
adding new code in a module to handle this mapping, the bulk of that
mapping is already being done by deps_parse(), so I was thinking
something a bit more higher-level, but I've not checked the various
implementations (including the ones within dpkg-dev), whether they'd
benefit from something more higher-level than deps_parse().

I'd imagine something taking a source deb822 control stanza from
debian/control or the entire thing, and then mapping
build-dependencies in there into Depends and Conflicts (according to
some selectable criteria or similar).

Thanks,
Guillem

Bug#1023922: dpkg-checkbuilddeps: please add an option to print the list of packages

[Johannes Schauer Marin Rodrigues]

Hello everybody! :)

Quoting Guillem Jover (2022-11-14 13:17:34)
> [ CCing devscripts, pbuilder and sbuild, as this is about some
>   potential functionality refactoring. ]

thank you!

> On Sun, 2022-11-13 at 11:22:50 +0100, Enrico Zini wrote:
> > On Sat, Nov 12, 2022 at 09:30:43PM +0100, Guillem Jover wrote:
> > > There was a bug filed requesting adding custom output format support
> > > (#214566) but it was closed “recently”. I think there might be some
> > > value in that, but not for the intended use the submitters seemed
> > > to want it.
> > > 
> > > I'd be interested to know how you'd want to use this new output/option
> > > as from the PoC script you provide it is not obvious to me, as it
> > > prints both build-depends and build-conflicts in an indistinguishable
> > > way, and it includes version constraints and alternative dependencies.
> > 
> > My specific use case at the moment is setting up a container
> > *description* (not a container) with all the dependencies I need to do
> > development on a package[1].
> > 
> > I could run apt-get build-dep inside the container and get the
> > development environment installed, but then I lose the ability of being
> > able to describe it in a terse way, and I can only do something along
> > the lines of listing all installed packages in the container and their
> > versions, which is too noisy for an average bug report.
> 
> Ok, so basing this description on .buildinfo would not seem to be
> satisfactory then.
> 
> > The way I chopped dpkg-checkbuilddeps was a first approximation. Given
> > that now we have `apt-get satisfy`, my next step would be to have my
> > hacked version print a list of arguments for it, which can include
> > "Conflicts:", but which can already be preprocessed to reduce some noise
> > like packages required or not required from the target architecture.
> 
> Ah, so from this I gather that in essence what we need is a way to map
> from build-dependencies into run-time dependencies, removing/filtering
> them from anything that is not accepted in the latter. That makes
> sense and does not seem to have the concerns of the previously filed
> bug request, as that required performing decisions in a layer too low
> where that required information was not available.
> 
> I could see gathering any build-dependency fields as restricted by
> (-A/-B/-I), remapping them based on current arch/profile, then
> outputting them as a pair of Depends/Conflicts fields (perhaps even
> an Architecture field if there was arch-restrictions applied?). For
> «apt satisfy» you might need to trim the «Depends: » part though.
> 
> Would that work for you? I think it would work for pbuilder.
> 
> > More generally, I'd like Debian to have, as a standard, something
> > similar to `rpmspec --parse filename.spec | grep BuildRequires`
> > because I see it reimplemented so many times (pbuilder, sbuild, and so
> > on) that my instincts screams invoking the rule of three and
> > refactor[2].
> 
> I think the above would solve your problem, and potentially could
> substantially reduce the code in pbuilder-satisfydepends. For sbuild
> and mk-build-deps (devscripts), which are already in perl, that would
> only potentially help if this is included as part of a public module.
> So I'll be going that way in case they want to (eventually) switch.

As far as I can see, this additional feature will not break anything in sbuild,
so I think I was CC-ed because the question is whether sbuild can use this? In
an earlier mail, Enrico writes:

> More generally, I'd like Debian to have, as a standard, something
> similar to `rpmspec --parse filename.spec | grep BuildRequires`
> because I see it reimplemented so many times (pbuilder, sbuild, and so
> on) that my instincts screams invoking the rule of three and
> refactor[2].

I like to remove code so that I don't have to care about it anymore. But I
don't understand which part of sbuild can be replaced by this. Enrico, can you
explain?

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#1023922: dpkg-checkbuilddeps: please add an option to print the list of packages

[Guillem Jover]

Hi!

[ CCing devscripts, pbuilder and sbuild, as this is about some
  potential functionality refactoring. ]

On Sun, 2022-11-13 at 11:22:50 +0100, Enrico Zini wrote:
> On Sat, Nov 12, 2022 at 09:30:43PM +0100, Guillem Jover wrote:
> > There was a bug filed requesting adding custom output format support
> > (#214566) but it was closed “recently”. I think there might be some
> > value in that, but not for the intended use the submitters seemed
> > to want it.
> > 
> > I'd be interested to know how you'd want to use this new output/option
> > as from the PoC script you provide it is not obvious to me, as it
> > prints both build-depends and build-conflicts in an indistinguishable
> > way, and it includes version constraints and alternative dependencies.
> 
> My specific use case at the moment is setting up a container
> *description* (not a container) with all the dependencies I need to do
> development on a package[1].
> 
> I could run apt-get build-dep inside the container and get the
> development environment installed, but then I lose the ability of being
> able to describe it in a terse way, and I can only do something along
> the lines of listing all installed packages in the container and their
> versions, which is too noisy for an average bug report.

Ok, so basing this description on .buildinfo would not seem to be
satisfactory then.

> The way I chopped dpkg-checkbuilddeps was a first approximation. Given
> that now we have `apt-get satisfy`, my next step would be to have my
> hacked version print a list of arguments for it, which can include
> "Conflicts:", but which can already be preprocessed to reduce some noise
> like packages required or not required from the target architecture.

Ah, so from this I gather that in essence what we need is a way to map
from build-dependencies into run-time dependencies, removing/filtering
them from anything that is not accepted in the latter. That makes
sense and does not seem to have the concerns of the previously filed
bug request, as that required performing decisions in a layer too low
where that required information was not available.

I could see gathering any build-dependency fields as restricted by
(-A/-B/-I), remapping them based on current arch/profile, then
outputting them as a pair of Depends/Conflicts fields (perhaps even
an Architecture field if there was arch-restrictions applied?). For
«apt satisfy» you might need to trim the «Depends: » part though.

Would that work for you? I think it would work for pbuilder.

> More generally, I'd like Debian to have, as a standard, something
> similar to `rpmspec --parse filename.spec | grep BuildRequires`
> because I see it reimplemented so many times (pbuilder, sbuild, and so
> on) that my instincts screams invoking the rule of three and
> refactor[2].

I think the above would solve your problem, and potentially could
substantially reduce the code in pbuilder-satisfydepends. For sbuild
and mk-build-deps (devscripts), which are already in perl, that would
only potentially help if this is included as part of a public module.
So I'll be going that way in case they want to (eventually) switch.

I've pondered for a while to add a dpkg-deb822 to parse and manipulate
control data, but perhaps something more specific to source packages
would be helpful too, where people want to query the list of binary
packages to build (dh_listpackages) and similar stuff, but that's for
later. :)

Thanks,
Guillem

Bug#1023922: dpkg-checkbuilddeps: please add an option to print the list of packages

[Enrico Zini]

On Sat, Nov 12, 2022 at 09:30:43PM +0100, Guillem Jover wrote:

> There was a bug filed requesting adding custom output format support
> (#214566) but it was closed “recently”. I think there might be some
> value in that, but not for the intended use the submitters seemed
> to want it.
> 
> I'd be interested to know how you'd want to use this new output/option
> as from the PoC script you provide it is not obvious to me, as it
> prints both build-depends and build-conflicts in an indistinguishable
> way, and it includes version constraints and alternative dependencies.

My specific use case at the moment is setting up a container
*description* (not a container) with all the dependencies I need to do
development on a package[1].

I could run apt-get build-dep inside the container and get the
development environment installed, but then I lose the ability of being
able to describe it in a terse way, and I can only do something along
the lines of listing all installed packages in the container and their
versions, which is too noisy for an average bug report.

The way I chopped dpkg-checkbuilddeps was a first approximation. Given
that now we have `apt-get satisfy`, my next step would be to have my
hacked version print a list of arguments for it, which can include
"Conflicts:", but which can already be preprocessed to reduce some noise
like packages required or not required from the target architecture.

More generally, I'd like Debian to have, as a standard, something
similar to `rpmspec --parse filename.spec | grep BuildRequires`
because I see it reimplemented so many times (pbuilder, sbuild, and so
on) that my instincts screams invoking the rule of three and
refactor[2].


Enrico

[1] some background context can be found in
https://github.com/ARPA-SIMC/moncic-ci/issues/74
although I'm expanding on that, see for example
https://github.com/ARPA-SIMC/moncic-ci/issues/77
[2] https://en.wikipedia.org/wiki/Rule_of_three_(computer_programming)
-- 
GPG key: 4096R/634F4BD1E7AD5568 2009-05-08 Enrico Zini 


signature.asc
Description: PGP signature

Processed (with 2 errors): Re: Bug#1023922: dpkg-checkbuilddeps: please add an option to print the list of packages

[Debian Bug Tracking System]

Processing control commands:

> user d...@packages.debian.org
Unknown command or malformed arguments to command.

> usertags -1 dpkg-checkbuilddeps
Unknown command or malformed arguments to command.

> tags -1 moreinfo
Bug #1023922 [dpkg-dev] dpkg-checkbuilddeps: please add an option to print the 
list of packages
Added tag(s) moreinfo.

-- 
1023922: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1023922
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

Bug#1023922: dpkg-checkbuilddeps: please add an option to print the list of packages

[Guillem Jover]

Control: user d...@packages.debian.org
Control: usertags -1 dpkg-checkbuilddeps
Control: tags -1 moreinfo

Hi!

On Sat, 2022-11-12 at 14:00:38 +0100, Enrico Zini wrote:
> Package: dpkg-dev
> Version: 1.21.9
> Severity: wishlist

> I often find myself[1] in need of a tool that, given a source package,
> prints a list of its build depends, given an architecture, a build
> profile, and so on.

There was a bug filed requesting adding custom output format support
(#214566) but it was closed “recently”. I think there might be some
value in that, but not for the intended use the submitters seemed
to want it.

I'd be interested to know how you'd want to use this new output/option
as from the PoC script you provide it is not obvious to me, as it
prints both build-depends and build-conflicts in an indistinguishable
way, and it includes version constraints and alternative dependencies.

> Would it be possible to add a way to print the unfiltered list?

Anything is possible, I guess my concern is whether this might create
confusion, or whether this is better solved already by some other
tool? Or if there is no workable alternative, finding the right
semantics to add this.

> Ideally, it can become a --print-depends/--print-conflicts option to
> dpkg-checkbuilddeps, instead of a separate tool. Unfortunately my
> perl-foo is too rusty to pretend I could propose a competent patch :/

I don't mind implementing it at all, but I'd like to understand what
this is needed for. :)

Thanks,
Guillem

Bug#1023922: dpkg-checkbuilddeps: please add an option to print the list of packages

[Enrico Zini]

Package: dpkg-dev
Version: 1.21.9
Severity: wishlist

Hello,

thank you for maintaining dpkg!

I often find myself[1] in need of a tool that, given a source package,
prints a list of its build depends, given an architecture, a build
profile, and so on.

dpkg-checkbuilddeps does internally generate it, and then only print the
list of packages not currently installed.

Would it be possible to add a way to print the unfiltered list?

I've made something that does it by chopping away the filtering bits
from dpkg-checkbuilddeps (see attachment).

Ideally, it can become a --print-depends/--print-conflicts option to
dpkg-checkbuilddeps, instead of a separate tool. Unfortunately my
perl-foo is too rusty to pretend I could propose a competent patch :/


Enrico

[1] and I'm apparently in good company, considering how many times this
is reimplemented in various places in Debian

-- Package-specific info:
This system uses merged-usr-via-aliased-dirs, going behind dpkg's
back, breaking its core assumptions. This can cause silent file
overwrites and disappearances, and its general tools misbehavior.
See .

-- System Information:
Debian Release: bookworm/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 6.0.0-2-amd64 (SMP w/4 CPU threads; PREEMPT)
Locale: LANG=en_IE.UTF-8, LC_CTYPE=en_IE.UTF-8 (charmap=UTF-8), 
LANGUAGE=en_IE:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages dpkg-dev depends on:
ii  binutils  2.39-8
ii  bzip2 1.0.8-5+b1
ii  libdpkg-perl  1.21.9
ii  make  4.3-4.1
ii  patch 2.7.6-7
ii  perl  5.36.0-4
ii  tar   1.34+dfsg-1
ii  xz-utils  5.2.7-0.1

Versions of packages dpkg-dev recommends:
ii  build-essential  12.9
ii  clang-14 [c-compiler]1:14.0.6-2
ii  fakeroot 1.29-1
ii  gcc [c-compiler] 4:12.2.0-1
ii  gcc-10 [c-compiler]  10.4.0-5
ii  gcc-12 [c-compiler]  12.2.0-9
ii  gnupg2.2.40-1
ii  gpgv 2.2.40-1
ii  libalgorithm-merge-perl  0.08-5

Versions of packages dpkg-dev suggests:
ii  debian-keyring  2022.08.11

-- no debconf information
#!/usr/bin/perl
#
# dpkg-checkbuilddeps
#
# Copyright © 2001 Joey Hess 
# Copyright © 2006-2009, 2011-2015 Guillem Jover 
# Copyright © 2007-2011 Raphael Hertzog 
# Copyright © 2022 Enrico Zini 
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see .

use strict;
use warnings;

use Getopt::Long qw(:config posix_default bundling_values no_ignorecase);

use Dpkg ();
use Dpkg::Gettext;
use Dpkg::ErrorHandling;
use Dpkg::Arch qw(get_host_arch);
use Dpkg::Vendor qw(run_vendor_hook);
use Dpkg::BuildProfiles qw(get_build_profiles set_build_profiles);
use Dpkg::Deps;
use Dpkg::Control::Info;

textdomain('dpkg-dev');

sub version()
{
printf g_("Debian %s version %s.\n"), $Dpkg::PROGNAME, $Dpkg::PROGVERSION;
}

sub usage {
printf g_(
'Usage: %s [...] []')
. "\n\n" . g_(
'Options:
  -A ignore Build-Depends-Arch and Build-Conflicts-Arch.
  -B ignore Build-Depends-Indep and Build-Conflicts-Indep.
  -I ignore built-in build dependencies and conflicts.
  -d build-deps  use given string as build dependencies instead of
 retrieving them from control file
  -c build-conf  use given string for build conflicts instead of
 retrieving them from control file
  -a archassume given host architecture
  -P profilesassume given build profiles (comma-separated list)
  --admindir=
 change the administrative directory.
  -?, --help show this help message.
  --version  show the version.')
. "\n\n" . g_(
' is the control file to process (default: debian/control).')
. "\n", $Dpkg::PROGNAME;
}

my $ignore_bd_arch = 0;
my $ignore_bd_indep = 0;
my $ignore_bd_builtin = 0;
my ($bd_value, $bc_value);
my $bp_value;
my $host_arch = get_host_arch();
my $admindir = $Dpkg::ADMINDIR;
my @options_spec = (
'help|?' => sub { usage(); exit(0); },
'version' => sub { version(); exit 0; },
'A' => \$ignore_bd_arch,
'B' => \$ignore_bd_indep,
'I' => \$ignore_bd_builtin,
'd=s' => \$bd_value,
'c=s' => \$bc_value,
'a=s' => \$host_arch,
'P=s' => \$bp_value,
'admindir=s' => \$admindir,