Package: devscripts
Version: 2.15.8
Severity: important
Tags: patch
Hi,
the only situation in which build-rdeps works reliably is if
- the source package specifies "Build-Depends: foo"
- foo is a real package
- all source packages using foo directly depend on it
- foo is Multi-Arch:no
the reality though is more complicated:
- the source package might have version restrictions in its
Build-Depends
- the source package might have architecture restrictions in its
Build-Depends
- the source package might have build profile restrictions in its
Build-Depends
- the source package might not directly but only indirectly
(transitively) depend on foo (in which case all complications of
binary package relationships are added to the mix)
- foo might be a virtual package
- foo might be part of a dependency alternative "A|foo"
- foo is not Multi-Arch:no and a crossbuild is required
To do a quantitative check of how often the simple "grep-like" mode of
operation that build-rdeps currently uses generates wrong results, lets
compare the capability of build-rdeps to find reverse dependencies with
the one of dose-ceve. Since dose3 is used by wanna-build we expect that
it correctly implements all dependency relationships used in Debian and
can thus serve as a ground truth.
For this endeavour we first generate the list of all binary packages
that are transitively build-depend upon by any source package in Debian:
grep-dctrl '' Sources -n -s Package | sort -u | sed 's/^/src:/' \
| xargs --max-chars=10000 echo | tr ' ' ',' \
| xargs --max-args=1 -I{} dose-ceve -T deb -G pkg \
--deb-native-arch=amd64 -c {} \
debsrc://Sources deb://Packages \
| grep-dctrl -n -s Package '' | sort -u
According to the above, there are 17107 binary packages that are part of
a source package dependency closure.
Then, for each of these binary packages, we calculate the source
packages in its reverse dependencies in two ways. First, with a nearly
unmodified (only sorting was added to allow easy comparison) build-rdeps
script and then secondly with the build-rdeps script with dose-ceve
support (patch attached).
The result is that:
- 6390 (37%) binary packages have the same reverse build dependencies
- 10717 (63%) binary packages have differing reverse build dependencies
If we indeed assume that dose3 can serve as a ground truth, then we can
conclude that build-rdeps calculates wrong reverse build dependencies
for 63% of all binary packages that it makes sense to ask this question.
This is my justification for making this patch of important severity.
To fix this problem I propose the attached patch.
If dose-ceve is installed (and new enough) then build-rdeps will use it
to calculate the reverse build dependencies of a binary packages. If
dose-ceve is not installed (or too old) then it will warn that it will
use the old unreliable behavior instead.
If dose-ceve is installed (and new enough) then the old behaviour can be
forced with the --without-ceve command line argument.
Additionally, the command line options --host-arch and --build-arch can
be used to set host and build architecture for resolving cross build
reverse build dependencies. This is possible because dose-ceve
understands multiarch.
Thanks!
cheers, josch
>From a317afb505e981d63185d8489d9dd5c7a09778bf Mon Sep 17 00:00:00 2001
From: Johannes Schauer <[email protected]>
Date: Sat, 29 Aug 2015 08:55:49 +0200
Subject: [PATCH] scripts/build-rdeps.pl: add dose-ceve support
---
scripts/build-rdeps.pl | 221 ++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 192 insertions(+), 29 deletions(-)
diff --git a/scripts/build-rdeps.pl b/scripts/build-rdeps.pl
index dbbb98f..d5031c7 100755
--- a/scripts/build-rdeps.pl
+++ b/scripts/build-rdeps.pl
@@ -1,5 +1,6 @@
#!/usr/bin/perl
# Copyright (C) Patrick Schoenfeld
+# 2015 Johannes Schauer <[email protected]>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@@ -27,6 +28,17 @@ B<build-rdeps> I<package>
B<build-rdeps> searches for all packages that build-depend on the specified package.
+The default behaviour is to just `grep` for the given dependency in the
+Build-Depends field of apt's Sources files.
+
+If the package dose-extra >= 4.0 is installed, then a more complete reverse
+build dependency computation is carried out. In particular, with that package
+installed, build-rdeps will find transitive reverse dependencies, respect
+architecture and build profile restrictions, take Provides relationships,
+Conflicts, Pre-Depends, Build-Depends-Arch and versioned dependencies into
+account and correctly resolve multiarch relationships for crossbuild reverse
+dependency resolution.
+
=head1 OPTIONS
=over 4
@@ -59,6 +71,28 @@ Restrict the search to only the specified origin (such as "Debian").
Print the value of the maintainer field for each package.
+=item B<--host-arch>
+
+Explicitly set the host architecture. The default is the value of
+`dpkg-architecture -qDEB_HOST_ARCH`. This option only works if dose-extra >=
+4.0 is installed.
+
+=item B<--without-ceve>
+
+Force the old behaviour without dose-ceve support even if dose-extra >= 4.0 is
+installed.
+
+Notice, that the old behaviour only finds direct dependencies, ignores virtual
+dependencies, does not find transitive dependencies and does not take version
+relationships, architecture restrictions, build profiles or multiarch
+relationships into account.
+
+=item B<--build-arch>
+
+Explicitly set the build architecture. The default is the value of
+`dpkg-architecture -qDEB_BUILD_ARCH`. This option only works if dose-extra >=
+4.0 is installed.
+
=item B<-d>, B<--debug>
Run the debug mode
@@ -100,6 +134,8 @@ my $version = '1.0';
my $dctrl = "grep-dctrl";
my $sources_path = "/var/lib/apt/lists/";
my $release_pattern = '(.*_dists_(sid|unstable))_(?:In)*Release$';
+my $use_ceve = 0;
+my $ceve_compatible;
my %seen_origins;
my @source_files;
my $opt_debug;
@@ -110,6 +146,9 @@ my $opt_mainonly;
my $opt_distribution;
my $opt_origin = 'Debian';
my @opt_exclude_components;
+my $opt_buildarch;
+my $opt_hostarch;
+my $opt_without_ceve;
if (system('command -v grep-dctrl >/dev/null 2>&1')) {
die "$progname: Fatal error. grep-dctrl is not available.\nPlease install the 'dctrl-tools' package.\n";
@@ -146,11 +185,32 @@ Options:
(Default: Debian)
--only-main Ignore contrib and non-free
--exclude-component COMPONENT Ignore the specified component (can be given multiple times)
+ --host-arch Set the host architecture (requires dose-extra >= 4.0)
+ --build-arch Set the build architecture (requires dose-extra >= 4.0)
+ --without-ceve Do not use dose-ceve for reverse dependency resolution
EOT
version;
}
+sub test_ceve {
+ return $ceve_compatible if defined $ceve_compatible;
+
+ # test if the debsrc input and output format is supported by the installed
+ # ceve version
+ system('dose-ceve -T debsrc debsrc:///dev/null > /dev/null 2>&1');
+ if ($? == -1) {
+ print STDERR "DEBUG: dose-ceve cannot be executed: $!\n" if ($opt_debug);
+ $ceve_compatible = 0;
+ } elsif ($? == 0) {
+ $ceve_compatible = 1;
+ } else {
+ print STDERR "DEBUG: dose-ceve is too old\n" if ($opt_debug);
+ $ceve_compatible = 0;
+ }
+ return $ceve_compatible;
+}
+
# Sub to test if a given section shall be included in the result
sub test_for_valid_component {
my $filebase = shift;
@@ -169,6 +229,28 @@ sub test_for_valid_component {
return -1;
}
+ if ($use_ceve) {
+ die "build arch undefined" if ! defined $opt_buildarch;
+ die "host arch undefined" if ! defined $opt_hostarch;
+ my $packages_path = "$sources_path/$filebase";
+ if ($filebase !~ /_source_Sources$/) {
+ print STDERR "Warning: Ignoring sources file $filebase because of unexpected postfix\n";
+ return -1;
+ }
+ $packages_path =~ s/_source_Sources$/_binary-${opt_buildarch}_Packages/;
+ if (! -e $packages_path) {
+ print STDERR "Warning: Ignoring sources file $filebase because no corresponding buildarch Packages file for $opt_buildarch was found (required for ceve)\n";
+ return -1;
+ }
+ if ($opt_buildarch ne $opt_hostarch) {
+ $packages_path =~ s/_source_Sources$/_binary-${opt_hostarch}_Packages/;
+ if (! -e $packages_path) {
+ print STDERR "Warning: Ignoring sources file $filebase because no corresponding buildarch Packages file for $opt_hostarch was found (required for ceve)\n";
+ return -1;
+ }
+ }
+ }
+
print STDERR "DEBUG: Component ($filebase) may not be excluded.\n" if ($opt_debug);
return 0;
}
@@ -216,46 +298,74 @@ sub addsources {
sub findreversebuilddeps {
my ($package, $source_file) = @_;
- my %packages;
- my $depending_package;
my $count=0;
- my $maintainer_info='';
- open(PACKAGES, '-|', $dctrl, '-r', '-F', 'Build-Depends,Build-Depends-Indep', "\\(^\\|, \\)$package", '-s', 'Package,Build-Depends,Build-Depends-Indep,Maintainer', $source_file);
+ if ($use_ceve) {
+ die "build arch undefined" if ! defined $opt_buildarch;
+ die "host arch undefined" if ! defined $opt_hostarch;
- while(<PACKAGES>) {
- chomp;
- print STDERR "$_\n" if ($opt_debug);
- if (/Package: (.*)$/) {
- $depending_package = $1;
- $packages{$depending_package}->{'Build-Depends'} = 0;
+ (my $buildarch_file = $source_file) =~ s/_source_Sources$/_binary-${opt_buildarch}_Packages/;
+
+ my $ceve_cmd = "dose-ceve -T debsrc -r $package -G pkg"
+ . " --deb-native-arch=$opt_buildarch deb://$buildarch_file"
+ . " debsrc://$source_file";
+ if ($opt_buildarch ne $opt_hostarch) {
+ $ceve_cmd .= " --deb-host-arch=$opt_hostarch";
+ (my $hostarch_file = $source_file) =~ s/_source_Sources$/_binary-${opt_hostarch}_Packages/;
+ $ceve_cmd .= " deb://$hostarch_file";
}
- elsif (/Maintainer: (.*)$/) {
- if ($depending_package) {
- $packages{$depending_package}->{'Maintainer'} = $1;
+ $ceve_cmd .= " | grep-dctrl -n -s Package '' | sort -u |";
+ print STDERR "DEBUG: executing: $ceve_cmd" if ($opt_debug);
+ open(SOURCES, $ceve_cmd);
+ while(<SOURCES>) {
+ chomp;
+ print "$_";
+ if ($opt_maintainer) {
+ my $maintainer = `apt-cache showsrc $_ | grep-dctrl -n -s Maintainer '' | sort -u`;
+ print " ($maintainer)";
}
+ print "\n";
+ $count += 1;
}
- elsif (/Build-Depends: (.*)$/ or /Build-Depends-Indep: (.*)$/) {
- if ($depending_package) {
- print STDERR "$1\n" if ($opt_debug);
- if ($1 =~ /^(.*\s)?\Q$package\E(?::[a-zA-Z0-9][a-zA-Z0-9-]*)?([\s,]|$)/) {
- $packages{$depending_package}->{'Build-Depends'} = 1;
+ } else {
+ my %packages;
+ my $depending_package;
+ open(PACKAGES, '-|', $dctrl, '-r', '-F', 'Build-Depends,Build-Depends-Indep', "\\(^\\|, \\)$package", '-s', 'Package,Build-Depends,Build-Depends-Indep,Maintainer', $source_file);
+
+ while(<PACKAGES>) {
+ chomp;
+ print STDERR "$_\n" if ($opt_debug);
+ if (/Package: (.*)$/) {
+ $depending_package = $1;
+ $packages{$depending_package}->{'Build-Depends'} = 0;
+ }
+ elsif (/Maintainer: (.*)$/) {
+ if ($depending_package) {
+ $packages{$depending_package}->{'Maintainer'} = $1;
+ }
+ }
+ elsif (/Build-Depends: (.*)$/ or /Build-Depends-Indep: (.*)$/) {
+ if ($depending_package) {
+ print STDERR "$1\n" if ($opt_debug);
+ if ($1 =~ /^(.*\s)?\Q$package\E(?::[a-zA-Z0-9][a-zA-Z0-9-]*)?([\s,]|$)/) {
+ $packages{$depending_package}->{'Build-Depends'} = 1;
+ }
}
}
}
- }
- while($depending_package = each(%packages)) {
- if ($packages{$depending_package}->{'Build-Depends'} != 1) {
- print STDERR "Ignoring package $depending_package because its not really build depending on $package.\n" if ($opt_debug);
- next;
- }
- print $depending_package;
- if ($opt_maintainer) {
- print " ($packages{$depending_package}->{'Maintainer'})";
+ while($depending_package = each(%packages)) {
+ if ($packages{$depending_package}->{'Build-Depends'} != 1) {
+ print STDERR "Ignoring package $depending_package because its not really build depending on $package.\n" if ($opt_debug);
+ next;
+ }
+ print $depending_package;
+ if ($opt_maintainer) {
+ print " ($packages{$depending_package}->{'Maintainer'})";
+ }
+ print "\n";
+ $count+=1;
}
- print "\n";
- $count+=1;
}
if ($count == 0) {
@@ -277,6 +387,12 @@ GetOptions(
"only-main" => \$opt_mainonly,
"exclude-component=s" => \@opt_exclude_components,
"origin=s" => \$opt_origin,
+ "host-arch=s" => \$opt_hostarch,
+ "build-arch=s" => \$opt_buildarch,
+# "profiles=s" => \$opt_profiles, # FIXME: add build profile support
+# once dose-ceve has a
+# --deb-profiles option
+ "without-ceve" => \$opt_without_ceve,
"d|debug" => \$opt_debug,
"h|help" => sub { usage; },
"v|version" => sub { version; }
@@ -290,6 +406,53 @@ if (!$package) {
print STDERR "DEBUG: Package => $package\n" if ($opt_debug);
+if ($opt_hostarch) {
+ if ($opt_without_ceve) {
+ die "$progname: the --host-arch option cannot be used together with --without-ceve\n";
+ }
+ if (test_ceve()) {
+ $use_ceve = 1;
+ } else {
+ die "$progname: the --host-arch option requires dose-extra >= 4.0 to be installed\n";
+ }
+}
+
+if ($opt_buildarch) {
+ if ($opt_without_ceve) {
+ die "$progname: the --build-arch option cannot be used together with --without-ceve\n";
+ }
+ if (test_ceve()) {
+ $use_ceve = 1;
+ } else {
+ die "$progname: the --build-arch option requires dose-extra >= 4.0 to be installed\n";
+ }
+}
+
+# if ceve usage has not been activated yet, check if it can be activated
+if (!$use_ceve and !$opt_without_ceve) {
+ if (test_ceve()) {
+ $use_ceve = 1;
+ } else {
+ print STDERR "WARNING: dose-extra >= 4.0 is not installed. Falling back to old unreliable behaviour.\n";
+ }
+}
+
+if ($use_ceve) {
+ # set hostarch and buildarch if they have not been set yet
+ if (!$opt_hostarch) {
+ $opt_hostarch = `dpkg-architecture --query DEB_HOST_ARCH`;
+ chomp $opt_hostarch;
+ }
+ if (!$opt_buildarch) {
+ $opt_buildarch = `dpkg-architecture --query DEB_BUILD_ARCH`;
+ chomp $opt_buildarch;
+ }
+ print STDERR "DEBUG: running with dose-ceve resolver\n" if ($opt_debug);
+ print STDERR "DEBUG: buildarch=$opt_buildarch hostarch=$opt_hostarch\n" if ($opt_debug);
+} else {
+ print STDERR "DEBUG: running with old resolver\n" if ($opt_debug);
+}
+
if ($opt_update) {
print STDERR "DEBUG: Updating apt-cache before search\n" if ($opt_debug);
my @cmd;
--
2.1.4
_______________________________________________
devscripts-devel mailing list
[email protected]
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/devscripts-devel
