Your message dated Sat, 19 Mar 2016 18:49:05 +0000 with message-id <[email protected]> and subject line Bug#797858: fixed in devscripts 2.16.2 has caused the Debian Bug report #797858, regarding devscripts: build-rdeps does not work for any but the most trivial dependency situatinos to be marked as done.
This means that you claim that the problem has been dealt with. If this is not the case it is now your responsibility to reopen the Bug report if necessary, and/or fix the problem forthwith. (NB: If you are a system administrator and have no idea what this message is talking about, this may indicate a serious mail system misconfiguration somewhere. Please contact [email protected] immediately.) -- 797858: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=797858 Debian Bug Tracking System Contact [email protected] with problems
--- Begin Message ---Package: devscripts Version: 2.15.8 Severity: important Tags: patch Hi, the only situation in which build-rdeps works reliably is if - the source package specifies "Build-Depends: foo" - foo is a real package - all source packages using foo directly depend on it - foo is Multi-Arch:no the reality though is more complicated: - the source package might have version restrictions in its Build-Depends - the source package might have architecture restrictions in its Build-Depends - the source package might have build profile restrictions in its Build-Depends - the source package might not directly but only indirectly (transitively) depend on foo (in which case all complications of binary package relationships are added to the mix) - foo might be a virtual package - foo might be part of a dependency alternative "A|foo" - foo is not Multi-Arch:no and a crossbuild is required To do a quantitative check of how often the simple "grep-like" mode of operation that build-rdeps currently uses generates wrong results, lets compare the capability of build-rdeps to find reverse dependencies with the one of dose-ceve. Since dose3 is used by wanna-build we expect that it correctly implements all dependency relationships used in Debian and can thus serve as a ground truth. For this endeavour we first generate the list of all binary packages that are transitively build-depend upon by any source package in Debian: grep-dctrl '' Sources -n -s Package | sort -u | sed 's/^/src:/' \ | xargs --max-chars=10000 echo | tr ' ' ',' \ | xargs --max-args=1 -I{} dose-ceve -T deb -G pkg \ --deb-native-arch=amd64 -c {} \ debsrc://Sources deb://Packages \ | grep-dctrl -n -s Package '' | sort -u According to the above, there are 17107 binary packages that are part of a source package dependency closure. Then, for each of these binary packages, we calculate the source packages in its reverse dependencies in two ways. First, with a nearly unmodified (only sorting was added to allow easy comparison) build-rdeps script and then secondly with the build-rdeps script with dose-ceve support (patch attached). The result is that: - 6390 (37%) binary packages have the same reverse build dependencies - 10717 (63%) binary packages have differing reverse build dependencies If we indeed assume that dose3 can serve as a ground truth, then we can conclude that build-rdeps calculates wrong reverse build dependencies for 63% of all binary packages that it makes sense to ask this question. This is my justification for making this patch of important severity. To fix this problem I propose the attached patch. If dose-ceve is installed (and new enough) then build-rdeps will use it to calculate the reverse build dependencies of a binary packages. If dose-ceve is not installed (or too old) then it will warn that it will use the old unreliable behavior instead. If dose-ceve is installed (and new enough) then the old behaviour can be forced with the --without-ceve command line argument. Additionally, the command line options --host-arch and --build-arch can be used to set host and build architecture for resolving cross build reverse build dependencies. This is possible because dose-ceve understands multiarch. Thanks! cheers, josch>From a317afb505e981d63185d8489d9dd5c7a09778bf Mon Sep 17 00:00:00 2001 From: Johannes Schauer <[email protected]> Date: Sat, 29 Aug 2015 08:55:49 +0200 Subject: [PATCH] scripts/build-rdeps.pl: add dose-ceve support --- scripts/build-rdeps.pl | 221 ++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 192 insertions(+), 29 deletions(-) diff --git a/scripts/build-rdeps.pl b/scripts/build-rdeps.pl index dbbb98f..d5031c7 100755 --- a/scripts/build-rdeps.pl +++ b/scripts/build-rdeps.pl @@ -1,5 +1,6 @@ #!/usr/bin/perl # Copyright (C) Patrick Schoenfeld +# 2015 Johannes Schauer <[email protected]> # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -27,6 +28,17 @@ B<build-rdeps> I<package> B<build-rdeps> searches for all packages that build-depend on the specified package. +The default behaviour is to just `grep` for the given dependency in the +Build-Depends field of apt's Sources files. + +If the package dose-extra >= 4.0 is installed, then a more complete reverse +build dependency computation is carried out. In particular, with that package +installed, build-rdeps will find transitive reverse dependencies, respect +architecture and build profile restrictions, take Provides relationships, +Conflicts, Pre-Depends, Build-Depends-Arch and versioned dependencies into +account and correctly resolve multiarch relationships for crossbuild reverse +dependency resolution. + =head1 OPTIONS =over 4 @@ -59,6 +71,28 @@ Restrict the search to only the specified origin (such as "Debian"). Print the value of the maintainer field for each package. +=item B<--host-arch> + +Explicitly set the host architecture. The default is the value of +`dpkg-architecture -qDEB_HOST_ARCH`. This option only works if dose-extra >= +4.0 is installed. + +=item B<--without-ceve> + +Force the old behaviour without dose-ceve support even if dose-extra >= 4.0 is +installed. + +Notice, that the old behaviour only finds direct dependencies, ignores virtual +dependencies, does not find transitive dependencies and does not take version +relationships, architecture restrictions, build profiles or multiarch +relationships into account. + +=item B<--build-arch> + +Explicitly set the build architecture. The default is the value of +`dpkg-architecture -qDEB_BUILD_ARCH`. This option only works if dose-extra >= +4.0 is installed. + =item B<-d>, B<--debug> Run the debug mode @@ -100,6 +134,8 @@ my $version = '1.0'; my $dctrl = "grep-dctrl"; my $sources_path = "/var/lib/apt/lists/"; my $release_pattern = '(.*_dists_(sid|unstable))_(?:In)*Release$'; +my $use_ceve = 0; +my $ceve_compatible; my %seen_origins; my @source_files; my $opt_debug; @@ -110,6 +146,9 @@ my $opt_mainonly; my $opt_distribution; my $opt_origin = 'Debian'; my @opt_exclude_components; +my $opt_buildarch; +my $opt_hostarch; +my $opt_without_ceve; if (system('command -v grep-dctrl >/dev/null 2>&1')) { die "$progname: Fatal error. grep-dctrl is not available.\nPlease install the 'dctrl-tools' package.\n"; @@ -146,11 +185,32 @@ Options: (Default: Debian) --only-main Ignore contrib and non-free --exclude-component COMPONENT Ignore the specified component (can be given multiple times) + --host-arch Set the host architecture (requires dose-extra >= 4.0) + --build-arch Set the build architecture (requires dose-extra >= 4.0) + --without-ceve Do not use dose-ceve for reverse dependency resolution EOT version; } +sub test_ceve { + return $ceve_compatible if defined $ceve_compatible; + + # test if the debsrc input and output format is supported by the installed + # ceve version + system('dose-ceve -T debsrc debsrc:///dev/null > /dev/null 2>&1'); + if ($? == -1) { + print STDERR "DEBUG: dose-ceve cannot be executed: $!\n" if ($opt_debug); + $ceve_compatible = 0; + } elsif ($? == 0) { + $ceve_compatible = 1; + } else { + print STDERR "DEBUG: dose-ceve is too old\n" if ($opt_debug); + $ceve_compatible = 0; + } + return $ceve_compatible; +} + # Sub to test if a given section shall be included in the result sub test_for_valid_component { my $filebase = shift; @@ -169,6 +229,28 @@ sub test_for_valid_component { return -1; } + if ($use_ceve) { + die "build arch undefined" if ! defined $opt_buildarch; + die "host arch undefined" if ! defined $opt_hostarch; + my $packages_path = "$sources_path/$filebase"; + if ($filebase !~ /_source_Sources$/) { + print STDERR "Warning: Ignoring sources file $filebase because of unexpected postfix\n"; + return -1; + } + $packages_path =~ s/_source_Sources$/_binary-${opt_buildarch}_Packages/; + if (! -e $packages_path) { + print STDERR "Warning: Ignoring sources file $filebase because no corresponding buildarch Packages file for $opt_buildarch was found (required for ceve)\n"; + return -1; + } + if ($opt_buildarch ne $opt_hostarch) { + $packages_path =~ s/_source_Sources$/_binary-${opt_hostarch}_Packages/; + if (! -e $packages_path) { + print STDERR "Warning: Ignoring sources file $filebase because no corresponding buildarch Packages file for $opt_hostarch was found (required for ceve)\n"; + return -1; + } + } + } + print STDERR "DEBUG: Component ($filebase) may not be excluded.\n" if ($opt_debug); return 0; } @@ -216,46 +298,74 @@ sub addsources { sub findreversebuilddeps { my ($package, $source_file) = @_; - my %packages; - my $depending_package; my $count=0; - my $maintainer_info=''; - open(PACKAGES, '-|', $dctrl, '-r', '-F', 'Build-Depends,Build-Depends-Indep', "\\(^\\|, \\)$package", '-s', 'Package,Build-Depends,Build-Depends-Indep,Maintainer', $source_file); + if ($use_ceve) { + die "build arch undefined" if ! defined $opt_buildarch; + die "host arch undefined" if ! defined $opt_hostarch; - while(<PACKAGES>) { - chomp; - print STDERR "$_\n" if ($opt_debug); - if (/Package: (.*)$/) { - $depending_package = $1; - $packages{$depending_package}->{'Build-Depends'} = 0; + (my $buildarch_file = $source_file) =~ s/_source_Sources$/_binary-${opt_buildarch}_Packages/; + + my $ceve_cmd = "dose-ceve -T debsrc -r $package -G pkg" + . " --deb-native-arch=$opt_buildarch deb://$buildarch_file" + . " debsrc://$source_file"; + if ($opt_buildarch ne $opt_hostarch) { + $ceve_cmd .= " --deb-host-arch=$opt_hostarch"; + (my $hostarch_file = $source_file) =~ s/_source_Sources$/_binary-${opt_hostarch}_Packages/; + $ceve_cmd .= " deb://$hostarch_file"; } - elsif (/Maintainer: (.*)$/) { - if ($depending_package) { - $packages{$depending_package}->{'Maintainer'} = $1; + $ceve_cmd .= " | grep-dctrl -n -s Package '' | sort -u |"; + print STDERR "DEBUG: executing: $ceve_cmd" if ($opt_debug); + open(SOURCES, $ceve_cmd); + while(<SOURCES>) { + chomp; + print "$_"; + if ($opt_maintainer) { + my $maintainer = `apt-cache showsrc $_ | grep-dctrl -n -s Maintainer '' | sort -u`; + print " ($maintainer)"; } + print "\n"; + $count += 1; } - elsif (/Build-Depends: (.*)$/ or /Build-Depends-Indep: (.*)$/) { - if ($depending_package) { - print STDERR "$1\n" if ($opt_debug); - if ($1 =~ /^(.*\s)?\Q$package\E(?::[a-zA-Z0-9][a-zA-Z0-9-]*)?([\s,]|$)/) { - $packages{$depending_package}->{'Build-Depends'} = 1; + } else { + my %packages; + my $depending_package; + open(PACKAGES, '-|', $dctrl, '-r', '-F', 'Build-Depends,Build-Depends-Indep', "\\(^\\|, \\)$package", '-s', 'Package,Build-Depends,Build-Depends-Indep,Maintainer', $source_file); + + while(<PACKAGES>) { + chomp; + print STDERR "$_\n" if ($opt_debug); + if (/Package: (.*)$/) { + $depending_package = $1; + $packages{$depending_package}->{'Build-Depends'} = 0; + } + elsif (/Maintainer: (.*)$/) { + if ($depending_package) { + $packages{$depending_package}->{'Maintainer'} = $1; + } + } + elsif (/Build-Depends: (.*)$/ or /Build-Depends-Indep: (.*)$/) { + if ($depending_package) { + print STDERR "$1\n" if ($opt_debug); + if ($1 =~ /^(.*\s)?\Q$package\E(?::[a-zA-Z0-9][a-zA-Z0-9-]*)?([\s,]|$)/) { + $packages{$depending_package}->{'Build-Depends'} = 1; + } } } } - } - while($depending_package = each(%packages)) { - if ($packages{$depending_package}->{'Build-Depends'} != 1) { - print STDERR "Ignoring package $depending_package because its not really build depending on $package.\n" if ($opt_debug); - next; - } - print $depending_package; - if ($opt_maintainer) { - print " ($packages{$depending_package}->{'Maintainer'})"; + while($depending_package = each(%packages)) { + if ($packages{$depending_package}->{'Build-Depends'} != 1) { + print STDERR "Ignoring package $depending_package because its not really build depending on $package.\n" if ($opt_debug); + next; + } + print $depending_package; + if ($opt_maintainer) { + print " ($packages{$depending_package}->{'Maintainer'})"; + } + print "\n"; + $count+=1; } - print "\n"; - $count+=1; } if ($count == 0) { @@ -277,6 +387,12 @@ GetOptions( "only-main" => \$opt_mainonly, "exclude-component=s" => \@opt_exclude_components, "origin=s" => \$opt_origin, + "host-arch=s" => \$opt_hostarch, + "build-arch=s" => \$opt_buildarch, +# "profiles=s" => \$opt_profiles, # FIXME: add build profile support +# once dose-ceve has a +# --deb-profiles option + "without-ceve" => \$opt_without_ceve, "d|debug" => \$opt_debug, "h|help" => sub { usage; }, "v|version" => sub { version; } @@ -290,6 +406,53 @@ if (!$package) { print STDERR "DEBUG: Package => $package\n" if ($opt_debug); +if ($opt_hostarch) { + if ($opt_without_ceve) { + die "$progname: the --host-arch option cannot be used together with --without-ceve\n"; + } + if (test_ceve()) { + $use_ceve = 1; + } else { + die "$progname: the --host-arch option requires dose-extra >= 4.0 to be installed\n"; + } +} + +if ($opt_buildarch) { + if ($opt_without_ceve) { + die "$progname: the --build-arch option cannot be used together with --without-ceve\n"; + } + if (test_ceve()) { + $use_ceve = 1; + } else { + die "$progname: the --build-arch option requires dose-extra >= 4.0 to be installed\n"; + } +} + +# if ceve usage has not been activated yet, check if it can be activated +if (!$use_ceve and !$opt_without_ceve) { + if (test_ceve()) { + $use_ceve = 1; + } else { + print STDERR "WARNING: dose-extra >= 4.0 is not installed. Falling back to old unreliable behaviour.\n"; + } +} + +if ($use_ceve) { + # set hostarch and buildarch if they have not been set yet + if (!$opt_hostarch) { + $opt_hostarch = `dpkg-architecture --query DEB_HOST_ARCH`; + chomp $opt_hostarch; + } + if (!$opt_buildarch) { + $opt_buildarch = `dpkg-architecture --query DEB_BUILD_ARCH`; + chomp $opt_buildarch; + } + print STDERR "DEBUG: running with dose-ceve resolver\n" if ($opt_debug); + print STDERR "DEBUG: buildarch=$opt_buildarch hostarch=$opt_hostarch\n" if ($opt_debug); +} else { + print STDERR "DEBUG: running with old resolver\n" if ($opt_debug); +} + if ($opt_update) { print STDERR "DEBUG: Updating apt-cache before search\n" if ($opt_debug); my @cmd; -- 2.1.4
--- End Message ---
--- Begin Message ---Source: devscripts Source-Version: 2.16.2 We believe that the bug you reported is fixed in the latest version of devscripts, which is due to be installed in the Debian FTP archive. A summary of the changes between this version and the previous one is attached. Thank you for reporting the bug, which will now be closed. If you have further comments please address them to [email protected], and the maintainer will reopen the bug report if appropriate. Debian distribution maintenance software pp. James McCoy <[email protected]> (supplier of updated devscripts package) (This message was generated automatically at their request; if you believe that there is a problem with it please contact the archive administrators by mailing [email protected]) -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Format: 1.8 Date: Sat, 19 Mar 2016 14:34:39 -0400 Source: devscripts Binary: devscripts Architecture: source Version: 2.16.2 Distribution: unstable Urgency: medium Maintainer: Devscripts Devel Team <[email protected]> Changed-By: James McCoy <[email protected]> Closes: 497320 610048 675867 679763 770642 794538 797858 809318 810976 814049 814686 815645 816022 816231 816645 816723 816910 818275 818299 Description: devscripts - scripts to make the life of a Debian Package maintainer easier Changes: devscripts (2.16.2) unstable; urgency=medium . [ Osamu Aoki ] * uscan: + Print proper warnings for bad rules. Closes: #814049 + Fix glitches around rules such as "s<...> <...>g" and add their tests. + Reactivate --no-verbose mode. Closes: #815645 + Fix --destdir for version 4. Closes:#814686 + Always use uscan_warn and uscan_die. + Die if the action script fails. Closes: #810976 * test_uscan: + Fix non-{Debian,Ubuntu} cases. Closes: #816645 * debsnap: + Add --list option. Closes: #610048, #679763, #816910 * manpage-alert: + Show package name etc. Closes: #497320 * bts: + Fail if --sendmail is broken. Closes: #809318 . [ Dominique Dumont ] * licensecheck: + parse (c) owner with email like [email protected] . [ James McCoy ] * debsnap: + Only make the destination directory if there is something to download. (Closes: #816022) + Add --first & --last options to define a range of versions to download. Based on a patch by Ivo De Decker. (Closes: #675867) * uscan: + Fix typo in dirversionmangle help. (Closes: #816231) * grep-excuses: + Report excuses even when autoremoval info isn't reachable. (Closes: #816723) * Replace references to debbindiff with diffoscope. (Closes: #818299) * Declare compliance with policy 3.9.7, no changes required. . [ Johannes Schauer ] * build-rdeps: + Generate correct dependency for cross builds. Closes: #794538 + Improve dependency resolver using dose3. Closes: #797858 . [ Andreas Bombe ] * mk-build-deps: + Typo fix. Closes: #770642 . [ Adam D. Barratt ] * dcmd: + Correctly filter .buildinfo files. This also avoids listing .dsc files twice. (Closes: #818275) Checksums-Sha1: 920f48f1eb0944d3e3ffb9b2ef42645fe70ca1fd 2368 devscripts_2.16.2.dsc dfb03e9df365684987d833b490cfc2ec0c025f0c 686080 devscripts_2.16.2.tar.xz Checksums-Sha256: 2a347cc8b012237b483809677c68d17f50b32344229c1e59343448d0458b17d1 2368 devscripts_2.16.2.dsc 847507a2636c4e2eb0bea50b77be70b52e149496a603c78b0c0143e763649f62 686080 devscripts_2.16.2.tar.xz Files: 6958ee678512d435c259ed40a6ae4d44 2368 devel optional devscripts_2.16.2.dsc 13f69bef89b3b9c0b9f6198ce86dbf07 686080 devel optional devscripts_2.16.2.tar.xz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQJ8BAEBCgBmBQJW7Z29XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ5MUJGQkY0RDY5NTZCRDVERjdCNzJEMjNE RkU2OTFBRTMzMUJBM0RCAAoJEN/mka4zG6PbyT4P/20o6P2bkF73qVQfUuptzujY whd6XwM1K/KiVvE2FZCysYaxpCXGQuWizX/itjTAffg6vfXxgqsXeUEgMVuSYGuo Pbl52VElJSVeoZLG5bOPSYKKfH4e7HzGP6WKTeeKqYAxwkjurunGT/M9ZqAMCIBH g+3FI7B/hmFP1LB6zff0br01d0jWHrxq8dsHvGLbRfAVMYStUy0NNREEWjuq7KX6 E/dPiqVH2CpO6cEHO5+VsiVIwammR/RmLARHvMBDcg2m7Fh1o23X6GrEGLFCZRaF qlkALWNZeUNJKsCUy13CTH2cAc40wzr11BjZF+Ho6FpHOTv/oWnn9fvggpLp/fC/ h2GHZDv1EtmziQXJXubDXi1bPk6FbPFZEngpBN2NadVJXGYCiRSVIHnQirZDlg/J /B3EOso7svbu4bc0jwzYXL7seutPrhx7IzorBv3aB05xKUcHJvAa9CBUph/C7vc0 b/Hmbo44SoduJ1er3qM4P1QMdvfBbRfXq/om9zThCWHpyp8j5scGDc1nIkGNBvi/ 9CCs+2AUBF848WRLLjGrMfLRnEe5lFsf1W2k7f7Jj9TeAxXEnKEmp+PRKIGrCapc tPlSWnqCutJfxQgNUB7lCnGnCMxZAuOrp8Y0N0NHmM4U0jQNDIRRu2e/ApR5LZXZ S351RZUI/hkqWNP7bd0Q =SU2Q -----END PGP SIGNATURE-----
--- End Message ---
_______________________________________________ devscripts-devel mailing list [email protected] http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/devscripts-devel
