Bug#797858: devscripts: build-rdeps does not work for any but the most trivial dependency situatinos
* Johannes Schauer , 2015-09-11, 10:00: How about doing it the following way then: If dose-ceve is not installed, throw an error message and tell the user to either install the dose-extra package or use the option --simple-mode which uses the old behaviour. Sounds good to me. -- Jakub Wilk
Bug#797858: devscripts: build-rdeps does not work for any but the most trivial dependency situatinos
Hi Jakub,
thanks a lot for your review - much appreciated!
Quoting Jakub Wilk (2015-09-10 16:21:26)
> >If dose-ceve is installed (and new enough) then build-rdeps will use it
> >to calculate the reverse build dependencies of a binary packages. If
> >dose-ceve is not installed (or too old) then it will warn that it will
> >use the old unreliable behavior instead.
>
> I don't like the idea of that the command semantics completely changes
> when you install or uninstall a package. I'd rather see build-rdeps fail
> with no output if dose-ceve can't be used.
fair enough.
How about doing it the following way then:
If dose-ceve is not installed, throw an error message and tell the user to
either install the dose-extra package or use the option --simple-mode which
uses the old behaviour.
> >If dose-ceve is installed (and new enough) then the old behaviour can be
> >forced with the --without-ceve command line argument.
>
> I don't like the option name. Most people don't know what ceve stands
> for, and it's an implementation detail anyway. (What if edos^Wdose
> maintainers decide to rename it to dose-evec? ;>)
That is actually a not too far fetched possibility XD
This problem is solved with my above proposal. If you don't like that proposal,
I'm open to suggestions of how to name it.
> Your patch doesn't seem to handle contrib and non-free correctly. I get:
>
> >Reverse Build-depends in contrib:
> >-
> >
> >No reverse build-depends found for dh-autoreconf.
>
> even though of course there ARE packages in contrib build-depending on
> dh-autoreconf.
I will address this problem in my next patch - thanks for spotting it!
> >+ $packages_path =~
> >s/_source_Sources$/_binary-${opt_buildarch}_Packages/;
>
> The need for *_Packages should be documented in the REQUIREMENTS section
> of the manpage.
Indeed. Will add that in my next patch.
> On my machine, your implementation is ~100x slower than the current one.
> :\
If you know a piece of software that analyzes the interdependencies of 40k
binary packages and 20k source packages in a shorter time, I'm open for
suggestions!
cheers, josch
signature.asc
Description: signature
Bug#797858: devscripts: build-rdeps does not work for any but the most trivial dependency situatinos
Hi Johannes!
[Disclaimer: I'm not a devscripts maintainer.]
* Johannes Schauer , 2015-09-03, 07:08:
the only situation in which build-rdeps works reliably is if
- the source package specifies "Build-Depends: foo"
- foo is a real package
- all source packages using foo directly depend on it
- foo is Multi-Arch:no
To be fair, the documentation doesn't promise that build-rdeps considers
indirect dependencies, or cross-arch dependencies, or virtual packages.
So it's not that build-rdeps works unreliably; it's just it doesn't
meet your expectations. That said, your expectations are of course
quite reasonable.
If dose-ceve is installed (and new enough) then build-rdeps will use it
to calculate the reverse build dependencies of a binary packages. If
dose-ceve is not installed (or too old) then it will warn that it will
use the old unreliable behavior instead.
I don't like the idea of that the command semantics completely changes
when you install or uninstall a package. I'd rather see build-rdeps fail
with no output if dose-ceve can't be used.
If dose-ceve is installed (and new enough) then the old behaviour can
be forced with the --without-ceve command line argument.
I don't like the option name. Most people don't know what ceve stands
for, and it's an implementation detail anyway. (What if edos^Wdose
maintainers decide to rename it to dose-evec? ;>)
Your patch doesn't seem to handle contrib and non-free correctly. I get:
Reverse Build-depends in contrib:
-
No reverse build-depends found for dh-autoreconf.
even though of course there ARE packages in contrib build-depending on
dh-autoreconf.
+ $packages_path =~ s/_source_Sources$/_binary-${opt_buildarch}_Packages/;
The need for *_Packages should be documented in the REQUIREMENTS section
of the manpage.
On my machine, your implementation is ~100x slower than the current one.
:\
--
Jakub Wilk
Bug#797858: devscripts: build-rdeps does not work for any but the most trivial dependency situatinos
Package: devscripts
Version: 2.15.8
Severity: important
Tags: patch
Hi,
the only situation in which build-rdeps works reliably is if
- the source package specifies "Build-Depends: foo"
- foo is a real package
- all source packages using foo directly depend on it
- foo is Multi-Arch:no
the reality though is more complicated:
- the source package might have version restrictions in its
Build-Depends
- the source package might have architecture restrictions in its
Build-Depends
- the source package might have build profile restrictions in its
Build-Depends
- the source package might not directly but only indirectly
(transitively) depend on foo (in which case all complications of
binary package relationships are added to the mix)
- foo might be a virtual package
- foo might be part of a dependency alternative "A|foo"
- foo is not Multi-Arch:no and a crossbuild is required
To do a quantitative check of how often the simple "grep-like" mode of
operation that build-rdeps currently uses generates wrong results, lets
compare the capability of build-rdeps to find reverse dependencies with
the one of dose-ceve. Since dose3 is used by wanna-build we expect that
it correctly implements all dependency relationships used in Debian and
can thus serve as a ground truth.
For this endeavour we first generate the list of all binary packages
that are transitively build-depend upon by any source package in Debian:
grep-dctrl '' Sources -n -s Package | sort -u | sed 's/^/src:/' \
| xargs --max-chars=1 echo | tr ' ' ',' \
| xargs --max-args=1 -I{} dose-ceve -T deb -G pkg \
--deb-native-arch=amd64 -c {} \
debsrc://Sources deb://Packages \
| grep-dctrl -n -s Package '' | sort -u
According to the above, there are 17107 binary packages that are part of
a source package dependency closure.
Then, for each of these binary packages, we calculate the source
packages in its reverse dependencies in two ways. First, with a nearly
unmodified (only sorting was added to allow easy comparison) build-rdeps
script and then secondly with the build-rdeps script with dose-ceve
support (patch attached).
The result is that:
- 6390 (37%) binary packages have the same reverse build dependencies
- 10717 (63%) binary packages have differing reverse build dependencies
If we indeed assume that dose3 can serve as a ground truth, then we can
conclude that build-rdeps calculates wrong reverse build dependencies
for 63% of all binary packages that it makes sense to ask this question.
This is my justification for making this patch of important severity.
To fix this problem I propose the attached patch.
If dose-ceve is installed (and new enough) then build-rdeps will use it
to calculate the reverse build dependencies of a binary packages. If
dose-ceve is not installed (or too old) then it will warn that it will
use the old unreliable behavior instead.
If dose-ceve is installed (and new enough) then the old behaviour can be
forced with the --without-ceve command line argument.
Additionally, the command line options --host-arch and --build-arch can
be used to set host and build architecture for resolving cross build
reverse build dependencies. This is possible because dose-ceve
understands multiarch.
Thanks!
cheers, josch
>From a317afb505e981d63185d8489d9dd5c7a09778bf Mon Sep 17 00:00:00 2001
From: Johannes Schauer
Date: Sat, 29 Aug 2015 08:55:49 +0200
Subject: [PATCH] scripts/build-rdeps.pl: add dose-ceve support
---
scripts/build-rdeps.pl | 221 ++---
1 file changed, 192 insertions(+), 29 deletions(-)
diff --git a/scripts/build-rdeps.pl b/scripts/build-rdeps.pl
index dbbb98f..d5031c7 100755
--- a/scripts/build-rdeps.pl
+++ b/scripts/build-rdeps.pl
@@ -1,5 +1,6 @@
#!/usr/bin/perl
# Copyright (C) Patrick Schoenfeld
+# 2015 Johannes Schauer
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@@ -27,6 +28,17 @@ B I
B searches for all packages that build-depend on the specified package.
+The default behaviour is to just `grep` for the given dependency in the
+Build-Depends field of apt's Sources files.
+
+If the package dose-extra >= 4.0 is installed, then a more complete reverse
+build dependency computation is carried out. In particular, with that package
+installed, build-rdeps will find transitive reverse dependencies, respect
+architecture and build profile restrictions, take Provides relationships,
+Conflicts, Pre-Depends, Build-Depends-Arch and versioned dependencies into
+account and correctly resolve multiarch relationships for crossbuild reverse
+dependency resolution.
+
=head1 OPTIONS
=over 4
@@ -59,6 +71,28 @@ Restrict the search to only the specified origin (such as "Debian").
Print the value of the maintainer field for each package.
+=item B<--host-arch>
+
+Explicitly set th
