Bug#452511: About dpkg-shlibdeps checks
[ CCing #452511 as I provide an explanation of why we shouldn't change back to --ignore-missing-info by default without careful consideration ] On Fri, 23 Nov 2007, Pierre Habouzit wrote: Damn I wanted to answer to that, and forgot: I don't think anyone wants a revert. I'd expect you to make lower the dpkg-shlibdeps expectations for a while, so that we can take our time to catalog every issues that it raise. If only things were as simple as that. Go read #452339 and understand that the new dpkg-shlibdeps fully respects any local ld.so.conf configuration, and any broken RPATH information in binaries. This means that because of unexpected local configuration, or because of bad RPATH, dpkg-shlibdeps might find a copy of the library that is not packaged (even in /usr/local/ [1]). If later on I don't find the dependency information because the library is not associated to any package, and if I silently ignore that information as you suggest it to me, we will have produced packages with missing dependencies. I'd rather be more strict and relax the rules as we identify cases where I have been too strict, than let people upload broken .debs during weeks and later discover that we have to scan the full archive to rebuild a bunch of them. Yes, the situation is not always as simple as it looks like. [1] BTW, should I just skip any directory matching ^/usr/local while trying to find the library used by a binary? Cheers, -- Raphaël Hertzog Premier livre français sur Debian GNU/Linux : http://www.ouaza.com/livre/admin-debian/
Bug#452577: dpkg-dev: dpkg-shlibdeps no longer optimizes dpkg --search usage
Raphael Hertzog [EMAIL PROTECTED] writes: Somehow I was thinking I had implemented a cache but I must have mixed up with something else. Thanks for the check! No problem. I did my best to make it understandable compared to the old code. :) Well done. Applied. But it doesn't give a huge performance boost. On a run on kdemultimedia, it saves 8 seconds out of 3m12. I think we could save much Thanks! As for the performance gain, it really depends on how many packages you have installed, and I suspect also on how much RAM you have. At any rate, it shouldn't hurt as long as it doesn't introduce any new bugs, which I'm pretty sure I've avoided doing this time around. ;-) But optimizing this part probably needs somewhat more care and is a bit less straightforward. I also wonder how much memory it would cost on big packages. Right, I was pondering that as well, but decided to stick with tthe straightforward optimization for now. But if you have some time to spend on it, I'll happily review a patch. Good to know, but I don't anticipate having enough time anytime soon. -- Aaron M. Ucko, KB1CJC (amu at alum.mit.edu, ucko at debian.org) http://www.mit.edu/~amu/ | http://stuff.mit.edu/cgi/finger/[EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#452577: dpkg-dev: dpkg-shlibdeps no longer optimizes dpkg --search usage
On Fri, 23 Nov 2007, Aaron M. Ucko wrote: dpkg-shlibdeps's latest incarnation (as of 1.14.8 and its experimental predecessors) introduces a performance regression: it runs dpkg --search once per executable or library being examined, rather than caching its results any fashion. As each call requires scanning every package's contents AFAICT, the resulting procedure can take a LONG time on systems with many packages installed. (It would be great if dpkg-query could itself run faster, but that's a separate issue.) Somehow I was thinking I had implemented a cache but I must have mixed up with something else. Thanks for the check! (At any rate, the rewrite at least resulted in much clearer and more readily patched logic.) I did my best to make it understandable compared to the old code. :) Could you please review and apply the attached patch (against 1.14.10) when you get a chance? Applied. But it doesn't give a huge performance boost. On a run on kdemultimedia, it saves 8 seconds out of 3m12. I think we could save much more by caching the Dpkg::Shlibs::Objdump::Object objects created by the line: my $id = $dumplibs_wo_symfile-parse($lib); But optimizing this part probably needs somewhat more care and is a bit less straightforward. I also wonder how much memory it would cost on big packages. But if you have some time to spend on it, I'll happily review a patch. Cheers, -- Raphaël Hertzog Premier livre français sur Debian GNU/Linux : http://www.ouaza.com/livre/admin-debian/
Bug#452577: dpkg-dev: dpkg-shlibdeps no longer optimizes dpkg --search usage
Package: dpkg-dev
Version: 1.14.8
Severity: normal
Tags: patch
dpkg-shlibdeps's latest incarnation (as of 1.14.8 and its experimental
predecessors) introduces a performance regression: it runs dpkg
--search once per executable or library being examined, rather than
caching its results any fashion. As each call requires scanning every
package's contents AFAICT, the resulting procedure can take a LONG
time on systems with many packages installed. (It would be great if
dpkg-query could itself run faster, but that's a separate issue.)
I previously reported the same high-level problem as #421290, but I'm
opening a new bug because the underlying code base is so different.
As before, I have put together a patch that I believe DTRT. This
time, I've even tested it against complicated cases such as libkcal2b,
to avoid a repeat of #425641, for which I do sincerely apologize. (At
any rate, the rewrite at least resulted in much clearer and more
readily patched logic.)
Could you please review and apply the attached patch (against 1.14.10)
when you get a chance?
Thanks!
-- System Information:
Debian Release: lenny/sid
APT prefers unstable
APT policy: (500, 'unstable'), (500, 'testing'), (500, 'stable')
Architecture: i386 (i686)
Kernel: Linux 2.6.22
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
--- dpkg-shlibdeps.1.14.10 2007-11-23 13:47:46.0 -0500
+++ dpkg-shlibdeps.optimized2007-11-23 13:53:20.0 -0500
@@ -483,9 +483,22 @@
return undef;
}
+my %cached_pkgmatch = ();
+
sub find_packages {
-my @files = (@_);
+my @files;
my $pkgmatch = {};
+
+foreach (@_) {
+ if (exists $cached_pkgmatch{$_}) {
+ $pkgmatch-{$_} = $cached_pkgmatch{$_};
+ } else {
+ push @files, $_;
+ $cached_pkgmatch{$_} = []; # placeholder to cache misses too.
+ }
+}
+return $pkgmatch unless @files;
+
my $pid = open(DPKG, -|);
syserr(_g(cannot fork for dpkg --search)) unless defined($pid);
if (!$pid) {
@@ -503,7 +516,7 @@
print(STDERR $_\n)
|| syserr(_g(write diversion info to stderr));
} elsif (m/^([^:]+): (\S+)$/) {
- $pkgmatch-{$2} = [ split(/, /, $1) ];
+ $cached_pkgmatch{$2} = $pkgmatch-{$2} = [ split(/, /, $1) ];
} else {
warning(_g(unknown output from dpkg --search: '%s'), $_);
}
Bug#120152: Why isn't the merge patch being merged?
Raphael, since you say that Jarno's current patch applies fine to the
current version of dpkg, would it be possible for you to merge it?
While waiting for somebody to implement a three way merge, I'd much
prefer a two way merge (Jarno's patch) to nothing (which is what we
have now).
Please merge the patch?
Regards //Johan
2007/10/24, Raphael Hertzog [EMAIL PROTECTED]:
On Wed, 24 Oct 2007, Johan Walles wrote:
We have many patches lying in the BTS but many of them are old and
probably don't apply anymore and need to be refreshed and discussed.
This one seems actively maintained; for a couple of years now the
author has been packaging his own patched dpkg (currently at 1.14.5):
http://elonen.iki.fi/code/unofficial-debs/dpkg-merge/
I saw this somewhat later on, after I finished my reply. Except for a file
rename (man/C/dpkg.1 - man/dpkg.1), the patch applied fine to the current
version.
On the other hand, it might be that a 2-way merge is not what we're
looking for. It's certainly better than nothing but OTOH it would be 2
times better if we could have 3-way merge so that we can even add a new
option --force-confmerge that tries automatically a 3-way merge and that
falls back to the usual behaviour --force-conf{new,old,def} if the merge
process failed.
But it requires more extensive changes as we need to keep a copy of the
conffiles as shippped by the previous version of the package.
Jarno, would you be interested to work on that?
Cheers,
--
Raphaël Hertzog
Premier livre français sur Debian GNU/Linux :
http://www.ouaza.com/livre/admin-debian/
Bug#452511: --ignore-missing-info should be the default for now
Package: dpkg-dev
Version: 1.14.10
Severity: serious
Tags: patch
Hi,
The recent change to check missing shlibdeps files and to error out if
not makes some packages insta-buggy where they otherwise wouldn't be. A
good example is in libm; there is no shlibdeps file for libm, which
makes dpkg-shlibdeps bail out at the moment. However, it's pretty hard
to link to libm while not linking to libc, which is in the same package,
and which *does* have a slibdeps file.
You really shouldn't do this. The following patch makes
--ignore-missing-info the default:
diff -ruN dpkg-1.14.10.orig/man/dpkg-shlibdeps.1
dpkg-1.14.10/man/dpkg-shlibdeps.1
--- dpkg-1.14.10.orig/man/dpkg-shlibdeps.1 2007-11-19 07:28:45.0
+0100
+++ dpkg-1.14.10/man/dpkg-shlibdeps.1 2007-11-23 12:26:44.0 +0100
@@ -173,11 +173,13 @@
package. This option can be used multiple times to exclude several
packages.
.TP
+.BI \-\-no\-ignore\-missing\-info
.BI \-\-ignore\-missing\-info
-Do not fail if dependency information can't be found for a shared library.
+Do (do not) fail if dependency information can't be found for a shared library.
Usage of this option is discouraged, all libraries should provide
dependency information (either with shlibs files, or with symbols files)
-even if they are not yet used by other packages.
+even if they are not yet used by other packages. The default is to ignore
+missing info.
.TP
.BI \-\-admindir= dir
Change the location of the \fBdpkg\fR database. The default location is
diff -ruN dpkg-1.14.10.orig/scripts/dpkg-shlibdeps.pl
dpkg-1.14.10/scripts/dpkg-shlibdeps.pl
--- dpkg-1.14.10.orig/scripts/dpkg-shlibdeps.pl 2007-11-23 03:42:51.0
+0100
+++ dpkg-1.14.10/scripts/dpkg-shlibdeps.pl 2007-11-23 12:25:30.0
+0100
@@ -32,7 +32,7 @@
my $dependencyfield= 'Depends';
my $varlistfile= 'debian/substvars';
my $varnameprefix= 'shlibs';
-my $ignore_missing_info= 0;
+my $ignore_missing_info= 1;
my $debug= 0;
my @exclude = ();
@@ -68,6 +68,8 @@
$exec{$1} = $dependencyfield;
} elsif (m/^--ignore-missing-info$/) {
$ignore_missing_info = 1;
+} elsif (m/^--no-ignore-missing-info$/) {
+ $ignore_missing_info = 0;
} elsif (m/^-t(.*)$/) {
$packagetype = $1;
} elsif (m/^-v$/) {
-- System Information:
Debian Release: lenny/sid
APT prefers unstable
APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: powerpc (ppc)
Kernel: Linux 2.6.22-3-powerpc
Locale: LANG=nl_BE.UTF-8, LC_CTYPE=nl_BE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Versions of packages dpkg-dev depends on:
ii binutils2.18.1~cvs20071027-1 The GNU assembler, linker and bina
ii cpio2.9-6GNU cpio -- a program to manage ar
ii dpkg1.14.9 package maintenance system for Deb
ii make3.81-3 The GNU version of the make util
ii patch 2.5.9-4 Apply a diff file to an original
ii perl [perl5]5.8.8-12 Larry Wall's Practical Extraction
ii perl-modules5.8.8-12 Core Perl modules
Versions of packages dpkg-dev recommends:
ii bzip2 1.0.3-7high-quality block-sorting file co
ii gcc [c-compiler] 4:4.2.1-6 The GNU C compiler
ii gcc-3.3 [c-compiler] 1:3.3.6-15 The GNU C compiler
ii gcc-3.4 [c-compiler] 3.4.6-6The GNU C compiler
ii gcc-4.1 [c-compiler] 4.1.2-17 The GNU C compiler
ii gcc-4.2 [c-compiler] 4.2.2-3The GNU C compiler
-- no debconf information
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#452577: marked as done (dpkg-dev: dpkg-shlibdeps no longer optimizes dpkg --search usage)
Your message dated Sat, 24 Nov 2007 05:47:02 +
with message-id [EMAIL PROTECTED]
and subject line Bug#452577: fixed in dpkg 1.14.11
has caused the attached Bug report to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what I am
talking about this indicates a serious mail system misconfiguration
somewhere. Please contact me immediately.)
Debian bug tracking system administrator
(administrator, Debian Bugs database)
---BeginMessage---
Package: dpkg-dev
Version: 1.14.8
Severity: normal
Tags: patch
dpkg-shlibdeps's latest incarnation (as of 1.14.8 and its experimental
predecessors) introduces a performance regression: it runs dpkg
--search once per executable or library being examined, rather than
caching its results any fashion. As each call requires scanning every
package's contents AFAICT, the resulting procedure can take a LONG
time on systems with many packages installed. (It would be great if
dpkg-query could itself run faster, but that's a separate issue.)
I previously reported the same high-level problem as #421290, but I'm
opening a new bug because the underlying code base is so different.
As before, I have put together a patch that I believe DTRT. This
time, I've even tested it against complicated cases such as libkcal2b,
to avoid a repeat of #425641, for which I do sincerely apologize. (At
any rate, the rewrite at least resulted in much clearer and more
readily patched logic.)
Could you please review and apply the attached patch (against 1.14.10)
when you get a chance?
Thanks!
-- System Information:
Debian Release: lenny/sid
APT prefers unstable
APT policy: (500, 'unstable'), (500, 'testing'), (500, 'stable')
Architecture: i386 (i686)
Kernel: Linux 2.6.22
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
--- dpkg-shlibdeps.1.14.10 2007-11-23 13:47:46.0 -0500
+++ dpkg-shlibdeps.optimized2007-11-23 13:53:20.0 -0500
@@ -483,9 +483,22 @@
return undef;
}
+my %cached_pkgmatch = ();
+
sub find_packages {
-my @files = (@_);
+my @files;
my $pkgmatch = {};
+
+foreach (@_) {
+ if (exists $cached_pkgmatch{$_}) {
+ $pkgmatch-{$_} = $cached_pkgmatch{$_};
+ } else {
+ push @files, $_;
+ $cached_pkgmatch{$_} = []; # placeholder to cache misses too.
+ }
+}
+return $pkgmatch unless @files;
+
my $pid = open(DPKG, -|);
syserr(_g(cannot fork for dpkg --search)) unless defined($pid);
if (!$pid) {
@@ -503,7 +516,7 @@
print(STDERR $_\n)
|| syserr(_g(write diversion info to stderr));
} elsif (m/^([^:]+): (\S+)$/) {
- $pkgmatch-{$2} = [ split(/, /, $1) ];
+ $cached_pkgmatch{$2} = $pkgmatch-{$2} = [ split(/, /, $1) ];
} else {
warning(_g(unknown output from dpkg --search: '%s'), $_);
}
---End Message---
---BeginMessage---
Source: dpkg
Source-Version: 1.14.11
We believe that the bug you reported is fixed in the latest version of
dpkg, which is due to be installed in the Debian FTP archive:
dpkg-dev_1.14.11_all.deb
to pool/main/d/dpkg/dpkg-dev_1.14.11_all.deb
dpkg_1.14.11.dsc
to pool/main/d/dpkg/dpkg_1.14.11.dsc
dpkg_1.14.11.tar.gz
to pool/main/d/dpkg/dpkg_1.14.11.tar.gz
dpkg_1.14.11_i386.deb
to pool/main/d/dpkg/dpkg_1.14.11_i386.deb
dselect_1.14.11_i386.deb
to pool/main/d/dpkg/dselect_1.14.11_i386.deb
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to [EMAIL PROTECTED],
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Guillem Jover [EMAIL PROTECTED] (supplier of updated dpkg package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing [EMAIL PROTECTED])
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Format: 1.7
Date: Sat, 24 Nov 2007 07:19:02 +0200
Source: dpkg
Binary: dpkg dselect dpkg-dev
Architecture: source i386 all
Version: 1.14.11
Distribution: unstable
Urgency: low
Maintainer: Dpkg Developers [EMAIL PROTECTED]
Changed-By: Guillem Jover [EMAIL PROTECTED]
Description:
dpkg - package maintenance system for Debian
dpkg-dev - package building tools for Debian
dselect- user tool to manage Debian packages
Closes: 452318 452577 452621
Changes:
dpkg (1.14.11) unstable; urgency=low
.
[ Raphael Hertzog ]
* dpkg-shlibdeps now ignores the lack of dependency information in some
specific cases (instead of failing):
- when the library is in the same package than the binary analyzed
- when
Bug#452621: marked as done (/usr/bin/dpkg-scanpackages: override file required)
Your message dated Sat, 24 Nov 2007 05:47:02 + with message-id [EMAIL PROTECTED] and subject line Bug#452621: fixed in dpkg 1.14.11 has caused the attached Bug report to be marked as done. This means that you claim that the problem has been dealt with. If this is not the case it is now your responsibility to reopen the Bug report if necessary, and/or fix the problem forthwith. (NB: If you are a system administrator and have no idea what I am talking about this indicates a serious mail system misconfiguration somewhere. Please contact me immediately.) Debian bug tracking system administrator (administrator, Debian Bugs database) ---BeginMessage--- Package: dpkg-dev Version: 1.14.7 Severity: minor File: /usr/bin/dpkg-scanpackages The man page says that the override file in invocation of dpkg-scanpackages is optional. However, dpkg-scanpackages fails if the file is not specified. workaround: specify /dev/null as the override file -- System Information: Debian Release: lenny/sid APT prefers testing APT policy: (990, 'testing'), (500, 'unstable'), (1, 'experimental') Architecture: i386 (i686) Kernel: Linux 2.6.23.3-src (SMP w/1 CPU core) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/bash Versions of packages dpkg-dev depends on: ii binutils 2.18-1 The GNU assembler, linker and bina ii cpio 2.9-4 GNU cpio -- a program to manage ar ii dpkg 1.14.7 package maintenance system for Deb ii make 3.81-3 The GNU version of the make util ii patch 2.5.9-4Apply a diff file to an original ii perl [perl5] 5.8.8-12 Larry Wall's Practical Extraction ii perl-modules 5.8.8-12 Core Perl modules Versions of packages dpkg-dev recommends: ii bzip2 1.0.3-7high-quality block-sorting file co ii gcc [c-compiler] 4:4.2.1-6 The GNU C compiler ii gcc-4.2 [c-compiler] 4.2.2-3The GNU C compiler -- no debconf information ---End Message--- ---BeginMessage--- Source: dpkg Source-Version: 1.14.11 We believe that the bug you reported is fixed in the latest version of dpkg, which is due to be installed in the Debian FTP archive: dpkg-dev_1.14.11_all.deb to pool/main/d/dpkg/dpkg-dev_1.14.11_all.deb dpkg_1.14.11.dsc to pool/main/d/dpkg/dpkg_1.14.11.dsc dpkg_1.14.11.tar.gz to pool/main/d/dpkg/dpkg_1.14.11.tar.gz dpkg_1.14.11_i386.deb to pool/main/d/dpkg/dpkg_1.14.11_i386.deb dselect_1.14.11_i386.deb to pool/main/d/dpkg/dselect_1.14.11_i386.deb A summary of the changes between this version and the previous one is attached. Thank you for reporting the bug, which will now be closed. If you have further comments please address them to [EMAIL PROTECTED], and the maintainer will reopen the bug report if appropriate. Debian distribution maintenance software pp. Guillem Jover [EMAIL PROTECTED] (supplier of updated dpkg package) (This message was generated automatically at their request; if you believe that there is a problem with it please contact the archive administrators by mailing [EMAIL PROTECTED]) -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Format: 1.7 Date: Sat, 24 Nov 2007 07:19:02 +0200 Source: dpkg Binary: dpkg dselect dpkg-dev Architecture: source i386 all Version: 1.14.11 Distribution: unstable Urgency: low Maintainer: Dpkg Developers [EMAIL PROTECTED] Changed-By: Guillem Jover [EMAIL PROTECTED] Description: dpkg - package maintenance system for Debian dpkg-dev - package building tools for Debian dselect- user tool to manage Debian packages Closes: 452318 452577 452621 Changes: dpkg (1.14.11) unstable; urgency=low . [ Raphael Hertzog ] * dpkg-shlibdeps now ignores the lack of dependency information in some specific cases (instead of failing): - when the library is in the same package than the binary analyzed - when the library is not versionned and can't have a shlibs file * dpkg-shlibdeps now only displays 10 warnings about symbols not found for each binary and a count of skipped warnings. Closes: #452318 * dpkg-shlibdeps: optimize dpkg -S lookups by caching results, patch from Aaron M. Ucko [EMAIL PROTECTED]. Closes: #452577 . [ Guillem Jover ] * Fix dpkg-scanpackages to properly support an optional override file. Closes: #452621 Files: 7370f9e5232f7ce4ff6c72d30d89eeff 997 admin required dpkg_1.14.11.dsc 15de657f6e7c7e0d91e32d0fa2e48c9d 6279288 admin required dpkg_1.14.11.tar.gz 7051160566f63bb923789baf2b56da77 2148512 admin required dpkg_1.14.11_i386.deb b21e6b77ee111ad94d78f607cf8ec319 511596 admin required dselect_1.14.11_i386.deb a29c6e2fa9323e0f4e9b00a92acf533f 318380 utils optional dpkg-dev_1.14.11_all.deb -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux)
