Bug#793404: massive waste of CPU time in debian/rules by inline commands
Hideki Yamane: > Hi Niels, > > Thanks for your explanation :) > > On Sat, 05 Jan 2019 09:52:00 + > Niels Thykier wrote: >> We are very far from being able to flip the default. There are some >> 800+ packages that need to be updated to follow latest policy >> requirements plus explicitly declare their need for (fake)root. > > Okay, maybe we can achieve it as compat level 13, 14 or 15... > but lg way to go, I agree with it. > > Sorry, but this cannot be done via a compat level. If it could have, I would have done it in compat 12 and moved on. Please see #884999 for the reasons why debhelper cannot change this default for us. Thanks, ~Niels
Bug#793404: massive waste of CPU time in debian/rules by inline commands
Hi Niels, Thanks for your explanation :) On Sat, 05 Jan 2019 09:52:00 + Niels Thykier wrote: > We are very far from being able to flip the default. There are some > 800+ packages that need to be updated to follow latest policy > requirements plus explicitly declare their need for (fake)root. Okay, maybe we can achieve it as compat level 13, 14 or 15... but lg way to go, I agree with it. -- Regards, Hideki Yamane henrich @ debian.org/iijmio-mail.jp
Bug#793404: massive waste of CPU time in debian/rules by inline commands
Hideki Yamane: > Hi, > > On Thu, 03 Jan 2019 11:11:00 + > Niels Thykier wrote: >> * Migrating to "Rules-Requires-Root: no" where possible. > > Is there any plan to set "Rules-Requires-Root: no" for default? > It seems that most of the packages doesn't require root privilege > to be built, so "default no" is better to reduce work, IMHO. > > Hi, I would love to make "Rules-Requires-Root: no" the default and I agree with you that it is a better default, but I have no plans to drag/spearhead this effort. We are very far from being able to flip the default. There are some 800+ packages that need to be updated to follow latest policy requirements plus explicitly declare their need for (fake)root. Known blockers include: * https://lintian.debian.org/tags/debian-rules-missing-recommended-target.html * https://lintian.debian.org/tags/should-specify-rules-requires-root.html ... and my understanding is that it will require NMU'ing a large amount of these 800 packages of the lot to get there (the build-arch|build-indep transition has been going on for years and will probably take 5-8 more years to completely finish if left on its own). I ran out of gas on finishing build-arch+build-indep, so I am not really feeling up for dragging that to the goal line. I have been doing an NMU here and there plus added MR requests on salsa for setting "R³: binary-targets" to fix a few up them but this the level I can motivate myself to do. In case, you are wondering why build-arch/build-indep is related to this problem: One of the reasons why Rules-Requires-Root can reduce the number of debian/rules runs is because we 1) disable dpkg's probing for build-arch and 2) we just call the binary target directly and let the debian/rules file handle that gracefully. Maybe older packages do not do this and just fail to build properly (despite the policy requirement that mandates otherwise) - among other dh before compat 9 (or 10?) does not handle dependencies between rules targets correctly either. In summary: I do not think we can flip the default before those two lintian tags are close to zero and I am not up for that challenge (but I hope someone is). Thanks, ~Niels
Bug#793404: massive waste of CPU time in debian/rules by inline commands
Hi, On Thu, 03 Jan 2019 11:11:00 + Niels Thykier wrote: > * Migrating to "Rules-Requires-Root: no" where possible. Is there any plan to set "Rules-Requires-Root: no" for default? It seems that most of the packages doesn't require root privilege to be built, so "default no" is better to reduce work, IMHO. -- Regards, Hideki Yamane henrich @ debian.org/iijmio-mail.jp
Bug#793404: massive waste of CPU time in debian/rules by inline commands
Hi,
In the past 3½ years, several things have been improved and I am
therefore taking the liberty of closing this bug against general
(remaining issues as I understand it will be in individual packages).
In particular, I think we have identified all major issues, solved most
of them and triaged/assigned the remaining ones.
For individual packages, improvements are often a question of:
* Migrating to "Rules-Requires-Root: no" where possible.
* Avoid calling dpkg-parsechangelog directly and instead use dpkg's
makefile snippets.
* Replace "comment only" override targets with completely empty
override targets.
(Not necessarily listed in order of "best performance for value" as that
depends on exactly what the package does)
On Fri, 24 Jul 2015 12:19:36 +0200 Jakub Wilk wrote:
> * Jonas Smedegaard , 2015-07-23, 21:40:
> >>One mistake boost makes is using ":=" instead of plain "=". Contrary
> >>to popular belief, the former almost always causes more evaluation of
> >>$(shell) stuff, specially when dh is involved.
> >Could you elaborate on that?
>
> dpkg-buildpackage -B will run debian/rules 4 times: once to determine if
> build-arch exist, and once for every target: clean, build(-arch),
> binary-arch.
>
> dh adds even more debian/rules invocations. It runs it once every target
> (clean, build(-arch), binary-arch), and once for every override.
>
> So your ":=" variable will be evaluated 4 times, or 7+N times if you use
> dh.
>
> "=" variables will be evaluated only when they are used, which is less
> than 4 or 7+N in most cases.
>
> --
> Jakub Wilk
>
>
This issue is still prevalent but we now have support for reducing the
number of calls to debian/rules, which is "opt-in" for packages at the
moment.
With "Rules-Requires-Root: no" (recently added to policy), you can
remove two debian/rules invocation on the dpkg side and one on the dh
side. Which brings us down to:
* 2 calls from dpkg (one for clean and one for the binary target)
* 2+N calls for dh to probe the rules file for override targets plus
the N (non-empty) overrides a package might have.
Guillem Jover wrote:
> Control: block -1 by 793330
>
> Hi!
>
> [...]
>
> There are multiple culprits that pile up here:
>
> 1) The /usr/share/dpkg/architecture.mk and /usr/share/dpkg/buildflags.mk
>lazy and caching value initialization is not effective. I had noticed
>it but had not yet checked if it was a problem with the makefiles or
>in make, etc. It appears is a bad interaction with the foreach, which
>defeats the lazy and cached evaluation. I guess I'll try to make the
>foreach work, or revert to an unrolled implementation.
>
I believe this has been fixed now.
> 2) debhelper's Dh_Lib.pm does not try to use existing dpkg-architecture
>variables from the environment. Those should not be expected to be
>present, but when using dpkg-buildpackage they will be present so it
>would be an optimization. I'll file a bug report about this.
>
This has been fixed now.
> 3) Slow dpkg-parsechangelog implementation and usage:
>
Lintian now flags use of dpkg-parsechangelog in most cases and
recommends people to migrate to the optimized makefile snippets in dpkg.
https://lintian.debian.org/tags/debian-rules-parses-dpkg-parsechangelog.html
>> In the emulated m68k environment, it spends about half an hour (guessed,
>> not measured) before starting the actual build, doing things like:
>>
>> | \_ /usr/bin/perl -w /usr/bin/dh build --with python2 --with python3
>> | \_ /usr/bin/make -f debian/rules override_dh_auto_configure
>> | \_ /bin/sh -c dpkg-parsechangelog | grep Version | cut -d' '
>> -f2
>> | \_ /usr/bin/perl /usr/bin/dpkg-parsechangelog
>> | | \_ /usr/bin/perl /usr/lib/dpkg/parsechangelog/debian
>> -ldebian/changelog --file debia
>
> 3.1) As mentioned in the thread, callers can avoid the other shell
> commands and pipes by using -S.
>
(Will be handled by the lintian tag)
> 3.2) debian/rules (or debhelper/cdbs) will still call the program for
> different changelog values. But dpkg-buildpackage has to parse the
> current and previous entries anyway, so we could preset values for
> those in the environment that could opportunistically be used by
> debian/rules and debhelper/cdbs. A possible drawback is that
> packages might accidentally rely on those variables w/o setting
> them beforehand.
>
This has not been implemented. However, debhelper has implemented two
features to improve performance here:
* debhelper now uses the Dpkg module internally instead of forking
dpkg-parsechangelog, which reduces a bit of runtime.
* debhelper now caches the result from d/changelog so we at most parse
that file once per helper needs to know the version ($dh{VERSION}).
(Down from "one per binary package built per helper needing
$dh{VERSION}")
In the default sequence, the only (non-error)
Bug#793404: massive waste of CPU time in debian/rules by inline commands
On 2015-07-25 10:27, Niels Thykier wrote:
[...]
So your := variable will be evaluated 4 times, or 7+N times if you use
dh.
[...]
It should be doable to reduce the dh side of it to 1+N by caching the
result of make (in a file-based cache).
~Niels
Correction, 2+N (of the dh caused one). Attached is a PoC patch for
implementing said cache. Unfortunately, it is only effective for
removing one parsing of debian/rules (2 when using -tc or doing repeated
builds from the same directory).
At this point, I am not entirely convinced the complexity is worth the
gains - especially not with Guillem's plans for dpkg/1.18.2, which is
likely to give more and make this patch less useful. I might reconsider
it if there is still an issue after next dpkg+debhelper upload.
Thanks,
~Niels
From 61c48ce2c5038a2cbd38b0cb63433e0589515c0e Mon Sep 17 00:00:00 2001
From: Niels Thykier ni...@thykier.net
Date: Tue, 28 Jul 2015 22:41:41 +0200
Subject: [PATCH] dh: Cache target information from make
Cache which targets are empty and which are not. This avoids at least
1 runs over debian/rules in a regular build. The cache has to be
purged during clean, so the solution is rather limited.
Example of when the cache is used and when it is not:
dh clean -- reads debian/rules
...
dh build -- reads debian/rules (clean leaves no cache)
...
dh binary -- uses cache
...
dh clean -- uses cache but purges it afterwards
The latter dh clean happens with repeated builds or
dpkg-buildpackage -tc.
Signed-off-by: Niels Thykier ni...@thykier.net
---
debian/changelog | 4
dh | 57
2 files changed, 61 insertions(+)
diff --git a/debian/changelog b/debian/changelog
index 650b294..62f1a8b 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -22,6 +22,10 @@ debhelper (9.20150629) UNRELEASED; urgency=medium
(Closes: #793443)
* dh_strip: Always compress debug sections of debug symbols
in ddebs.
+ * dh: Cache the output of make(1) to reduce the number of times
+debian/rules is evaluated. This will primarily speed up
+builds that use many or expensive := variables in their
+debian/rules files (Fore more info, please see #793404)
-- Niels Thykier ni...@thykier.net Sun, 28 Jun 2015 15:08:19 +0200
diff --git a/dh b/dh
index 07facd0..e5050ed 100755
--- a/dh
+++ b/dh
@@ -343,6 +343,9 @@ if ($sequence eq 'debian/rules' ||
exit 0;
}
+# Some cache variables
+my $CACHE_DIR = 'debian/.debhelper';
+my $RULES_CACHE = ${CACHE_DIR}/rules-target;
# Definitions of sequences.
my %sequences;
@@ -647,6 +650,14 @@ elsif ($dh{BEFORE}) {
$stoppoint=command_pos($dh{BEFORE}, @sequence) - 1;
}
+if (not $dh{NO_ACT} and @sequence and $sequence[-1] ne 'dh_clean') {
+ # Only write the cache if there is something in the sequence (in
+ # the off chance that someone had a --remaining after dh_clean).
+ # Also, don't bother writing the cache if the sequence will remove
+ # it (clean does not imply any dh target calls).
+ update_rules_cache();
+}
+
# Now run the commands in the sequence.
foreach my $i (0..$stoppoint) {
my $command=$sequence[$i];
@@ -843,6 +854,26 @@ sub rules {
{
my %targets;
my $rules_parsed;
+my $cache_outdated;
+
+sub update_rules_cache {
+ return if not $cache_outdated;
+
+ install_dir($CACHE_DIR);
+ # Ignore errors - it is not a critical file
+ if (open(my $fd, '', $RULES_CACHE)) {
+ my (@explicit, @empty);
+ for my $target (keys(%targets)) {
+ if ($targets{$target}) {
+push(@explicit, $target);
+ } else {
+push(@empty, $target);
+ }
+ }
+ print {$fd} 'explicit: ' . join(',', @explicit) . \n if @explicit;
+ print {$fd} 'empty: ' . join(',', @empty) . \n if @empty;
+ }
+}
sub rules_explicit_target {
# Checks if a specified target exists as an explicit target
@@ -850,6 +881,32 @@ sub rules_explicit_target {
# undef is returned if target does not exist, 0 if target is noop
# and 1 if target has dependencies or executes commands.
my $target=shift;
+ if (not $rules_parsed) {
+ # Check if we have a (valid) cache from the last run.
+ my $mtime_rules = ((stat('debian/rules'))[9]) // 0;
+ my $mtime_cache = ((stat($RULES_CACHE))[9]) // -1;
+ # Ignore errors from open - it is not a critical file.
+ if ($mtime_cache $mtime_rules and open(my $fd, '', $RULES_CACHE)) {
+ while (my $line = $fd) {
+my $value;
+chomp($line);
+if ($line =~ s/^explicit: //) {
+ $value = 1;
+} elsif ($line =~ s/^empty: //) {
+ $value = 0;
+} else {
+ next;
+}
+for my $target (split(' ', $line)) {
+ $targets{$target} = $value;
+}
+ }
+ $rules_parsed = 1;
+ close($fd);
+ } else {
+ $cache_outdated = 1;
+ }
+ }
if (! $rules_parsed) {
my $processing_targets = 0;
--
2.4.6
Bug#793404: massive waste of CPU time in debian/rules by inline commands
On 2015-07-24 12:19, Jakub Wilk wrote:
* Jonas Smedegaard d...@jones.dk, 2015-07-23, 21:40:
One mistake boost makes is using := instead of plain =. Contrary
to popular belief, the former almost always causes more evaluation of
$(shell) stuff, specially when dh is involved.
Could you elaborate on that?
dpkg-buildpackage -B will run debian/rules 4 times: once to determine if
build-arch exist, and once for every target: clean, build(-arch),
binary-arch.
dh adds even more debian/rules invocations. It runs it once every target
(clean, build(-arch), binary-arch), and once for every override.
Indeed - though hardly the common use for override targets, you /can/
exclude a command for free by declaring a /completely/ empty target.
Have a look at:
$ dh binary --no-act | grep debian/rules | wc -l
$ dh clean --no-act | grep debian/rules | wc -l
Which should tell you how many override non-empty targets you have.
Output (of dh foo --no-act) is something like:
dh_testdir
debian/rules override_dh_auto_configure
dh_auto_build
[...]
dh_installdirs
debian/rules override_dh_auto_install
[...]
If your override target is completely empty, the command/target simply
disappears from the list. Though it does not account for the extra
overhead from dpkg-buildpackage calling build{,-arch,-indep} first in a
separate step.
So your := variable will be evaluated 4 times, or 7+N times if you use
dh.
[...]
It should be doable to reduce the dh side of it to 1+N by caching the
result of make (in a file-based cache).
~Niels
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/55b34861.6040...@thykier.net
Bug#793404: massive waste of CPU time in debian/rules by inline commands
* Jonas Smedegaard d...@jones.dk, 2015-07-23, 21:40: One mistake boost makes is using := instead of plain =. Contrary to popular belief, the former almost always causes more evaluation of $(shell) stuff, specially when dh is involved. Could you elaborate on that? dpkg-buildpackage -B will run debian/rules 4 times: once to determine if build-arch exist, and once for every target: clean, build(-arch), binary-arch. dh adds even more debian/rules invocations. It runs it once every target (clean, build(-arch), binary-arch), and once for every override. So your := variable will be evaluated 4 times, or 7+N times if you use dh. = variables will be evaluated only when they are used, which is less than 4 or 7+N in most cases. -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150724101936.ga7...@jwilk.net
Bug#793404: massive waste of CPU time in debian/rules by inline commands
Control: block -1 by 657390 [ Blocking on that, because there's currently no other such bug. So not to imply this is lintian maintainers sole responsibility. ] Hi! On Fri, 2015-07-24 at 12:19:36 +0200, Jakub Wilk wrote: dpkg-buildpackage -B will run debian/rules 4 times: once to determine if build-arch exist, and once for every target: clean, build(-arch), binary-arch. Ideally the first would disappear though. Please see the blocking bug report for a tentative plan on how to do so. Thanks, Guillem -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150724143301.ga32...@gaara.hadrons.org
Processed: Re: Bug#793404: massive waste of CPU time in debian/rules by inline commands
Processing control commands: block -1 by 657390 Bug #793404 [general] massive waste of CPU time in debian/rules by inline commands 793404 was blocked by: 793440 793330 793443 793404 was not blocking any bugs. Added blocking bug(s) of 793404: 657390 -- 793404: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=793404 Debian Bug Tracking System Contact ow...@bugs.debian.org with problems -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/handler.s.b793404.143774839927015.transcr...@bugs.debian.org
Processed: Re: Bug#793404: massive waste of CPU time in debian/rules by inline commands
Processing control commands: block -1 by 793330 Bug #793404 [general] massive waste of CPU time in debian/rules by inline commands 793404 was not blocked by any bugs. 793404 was not blocking any bugs. Added blocking bug(s) of 793404: 793330 -- 793404: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=793404 Debian Bug Tracking System Contact ow...@bugs.debian.org with problems -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/handler.s.b793404.143769362926628.transcr...@bugs.debian.org
Bug#793404: massive waste of CPU time in debian/rules by inline commands
Quoting Jakub Wilk (2015-07-23 21:08:15) * Eduard Bloch e...@gmx.de, 2015-07-23, 19:43: The problem: I see lots of $(shell ...) stuff. In boost, there are about 12 such calls. And they run dpkg-architecture or dpkg-parsechangelogs or similar commands. When this was done a just couple of times (i.e. before dh(7)), that's acceptable. But now, it looks like debian/rules is called many, many times through dh. One mistake boost makes is using := instead of plain =. Contrary to popular belief, the former almost always causes more evaluation of $(shell) stuff, specially when dh is involved. Could you elaborate on that? I never use short-form dh but would like to understand if the optimizations I've tried to do in CDBS might do more harm than good. - Jonas -- * Jonas Smedegaard - idealist Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Re: Bug#793404: massive waste of CPU time in debian/rules by inline commands
Jonas Smedegaard wrote: Quoting Jakub Wilk (2015-07-23 21:08:15) * Eduard Bloch e...@gmx.de, 2015-07-23, 19:43: The problem: I see lots of $(shell ...) stuff. In boost, there are about 12 such calls. And they run dpkg-architecture or dpkg-parsechangelogs or similar commands. When this was done a just couple of times (i.e. before dh(7)), that's acceptable. But now, it looks like debian/rules is called many, many times through dh. One mistake boost makes is using := instead of plain =. Contrary to popular belief, the former almost always causes more evaluation of $(shell) stuff, specially when dh is involved. Could you elaborate on that? I never use short-form dh but would like to understand if the optimizations I've tried to do in CDBS might do more harm than good. If you use :=, the shell command is immediately run and the result stored in the variable. So if you reference the variable a dozen times, the shell command gets run once. And if you reference the variable zero times, the shell command gets run once. If you use =, the shell command is run when the variable is evaluated. So if you reference the variable a dozen times, the shell command gets run a dozen times. And if you reference the variable zero times, the shell command gets run zero times. So := is an optimization if you expect to evaluate the variable more than once. It's a pessimization if you don't evaluate the variable. If the invocations of make for dh aren't referencing those variables, using := is a pessimization. - Josh Triplett -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150723205749.GA8552@jtriplet-mobl1
Bug#793404: massive waste of CPU time in debian/rules by inline commands
On Thu, Jul 23, 2015 at 09:08:15PM +0200, Jakub Wilk wrote: * Eduard Bloch e...@gmx.de, 2015-07-23, 19:43: The problem: I see lots of $(shell ...) stuff. In boost, there are about 12 such calls. And they run dpkg-architecture or dpkg-parsechangelogs or similar commands. When this was done a just couple of times (i.e. before dh(7)), that's acceptable. But now, it looks like debian/rules is called many, many times through dh. One mistake boost makes is using := instead of plain =. Contrary to popular belief, the former almost always causes more evaluation of $(shell) stuff, specially when dh is involved. Or you can use: lazy = $(eval $(1) = $$(if $$(___$(1)),,$$(eval ___$(1) := $(2)))$$(___$(1))) $(call lazy,VARIABLE_NAME,$$(shell command)) And then the command will only execute when you actually use VARIABLE_NAME for the first time. Mike -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150723221027.ga11...@glandium.org
Bug#793404: massive waste of CPU time in debian/rules by inline commands
Control: block -1 by 793330 Hi! On Thu, 2015-07-23 at 19:43:46 +0200, Eduard Bloch wrote: Package: general Severity: minor The problem: I see lots of $(shell ...) stuff. In boost, there are about 12 such calls. And they run dpkg-architecture or dpkg-parsechangelogs or similar commands. When this was done a just couple of times (i.e. before dh(7)), that's acceptable. But now, it looks like debian/rules is called many, many times through dh. Making many, many calls of that inline commands. Wasting many, many CPU cycles. All that just to retrieve the same information all over again. There are multiple culprits that pile up here: 1) The /usr/share/dpkg/architecture.mk and /usr/share/dpkg/buildflags.mk lazy and caching value initialization is not effective. I had noticed it but had not yet checked if it was a problem with the makefiles or in make, etc. It appears is a bad interaction with the foreach, which defeats the lazy and cached evaluation. I guess I'll try to make the foreach work, or revert to an unrolled implementation. 2) debhelper's Dh_Lib.pm does not try to use existing dpkg-architecture variables from the environment. Those should not be expected to be present, but when using dpkg-buildpackage they will be present so it would be an optimization. I'll file a bug report about this. 3) Slow dpkg-parsechangelog implementation and usage: In the emulated m68k environment, it spends about half an hour (guessed, not measured) before starting the actual build, doing things like: | \_ /usr/bin/perl -w /usr/bin/dh build --with python2 --with python3 | \_ /usr/bin/make -f debian/rules override_dh_auto_configure | \_ /bin/sh -c dpkg-parsechangelog | grep Version | cut -d' ' -f2 | \_ /usr/bin/perl /usr/bin/dpkg-parsechangelog | | \_ /usr/bin/perl /usr/lib/dpkg/parsechangelog/debian -ldebian/changelog --file debia 3.1) As mentioned in the thread, callers can avoid the other shell commands and pipes by using -S. 3.2) debian/rules (or debhelper/cdbs) will still call the program for different changelog values. But dpkg-buildpackage has to parse the current and previous entries anyway, so we could preset values for those in the environment that could opportunistically be used by debian/rules and debhelper/cdbs. A possible drawback is that packages might accidentally rely on those variables w/o setting them beforehand. 3.3) dpkg-parsechangelog supports other changelog formats, and those are implemented by external parsers. This means it needs to scan the changelog twice, and then parse+output+parse the data from the parser. I've already implemented an optimization (to be included in dpkg 1.18.2) when forcing the format to be debian, that uses a builtin parser, which halves the execution time. «dpkg-parsechangelog -Fdebian». I guess I can take this further and use the builtin parser whenever the format is debian. And probably some others… Thanks, Guillem -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150723232012.ga32...@gaara.hadrons.org
Bug#793404: massive waste of CPU time in debian/rules by inline commands
Quoting Eduard Bloch (2015-07-23 19:43:46) The problem: I see lots of $(shell ...) stuff [in debian/rules files]. In boost, there are about 12 such calls. And they run dpkg-architecture or dpkg-parsechangelogs or similar commands. debian/rules is a make file. In the make language, variables can be expanded either immediately (similar to shell) or when used. This is expensive (when used multiple times as is the case with Boost): pyversions = $(shell pyversions -rv) $(shell py3versions -rv) Changing to early expanded saves CPU cycles: pyversions := $(shell pyversions -rv) $(shell py3versions -rv) Hope that helps, - Jonas -- * Jonas Smedegaard - idealist Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Bug#793404: massive waste of CPU time in debian/rules by inline commands
On Thu, 23 Jul 2015 19:43:46 +0200 Eduard Bloch e...@gmx.de wrote: please tell me that I am wrong or that I start fighting the windmills if that's the case, but I have a general impression that something smells in lots of debian/rules files nowadays and we need a concept to improve that. The problem: I see lots of $(shell ...) stuff. In boost, there are about 12 such calls. And they run dpkg-architecture or dpkg-parsechangelogs or similar commands. When this was done a just couple of times (i.e. before dh(7)), that's acceptable. But now, it looks like debian/rules is called many, many times through dh. Making many, many calls of that inline commands. Wasting many, many CPU cycles. All that just to retrieve the same information all over again. In the emulated m68k environment, it spends about half an hour (guessed, not measured) before starting the actual build, doing things like: | \_ /usr/bin/perl -w /usr/bin/dh build --with python2 --with python3 | \_ /usr/bin/make -f debian/rules override_dh_auto_configure | \_ /bin/sh -c dpkg-parsechangelog | grep Version | cut -d' ' -f2 | \_ /usr/bin/perl /usr/bin/dpkg-parsechangelog | | \_ /usr/bin/perl /usr/lib/dpkg/parsechangelog/debian -ldebian/changelog --file debia I think you're right. $(shell ...) isn't a good idea in make, as it always runs regardless of the target. Worse yet, those shell commands are not a trivial amount of processing. Ideally, this information should be obtained *once* at the start of the build, cached in a file that depends on the necessary bits (e.g. parse the version from the changelog into a file that depends on debian/changelog), and then used from there. Whether that's done via make, or by having commands like dpkg-parsechangelog cache the relevant bits of thir output and check mtime on the files they parse. We should try to reduce the number of such things that are actually needed in debian/rules; if they're only needed in a particular target, or in some specific command, let's put them in that target or command. Also, things like grep Version | cut ... should really be replaced with things like dpkg-parsechangelog -SVersion. It's unfortunate that that make can't run a command without using the shell. For that matter, all the override_* targets should ideally be handled in some way that doesn't involve a full invocation of make, if the target doesn't actually exist. That's harder, but it should be possible to run make once, parse the list of targets, and use those. If we can reduce the number of invocations of make, that would help as well. - Josh Triplett -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150723190551.GA2159@jtriplet-mobl1
Bug#793404: massive waste of CPU time in debian/rules by inline commands
* Eduard Bloch e...@gmx.de, 2015-07-23, 19:43: The problem: I see lots of $(shell ...) stuff. In boost, there are about 12 such calls. And they run dpkg-architecture or dpkg-parsechangelogs or similar commands. When this was done a just couple of times (i.e. before dh(7)), that's acceptable. But now, it looks like debian/rules is called many, many times through dh. One mistake boost makes is using := instead of plain =. Contrary to popular belief, the former almost always causes more evaluation of $(shell) stuff, specially when dh is involved. Anyway, in my packages, I fixed all problems with dh by not using dh. :-) -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150723190815.ga4...@jwilk.net
Bug#793404: massive waste of CPU time in debian/rules by inline commands
Package: general Severity: minor Hello Fellow Maintainers, please tell me that I am wrong or that I start fighting the windmills if that's the case, but I have a general impression that something smells in lots of debian/rules files nowadays and we need a concept to improve that. The problem: I see lots of $(shell ...) stuff. In boost, there are about 12 such calls. And they run dpkg-architecture or dpkg-parsechangelogs or similar commands. When this was done a just couple of times (i.e. before dh(7)), that's acceptable. But now, it looks like debian/rules is called many, many times through dh. Making many, many calls of that inline commands. Wasting many, many CPU cycles. All that just to retrieve the same information all over again. In the emulated m68k environment, it spends about half an hour (guessed, not measured) before starting the actual build, doing things like: | \_ /usr/bin/perl -w /usr/bin/dh build --with python2 --with python3 | \_ /usr/bin/make -f debian/rules override_dh_auto_configure | \_ /bin/sh -c dpkg-parsechangelog | grep Version | cut -d' ' -f2 | \_ /usr/bin/perl /usr/bin/dpkg-parsechangelog | | \_ /usr/bin/perl /usr/lib/dpkg/parsechangelog/debian -ldebian/changelog --file debia I think we need to find a shortcut for this. The best idea I got after short brainstorming is hacking fakeroot to provide a cache of stdout data from certain commands. If someone has a better idea or could point to an existing concept or implementation, please tell me. Regards, Eduard. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150723174346.ga28...@rotes76.wohnheim.uni-kl.de
