Bug#560423: debhelper: Add debian/dh_norun file to list, commands not to run (ver 8?)
X-Enigmail-Version: 1.4.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Joey Hess wrote: There is some potential for speeding up dh. Josh Tripplet and I once discussed: * Having dh optimize out calls to debhelper commands that will be no-ops. For a subset of commands, this can be determined just by looking for their config files. Some kind of metadata would be necessary for dh to tell which commands are in that subset. Hi, Perhaps the sequence modules could contain such information? This would allow third-party sequences to take advantage of this optimization as well. * Speed up dh_gencontrol, er, I mean dpkg-gencontrol, which by itself is responsible for more than 1/3 of the total runtime of dh binary in a package where the other debhelper commands are all no-ops. Can you remember what method you used to determine this? I would like to see if this is still the case, before I try look at optimizing it (if only to know when I am done ;) ). ~Niels -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#560423: debhelper: Add debian/dh_norun file to list, commands not to run (ver 8?)
On 2012-07-12 12:41, Niels Thykier wrote: Joey Hess wrote: There is some potential for speeding up dh. Josh Tripplet and I once discussed: * Having dh optimize out calls to debhelper commands that will be no-ops. For a subset of commands, this can be determined just by looking for their config files. Some kind of metadata would be necessary for dh to tell which commands are in that subset. Hi, Perhaps the sequence modules could contain such information? This would allow third-party sequences to take advantage of this optimization as well. [...] ~Niels I have developed a prototype patch for optimizing out unused debhelper commands from a dh sequence. The basic idea it is to maintain a table in dh mapping skippable commands to their pkgfile-files. If a command is in the table and none of these files are present (for any package to be acted on), then the command is skipped. It gives backwards compatible semantics for add-on sequences (their commands are not present in the table). Commands are checked just before they are run, but after their override targets are run (if any). This allows the build to auto-generate the files and should be backwards compatible as far as I can tell. The prototype does /not/ include an API extension for sequence modules to add their commands to the auto skip table. However, it should be fairly trivial to create and I will happily add it if this approach is deemed reasonable. I have included a small print statement to inform when a command is skipped. I have no strong feelings for whether or not it should stay; I merely added it to assist the development. I have included the following skip rules (long lines truncated): dh_bugfiles = [qw(bug-script bug-control bug-presubj)], dh_install = [install], dh_installcron = [qw(cron.daily ...)], dh_installcatalogs = [sgmlcatalogs], dh_installemacsen = [qw(emacsen-install ...)], dh_installexamples = [examples], dh_installifupdown = [qw(if-up if-down ...)], dh_installinfo = [info], dh_lintian = [lintian-overrides], dh_installmenu = [menu, menu-method], dh_ucf = [ucf], The commands were added based on memory or/and a quick view in the manpage. It can most likely be extended and (to my knowledge) does not include commands that may do stuff based on what it is in debian/pkg (but do correct me if I am wrong). ~Niels sample.patch Description: application/wine-extension-patch
Bug#560423: debhelper: Add debian/dh_norun file to list commands not to run (ver 8?)
Modestas Vainius modes...@vainius.eu writes: * Detect noop (empty) override targets when parsing debian/rules and avoid make calls for them. Patch attached. Any progress with this patch? -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#560423: debhelper: Add debian/dh_norun file to list commands not to run (ver 8?)
retitle 560423 debhelper: Add debian/dh.norun file to list commands not to run (ver 8?) thanks debian/dh_norun As the files under debian/* use the dot(.) notation, like in: debian/package.install Perhaps it is better to use similar convention for dh specific instructions: debian/dh.filename Like in debian/dh.norun Jari -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#560423: debhelper: Add debian/dh_norun file to list commands not to run (ver 8?)
Hello, On penktadienis 11 Gruodis 2009 03:41:44 Jari Aalto wrote: The debhelper 7 is good and in some ways excessive. To build a simple package, it churns over all possible commands (see below). SUGGESTION Please add support for file debian/dh_norun The file would list those commands that do not need to be run for the current package. Commands would be listed one at a line. Comments would be allowed with #. IMHO, this would be very confusing. If you want this, do not use dh sequencer at all and list all commands you need in debian/rules (conventional packaging method). If your suggestion is implemented, somebody else looking at the package will have hard time figuring out what commands will actually be run. If you want to skip only a few commands, use .PHONY overrides, e.g. only this line is enough: PHONY: override_dh_auto_test If you want to skip a bunch of sequential commands, you may also use --before / --until / --after. This would speed up packaging in low end machines. If debhelper program has nothing to do, it will exit fast. Slow down should be minimal. Of those, for simple package, sometimes only these are needed (target: binary-arch): dh_testdir dh_testroot dh_installchangelogs dh_installdocs dh_installexamples dh_link dh_strip dh_compress dh_fixperms dh_installdeb dh_shlibdeps dh_gencontrol dh_md5sums dh_builddeb This is exactly what conventional packaging with debhelper is for. Since you want full control over sequence, use this method. -- Modestas Vainius modes...@vainius.eu signature.asc Description: This is a digitally signed message part.
Bug#560423: debhelper: Add debian/dh_norun file to list commands not to run (ver 8?)
Modestas Vainius modes...@vainius.eu writes: SUGGESTION Please add support for file c debian/dh_norun The file would list those commands that do not need to be run for the current package. Commands would be listed one at a line. Comments would be allowed with #. IMHO, this would be very confusing. If you want this, do not use dh sequencer at all and list all commands you need in debian/rules (conventional packaging method). If your suggestion is implemented, somebody else looking at the package will have hard time figuring out what commands will actually be run. The commands that are rub by dh are displayed to the stdout. The commands that would be left out would simply not appear in that listing. This would concentrate on what is actually being run and used in the package. It is in fact confusing to see run of dh_perl if that is no-op for the package. The idea of debhelper is to use debian/file to control behavior (Cf. debian/docs, debian/dirs ...) If you want to skip only a few commands, use .PHONY overrides, e.g. only this line is enough: PHONY: override_dh_auto_test For single command yes, but as in my example, there are tens of commands that need not to be run. If you want to skip a bunch of sequential commands, you may also use --before / --until / --after. That's good for finer tuning, but a simple debian/dh_norun file would just announce these commands are not used by this package; they have no relevance This would speed up packaging in low end machines. If debhelper program has nothing to do, it will exit fast. Slow down should be minimal. The cost of startup of dh_* is an issue because they are Perl program. I'm sorry to tell you, but there are PII and PIII CPUs still (I have those) that I use for testing and the speed is an issue when iterating over package. Of those, for simple package, sometimes only these are needed (target: binary-arch): dh_testdir dh_testroot dh_installchangelogs dh_installdocs dh_installexamples dh_link dh_strip dh_compress dh_fixperms dh_installdeb dh_shlibdeps dh_gencontrol dh_md5sums dh_builddeb This is exactly what conventional packaging with debhelper is for. Since you want full control over sequence, use this method. Not full control. There is no need to write full blown debian/rules when it could simply read: # Makefile debian/rules %: dh $@ And the commads not used listed in: debian/dh_norun This would be both simple, elegant and maintenence friendly. Jari -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#560423: debhelper: Add debian/dh_norun file to list commands not to run (ver 8?)
Modestas Vainius wrote: If debhelper program has nothing to do, it will exit fast. Slow down should be minimal. There is some potential for speeding up dh. Josh Tripplet and I once discussed: * Having dh optimize out calls to debhelper commands that will be no-ops. For a subset of commands, this can be determined just by looking for their config files. Some kind of metadata would be necessary for dh to tell which commands are in that subset. * Make dh run debhelper commands directly in one perl process, rather than through exec. (However, I think this is unlikely to be worth the insanity.) * Writing a faster parser for simple command-line options. Benchmarks show that parsing options is 50% of the overhead of running no-op debhelper commands. This is because if there are options, it loads Getopt::Long, which is slow. This overhead is particularly bad when dh needs to pass -i / -a to all commands, which otherwise would need no options. * Removing calls to dpkg-architecture from dh_auto_configure and dh_auto_build when they will not actually be configuring or building. * Speed up dh_gencontrol, er, I mean dpkg-gencontrol, which by itself is responsible for more than 1/3 of the total runtime of dh binary in a package where the other debhelper commands are all no-ops. -- see shy jo signature.asc Description: Digital signature
Bug#560423: debhelper: Add debian/dh_norun file to list commands not to run (ver 8?)
Hello,
On penktadienis 11 Gruodis 2009 20:58:44 Joey Hess wrote:
Modestas Vainius wrote:
If debhelper program has nothing to do, it will exit fast. Slow down
should be minimal.
There is some potential for speeding up dh. Josh Tripplet and I once
discussed:
* Having dh optimize out calls to debhelper commands that will be
no-ops. For a subset of commands, this can be determined just
by looking for their config files. Some kind of metadata would be
necessary for dh to tell which commands are in that subset.
* Make dh run debhelper commands directly in one perl process, rather
than through exec. (However, I think this is unlikely to be worth the
insanity.)
* Writing a faster parser for simple command-line options. Benchmarks
show that parsing options is 50% of the overhead of running no-op
debhelper commands. This is because if there are options, it loads
Getopt::Long, which is slow. This overhead is particularly bad
when dh needs to pass -i / -a to all commands, which otherwise
would need no options.
* Removing calls to dpkg-architecture from dh_auto_configure and
dh_auto_build when they will not actually be configuring or building.
* Speed up dh_gencontrol, er, I mean dpkg-gencontrol, which by itself
is responsible for more than 1/3 of the total runtime of dh binary
in a package where the other debhelper commands are all no-ops.
* Detect noop (empty) override targets when parsing debian/rules and avoid
make calls for them. Patch attached.
--
Modestas Vainius modes...@vainius.eu
From cbd509e6b6fe690c0c59df7f117b402be8c89bff Mon Sep 17 00:00:00 2001
From: Modestas Vainius modes...@vainius.eu
Date: Fri, 11 Dec 2009 23:10:01 +0200
Subject: [PATCH] Do not call make for noop overrides.
1) Detect if target is noop when parsing debian/rules.
2) If override target is noop, do not call make for it.
---
dh | 79 +++
1 files changed, 55 insertions(+), 24 deletions(-)
diff --git a/dh b/dh
index 11fb27b..f1b1b68 100755
--- a/dh
+++ b/dh
@@ -526,30 +526,44 @@ sub run {
# Check for override targets in debian/rules and
# run them instead of running the command directly.
my $override_command;
- if (rules_explicit_target(override_.$command)) {
+ my $has_explicit_target = rules_explicit_target(override_.$command);
+ if (defined $has_explicit_target) {
$override_command=$command;
- # This passes the options through to commands called
- # inside the target.
- $ENV{DH_INTERNAL_OPTIONS}=join(\x1e, @options);
- $command=debian/rules;
- @options=override_.$override_command;
+ # Check if target isn't noop
+ if ($has_explicit_target) {
+ # This passes the options through to commands called
+ # inside the target.
+ $ENV{DH_INTERNAL_OPTIONS}=join(\x1e, @options);
+ $command=debian/rules;
+ @options=override_.$override_command;
+ }
+ else {
+ $command = undef;
+ }
}
else {
# Pass additional command options if any
unshift @options, @{$command_opts{$command}} if exists $command_opts{$command};
}
- # 3 space indent lines the command being run up under the
- # sequence name after dh .
- print.escape_shell($command, @options).\n;
+ if (defined $command) {
+ # 3 space indent lines the command being run up under the
+ # sequence name after dh .
+ print.escape_shell($command, @options).\n;
+ }
+ else {
+ print, # Skipping , $override_command, - empty override, \n;
+ }
if (! $dh{NO_ACT}) {
- my $ret=system($command, @options);
- if ($ret 8 != 0) {
- exit $ret 8;
- }
- elsif ($ret) {
- exit 1;
+ if (defined $command) {
+ my $ret=system($command, @options);
+ if ($ret 8 != 0) {
+exit $ret 8;
+ }
+ elsif ($ret) {
+exit 1;
+ }
}
if (defined $override_command) {
@@ -575,11 +589,14 @@ my $rules_parsed;
sub rules_explicit_target {
# Checks if a specified target exists as an explicit target
# in debian/rules.
+ # undef is returned if target does not exist, 0 if target is noop
+ # and 1 if target has dependencies or executes commands.
my $target=shift;
-
- if (! $rules_parsed) {
+
+ if (! $rules_parsed) {
my $processing_targets = 0;
my $not_a_target = 0;
+ my $current_target;
open(MAKE, LC_ALL=C make -Rrnpsf debian/rules debhelper-fail-me 2/dev/null |);
while (MAKE) {
if ($processing_targets) {
@@ -587,12 +604,26 @@ sub rules_explicit_target {
$not_a_target = 1;
}
else {
- if (!$not_a_target /^([^#:]+)::?/) {
- # Target is defined.
- # NOTE: if it is a depenency
- # of .PHONY it will be
- # defined too but that's ok.
- $targets{$1} = 1;
+ if (!$not_a_target /^([^#:]+)::?\s*(.*)$/) {
+ # Target is defined. NOTE: if it is a depenency of
+ # .PHONY it will be defined too but that's ok.
+ # $2 contains target dependencies if any.
+ $current_target = $1;
+ $targets{$current_target} = ($2) ? 1 : 0;
+ }
+ else {
Bug#560423: debhelper: Add debian/dh_norun file to list commands not to run (ver 8?)
Hello, On penktadienis 11 Gruodis 2009 13:23:27 Jari Aalto wrote: That's good for finer tuning, but a simple debian/dh_norun file would just announce these commands are not used by this package; they have no relevance I don't understand why you want another file for this. Sequence setup naturally *belongs* to debian/rules, it should be visible from there. This would speed up packaging in low end machines. If debhelper program has nothing to do, it will exit fast. Slow down should be minimal. The cost of startup of dh_* is an issue because they are Perl program. I'm sorry to tell you, but there are PII and PIII CPUs still (I have those) that I use for testing and the speed is an issue when iterating over package. You must *really* hate CDBS then. Its inefficient use of dh_* is what made me so excited about dh in the first place. Of those, for simple package, sometimes only these are needed (target: binary-arch): dh_testdir dh_testroot dh_installchangelogs dh_installdocs dh_installexamples dh_link dh_strip dh_compress dh_fixperms dh_installdeb dh_shlibdeps dh_gencontrol dh_md5sums dh_builddeb This is exactly what conventional packaging with debhelper is for. Since you want full control over sequence, use this method. Not full control. There is no need to write full blown debian/rules when it could simply read: # Makefile debian/rules %: dh $@ And the commads not used listed in: debian/dh_norun This would be both simple, elegant and maintenence friendly. It might be elegant, but it's no way simple. You add another file, which is rather unexpected dh interface detail for contributors to learn. In general, there is nothing wrong with your wish to skip pointless dh_* calls due to performance reasons. I just do not believe debian/dh_norun is the way to go. Disclaimer: I'm not maintainer. -- Modestas Vainius modes...@vainius.eu signature.asc Description: This is a digitally signed message part.
Bug#560423: debhelper: Add debian/dh_norun file to list commands not to run (ver 8?)
Modestas Vainius modes...@vainius.eu writes: I don't understand why you want another file for this. Sequence setup naturally *belongs* to debian/rules, it should be visible from there. If dh(1) is in use, there are not many sequences in debian/rules. The point of moving to single dh(1) is to move away from hand maintained and possibly long debian/rules file. There is a difference in maintaining packages and learning packages. The dh(1) addresses the scalability to being able to maintain numerous packages with ease. Starters usually do it by hand with traditional debhelper (I did). You end up writing copies of debian/rules files; over and over. They are mostly the same and contain duplicate code that are better handed to dh(1), thus simplifying the maintenance. As explained, compare this to traditional debian/rules file: There is no need to write full blown debian/rules when it could simply read: # Makefile debian/rules %: dh $@ And the commads not used listed in: debian/dh_norun This would be both simple, elegant and maintenence friendly. It might be elegant, but it's no way simple. You add another file, which is rather unexpected dh interface detail for contributors to learn. Maintaining separate files under debian/* directory is much better way than a monolithic debian/rules. I understand that it may look like more clear to have single file to do it all: it cerainly is easy for a new maintainer to study and undeerstand. But as experience has shown, the separate files approach: * is scalable * can be versioned (version control) and modified independently. Thus bits and pieces can be even shared between projects. * they separate certain logical structures to their own spots. This unifies usage accross all packages. * they are easier to process with external tools like maintainer scripts. You can't edit or extract content from a monlolithic debian/rules in any straight forward manner. Examples of decoupling can be witnessed everywhere. The exim4 package is a fine example how the maintenance in separate files makes upgrades a breeze because you can merge changes file-by-file basis instead of having to have to deal with monolithic changes in one file. The RedHat /etc/profile.d/ is another fine example Various others, lie /etc/apt/conf.d/ directories. Joey has presented ways how the makefile could be modified to prevent running extra dh_calls in Bug#560421, but as a generic solution it would be more elegant to stadrdize to a single file. Otherwise everyone cook their own overrides in debian/rules however they happen to write them (not all are GNUMakefile experts). debian/dh_norun The improved solution would be for the debhelper to heuristically guess automatically which rules need not be run according to files in debian/ directory -- thus making this almost transparent. But we know programs cannot make perfect guesses, so there would always be a need to specify which dh_* calls are not needed. A debian/dh_norun would be a generalized solution. Jari -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#560423: debhelper: Add debian/dh_norun file to list commands not to run (ver 8?)
Package: debhelper Version: 7.4.3 Severity: wishlist The debhelper 7 is good and in some ways excessive. To build a simple package, it churns over all possible commands (see below). SUGGESTION Please add support for file debian/dh_norun The file would list those commands that do not need to be run for the current package. Commands would be listed one at a line. Comments would be allowed with #. This would speed up packaging in low end machines. List of commands run to make the package: dh_installdocs dh_installchangelogs dh_installexamples dh_installman dh_installcatalogs dh_installcron dh_installdebconf dh_installemacsen dh_installifupdown dh_installinfo dh_pysupport dh_installinit dh_installmenu dh_installmime dh_installmodules dh_installlogcheck dh_installlogrotate dh_installpam dh_installppp dh_installudev dh_installwm dh_installxfonts dh_bugfiles dh_lintian dh_gconf dh_icons dh_perl dh_usrlocal dh_link dh_compress dh_fixperms dh_strip dh_makeshlibs dh_shlibdeps dh_installdeb dh_gencontrol dh_md5sums dh_builddeb Of those, for simple package, sometimes only these are needed (target: binary-arch): dh_testdir dh_testroot dh_installchangelogs dh_installdocs dh_installexamples dh_link dh_strip dh_compress dh_fixperms dh_installdeb dh_shlibdeps dh_gencontrol dh_md5sums dh_builddeb -- System Information: Debian Release: squeeze/sid APT prefers testing APT policy: (990, 'testing'), (500, 'unstable'), (1, 'experimental') Architecture: amd64 (x86_64) Kernel: Linux 2.6.30-2-amd64 (SMP w/2 CPU cores) Locale: LANG=en_DK.UTF-8, LC_CTYPE=en_DK.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Versions of packages debhelper depends on: ii binutils 2.20-4 The GNU assembler, linker and bina ii dpkg-dev 1.15.5.4 Debian package development tools ii file 5.03-3 Determines file type using magic ii html2text 1.3.2a-14 advanced HTML to text converter ii man-db2.5.6-4on-line manual pager ii perl 5.10.1-8 Larry Wall's Practical Extraction ii perl-base 5.10.1-8 minimal Perl system ii po-debconf1.0.16 tool for managing templates file t debhelper recommends no packages. Versions of packages debhelper suggests: ii dh-make 0.50 tool that converts source archives -- no debconf information -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
