[Resending to actual emacsen-common bug, not to ancient devscripts-el bug.
Sorry for the noise ]
On Sun, Feb 02, 2014 at 11:22:09AM +0000, Julian Gilbey wrote:
>
> Hi Rob,
>
> Please do have a look at this one - it's messy :-/
>
> On Fri, Jan 31, 2014 at 09:57:08PM +0900, Tatsuya Kinoshita wrote:
> > On January 31, 2014 at 11:33AM +0100, anbe (at debian.org) wrote:
> > > Install emacsen-common for emacs23
> > [...]
> > > install/devscripts-el: Handling emacs23, logged in /tmp/elc_tCbg9i.log
> > > ERROR: install script from devscripts-el package failed
> > [...]
> > > devscripts.el:19:1:Error: Cannot open load file: mcharset
> >
> > It seems apel's mcharset.elc is not found when byte-compiling.
> >
> > The devscripts-el package depends on apel, so install/devscripts-el
> > called from devscripts-el.postinst succeed, but install/devscripts-el
> > called from emacsen-common or emacsen could fail as above.
>
> I've looked into this one. I've applied a temporary patch to
> devscripts-el to avoid compiling if apel is not compiled, but it seems
> to me that the source of the breakage is in emacsen-common's handling
> of dependencies.
>
> I thought I had figured out what was going on, but now I have no idea:
> I simply don't understand why emacs-install was attempting to
> byte-compile devscripts-el when it had been unpackage earlier on
> during the upgrade (and hence run the emacs-package-remove script).
> The only thing I can think is that maybe the wheezy version of
> emacsen-common's scripts didn't correctly remove the installed marker
> file.
Hi,
I have been wondering for some time if there is a simpler way to handle all
this sorting without using explicit tsort.
I have been playing for some time with a proof of concept of a perl-only
method to handle this, and tried today to put it into emacsen-common lib.pl.
I am attaching current incarnation of it, both as a plain lib.pl and as a
git diff. I have tested in my system and it seems to work well, but consider
it highly experimental code.
To get dependencies it relies in a single 'dpkg-query' call for all add-ons
to be processed. That dependencies are processed and info put into a
dependencies hash, used to sort after its contents by perl sort function.
To make things more compact and integrated with my tests I have merged some
tightly related functions into a single `generate_add_on_install_list'
function. In my current tests, `get_installed_add_on_packages' has been
changed to return a hash reference, and `generate_add_on_install_list'
changed to accept references to either hash or array.
If `emacsen-common' is one of the packages to be processed, it will always
be put first (new dependency should have the same result), then packages
having no other dependencies, and then packages depending on other of the
add-ons to be installed.
To enable debugging output, just set
export EMACSEN_COMMON_DEBUG=1
As said, this is to be considered highly experimental, but I hope it can be
useful. Please have mercy on my perl :-).
Regards,
--
Agustin
>From 12c1ce80f1a86ff27c77f3a7e81cb30761c72d7f Mon Sep 17 00:00:00 2001
From: Agustin Martin Domingo <[email protected]>
Date: Fri, 23 May 2014 16:59:35 +0200
Subject: [PATCH] lib.pl: First cut for using a perl-only based sorting
algorithm.
* Make more use of hashes.
* Use a single dpkg-query call for all add-ons to be installed.
* Use a perl-only sorting algorithm.
* Merged sone interrelated functions into a single function.
---
lib.pl | 185 +++++++++++++++++++++++++++++++----------------------------------
1 file changed, 88 insertions(+), 97 deletions(-)
mode change 100755 => 100644 lib.pl
diff --git a/lib.pl b/lib.pl
old mode 100755
new mode 100644
index f0aa670..2d0097e
--- a/lib.pl
+++ b/lib.pl
@@ -3,6 +3,8 @@
use strict;
use Cwd;
+my $debug++ if $ENV{'EMACSEN_COMMON_DEBUG'};
+
# depends on: dpkg, tsort, perl
my $lib_dir = "/usr/lib/emacsen-common";
@@ -63,8 +65,7 @@ sub get_installed_add_on_packages
{
delete $ready_pkgs{$p} unless (-e "$::installed_package_state_dir/$p");
}
- my @result = keys %ready_pkgs;
- return \@result;
+ return \%ready_pkgs;
}
sub get_installed_flavors
@@ -73,110 +74,100 @@ sub get_installed_flavors
return @$flavors;
}
-sub get_package_status
-{
- my($pkg) = @_;
- my $status = `dpkg --status $pkg`;
- die 'emacsen-common: dpkg invocation failed' if($? != 0);
- $status =~ s/\n\s+//gmo; # handle any continuation lines...
- return $status;
-}
-
-sub filter_depends
-{
- my($depends_string, $installed_add_ons) = @_;
-
- # Filter out all the "noise" (version number dependencies, etc)
- # and handle or deps too, i.e. "Depends: foo, bar | baz"
- my @relevant_depends = split(/[,|]/, $depends_string);
- @relevant_depends = map { /\s*(\S+)/o; $1; } @relevant_depends;
-
- # Filter out all non-add-on packages.
- @relevant_depends = grep {
- my $candidate = $_;
- grep { $_ eq $candidate } @$installed_add_ons;
- } @relevant_depends;
-
- return @relevant_depends;
-}
+# ------------------------------------------------------------
+sub generate_add_on_install_list {
+# ------------------------------------------------------------
+# generate_add_on_install_list_new \@packages_to_sort
+# generate_add_on_install_list_new \%packages_to_sort
+# ------------------------------------------------------------
+ my $packages_to_sort = shift;
+ my $installed_add_ons = get_installed_add_on_packages;
+ my %depends_hash = ();
+
+ return unless $packages_to_sort;
+
+ # emacsen-install calls this function as the output of
+ # get_installed_add_on_packages, which has been changed to an ARRAY
+ # reference. Convert to an array reference if so.
+ if ( ref($packages_to_sort) eq 'HASH' ){
+ $packages_to_sort = [sort keys %$packages_to_sort];
+ }
-sub generate_relevant_tsort_dependencies_internals
-{
- my($pkglist, $installed_add_ons, $progress_hash) = @_;
-
- # print "GRD: " . join(" ", @$pkglist) . "\n";
-
- my $pkg = shift @$pkglist;
-
- if(!$pkg || $$progress_hash{$pkg}) {
- return ();
- } else {
- my $status = get_package_status($pkg);
- $status =~ /^Depends:\s+(.*)/mo;
- my $depends = $1; $depends = "" if ! $depends;
- my @relevant_depends = filter_depends($depends, $installed_add_ons);
- my $newpkglist = [@$pkglist, @relevant_depends];
-
- $$progress_hash{$pkg} = 1;
-
- # pkg is in twice so we don't have to worry about package with no
- # relevant dependencies. tsort can't handle that.
- my @tsort_strings = "$pkg $pkg\n";
- map { push @tsort_strings, "$_ $pkg\n"; } @relevant_depends;
-
- return (@tsort_strings,
- generate_relevant_tsort_dependencies_internals($newpkglist,
- $installed_add_ons,
- $progress_hash));
+ my $packages_to_sort_string = join(' ', @$packages_to_sort);
+ my $dpkg_query_output = `dpkg-query -W -f='package:\${Package}, \${Depends}\n' $packages_to_sort_string`;
+ die 'emacsen-common: dpkg-query invocation failed' unless ($? == 0);
+
+ if ( $debug ){
+ print "------------------------------------------------------------------------------\n";
+ print "Packages to sort:\n$packages_to_sort_string\n";
+ print "-------------------------------------------------------------\n";
+ print "dpkg-query output:\n---\n$dpkg_query_output---\n";
+ print "-------------------------------------------------------------\n";
+ print "Installed add-ons:\n", join(', ',sort keys %{$installed_add_ons}), "\n";
+ print "-------------------------------------------------------------\n";
}
-}
-sub generate_relevant_tsort_dependencies
-{
- my($pkglist, $installed_add_ons, $progress_hash) = @_;
- # Make a copy because we're going to mangle it.
- my @listcopy = @$pkglist;
- shift @_;
- return(generate_relevant_tsort_dependencies_internals(\@listcopy, @_));
-}
+ foreach my $dpkg_query_line ( split("\n", $dpkg_query_output) ){
+ my @package_depends = split(/[,|]/, $dpkg_query_line);
+ my $package = shift @package_depends;
+ # Remove consistency string or ignore line if missing.
+ next unless $package =~ s/^package://;
-sub reorder_add_on_packages
-{
- my($pkglist, $installed_add_ons) = @_;
- my @depends = generate_relevant_tsort_dependencies($pkglist,
- $installed_add_ons,
- {});
- my $pid = open(TSORT, "-|");
- die "Couldn't fork for tsort: $!" unless defined($pid);
-
- # What a strange idiom...
- if($pid == 0) {
- my $sub_pid = open(IN, "|-");
- die "Couldn't sub-fork for tsort: $!" unless defined($sub_pid);
- if($sub_pid == 0) {
- exec 'tsort' or die "Couldn't run tsort: $!";
+ # Filter out all the "noise" (version number dependencies, etc)
+ @package_depends = map { /\s*(\S+)/o; $1; } @package_depends;
+
+ my %tmp_deps = ();
+ foreach my $dependency ( @package_depends ){
+ # dpkg-query regexp above will result in empty dependency for
+ # packages with no dependencies at all. Discard if so.
+ next unless $dependency;
+
+ # Filter out dependencies on non-add-on packages.
+ next unless ( defined $installed_add_ons->{"$dependency"} );
+
+ # Populate the temporal dependencies hash for this package
+ $tmp_deps{$dependency}++;
+ }
+
+ if ( %tmp_deps ){
+ foreach my $dependency ( sort keys %tmp_deps ){
+ # Populate the real add-ons dependencies hash
+ print "* Setting $package dependency on $dependency.\n" if $debug;
+ $depends_hash{'deps'}{$package}{$dependency}++;
+ }
+ } else {
+ # $package does not depend on any add-on
+ print "* Setting $package to not depend on add-ons.\n" if $debug;
+ $depends_hash{'nodeps'}{$package}++;
}
- print IN @depends;
- exit 0;
}
- my @ordered_pkgs = <TSORT>;
- chomp @ordered_pkgs;
- return @ordered_pkgs
-}
-sub generate_add_on_install_list
-{
- my($packages_to_sort) = @_;
- my @sorted_pkgs = reorder_add_on_packages($packages_to_sort,
- get_installed_add_on_packages());
- return(@sorted_pkgs);
+ # Sort add-on packages to byte-compile
+ my @sorted_add_ons =sort {
+ # Sort emacsen-common first if is to be byte-compiled
+ $b =~ m/emacsen-common/ <=> $a =~ m/emacsen-common/
+ ||
+ # Then sort add-ons without dependencies first
+ defined $depends_hash{'nodeps'}{$b} cmp defined $depends_hash{'nodeps'}{$a}
+ ||
+ # Then sort add-ons depending on another add-on after it
+ defined $depends_hash{'deps'}{$a}{$b} cmp defined $depends_hash{'deps'}{$b}{$a}
+ ||
+ # Should be none, but sort circular dependencies alphabetically.
+ $a cmp $b;
+ } @$packages_to_sort;
+
+ # More debugging code
+ print "--------------------------------------------\n",
+ "Sorted packages:\n",
+ join(', ',@sorted_add_ons),
+ "\n",
+ "--------------------------------------------\n"
+ if $debug;
+
+ return @sorted_add_ons;
}
-# Test code
-# my @input_packages = <STDIN>;
-# my @result = generate_add_on_install_list(@input_packages);
-# print " " . join("\n ", @result);
-
# To make require happy...
1;
--
2.0.0.rc4
#!/usr/bin/perl -w
use strict;
use Cwd;
my $debug++ if $ENV{'EMACSEN_COMMON_DEBUG'};
# depends on: dpkg, tsort, perl
my $lib_dir = "/usr/lib/emacsen-common";
my $var_dir = "/var/lib/emacsen-common";
$::installed_package_state_dir = "${var_dir}/state/package/installed";
$::installed_flavor_state_dir = "${var_dir}/state/flavor/installed";
sub ex
{
my(@cmd) = @_;
if(system(@cmd) != 0)
{
die join(" ", @cmd) . " failed";
}
}
sub glob_in_dir
{
my ($dir, $pattern) = @_;
my $oldir = getcwd;
chdir($dir) or die "chdir $dir: $!";
my @files = glob("*[!~]");
chdir($oldir);
return \@files;
}
sub validate_add_on_pkg
{
my ($pkg, $script, $old_invocation_style) = @_;
if($old_invocation_style)
{
if(-e "$lib_dir/packages/compat/$pkg")
{
print STDERR "ERROR: $pkg is broken - called $script as an old-style add-on, but has compat file.\n";
#exit(1);
}
}
else # New invocation style.
{
unless(-e "$lib_dir/packages/compat/$pkg")
{
print STDERR "ERROR: $pkg is broken - called $script as a new-style add-on, but has no compat file.\n";
#exit(1);
}
}
}
sub get_installed_add_on_packages
{
# Return all of the old format packages, plus all of the new-format
# packages that are ready (i.e. have a state/installed file). In
# this case ready means ready for compilation.
my $all_pkgs = glob_in_dir("$lib_dir/packages/install", '*[!~]');
my $new_format_pkgs = glob_in_dir("$lib_dir/packages/compat", '*[!~]');
my %ready_pkgs = map { $_ => 1 } @$all_pkgs;
for my $p (@$new_format_pkgs)
{
delete $ready_pkgs{$p} unless (-e "$::installed_package_state_dir/$p");
}
return \%ready_pkgs;
}
sub get_installed_flavors
{
my $flavors = glob_in_dir($::installed_flavor_state_dir, '*[!~]');
return @$flavors;
}
# ------------------------------------------------------------
sub generate_add_on_install_list {
# ------------------------------------------------------------
# generate_add_on_install_list_new \@packages_to_sort
# generate_add_on_install_list_new \%packages_to_sort
# ------------------------------------------------------------
my $packages_to_sort = shift;
my $installed_add_ons = get_installed_add_on_packages;
my %depends_hash = ();
return unless $packages_to_sort;
# emacsen-install calls this function as the output of
# get_installed_add_on_packages, which has been changed to an ARRAY
# reference. Convert to an array reference if so.
if ( ref($packages_to_sort) eq 'HASH' ){
$packages_to_sort = [sort keys %$packages_to_sort];
}
my $packages_to_sort_string = join(' ', @$packages_to_sort);
my $dpkg_query_output = `dpkg-query -W -f='package:\${Package}, \${Depends}\n' $packages_to_sort_string`;
die 'emacsen-common: dpkg-query invocation failed' unless ($? == 0);
if ( $debug ){
print "------------------------------------------------------------------------------\n";
print "Packages to sort:\n$packages_to_sort_string\n";
print "-------------------------------------------------------------\n";
print "dpkg-query output:\n---\n$dpkg_query_output---\n";
print "-------------------------------------------------------------\n";
print "Installed add-ons:\n", join(', ',sort keys %{$installed_add_ons}), "\n";
print "-------------------------------------------------------------\n";
}
foreach my $dpkg_query_line ( split("\n", $dpkg_query_output) ){
my @package_depends = split(/[,|]/, $dpkg_query_line);
my $package = shift @package_depends;
# Remove consistency string or ignore line if missing.
next unless $package =~ s/^package://;
# Filter out all the "noise" (version number dependencies, etc)
@package_depends = map { /\s*(\S+)/o; $1; } @package_depends;
my %tmp_deps = ();
foreach my $dependency ( @package_depends ){
# dpkg-query regexp above will result in empty dependency for
# packages with no dependencies at all. Discard if so.
next unless $dependency;
# Filter out dependencies on non-add-on packages.
next unless ( defined $installed_add_ons->{"$dependency"} );
# Populate the temporal dependencies hash for this package
$tmp_deps{$dependency}++;
}
if ( %tmp_deps ){
foreach my $dependency ( sort keys %tmp_deps ){
# Populate the real add-ons dependencies hash
print "* Setting $package dependency on $dependency.\n" if $debug;
$depends_hash{'deps'}{$package}{$dependency}++;
}
} else {
# $package does not depend on any add-on
print "* Setting $package to not depend on add-ons.\n" if $debug;
$depends_hash{'nodeps'}{$package}++;
}
}
# Sort add-on packages to byte-compile
my @sorted_add_ons =sort {
# Sort emacsen-common first if is to be byte-compiled
$b =~ m/emacsen-common/ <=> $a =~ m/emacsen-common/
||
# Then sort add-ons without dependencies first
defined $depends_hash{'nodeps'}{$b} cmp defined $depends_hash{'nodeps'}{$a}
||
# Then sort add-ons depending on another add-on after it
defined $depends_hash{'deps'}{$a}{$b} cmp defined $depends_hash{'deps'}{$b}{$a}
||
# Should be none, but sort circular dependencies alphabetically.
$a cmp $b;
} @$packages_to_sort;
# More debugging code
print "--------------------------------------------\n",
"Sorted packages:\n",
join(', ',@sorted_add_ons),
"\n",
"--------------------------------------------\n"
if $debug;
return @sorted_add_ons;
}
# To make require happy...
1;
