from:"Vincent Ladeuil"

Bug#677250: Acknowledgement (xz files with multiple blocks are not recognized)

2012-07-11 Thread Vincent Ladeuil

 Joey Hess jo...@debian.org writes:

 Ok, I've applied this in git.

Thanks.

 I'll release it later, so send more patches if needed.

Sorry for the delay, I was waiting for xz-utils maintainer to commit the
patch I was testing.

This has now occurred and the code will be available as part of xz-utils
5.1.2alpha (whenever it's released, hopefully soon according to the xz
maintainer).

But, the option name is now --block-list instead of --block-split.

Therefore, the attached additional patch change the option name and
activates it.

If the option is not available (because the installed xz is too old), it
will fail and fallback to the usual guessing.

pristine-xz will ultimately fails but without a newer xz there is
nothing we can do anyway.

I've tested the attached patch against a newer xz and it works as
expected.

Vincent

P.S.: I think I've got the indent style right this time ;)


   modified  pristine-xz


=== modified file 'pristine-xz'
--- pristine-xz	2012-06-19 13:28:41 +
+++ pristine-xz	2012-07-11 08:32:02 +
@@ -168,7 +168,7 @@
 sub predict_xz_args {
 	my ($xz) = @_;
 	my $presets = undef;
-	my $block_split = undef;
+	my $block_list = undef;
 	my $blocks = $xz-{blocks};
 	if (scalar(@$blocks)) {
 		# There is at least one block. We assume the same compression
@@ -199,7 +199,7 @@
 		}
 		if (scalar(@$blocks)  1) {
 			# Gather the block uncompressed sizes
-			$block_split = join(',', map {$_-{uncompressed_size}}
+			$block_list = join(',', map {$_-{uncompressed_size}}
 	@$blocks);
 		}
 	}
@@ -215,11 +215,8 @@
 
 	my $possible_args = [];
 	my $common = [--check=$check_kwd, -z];
-	# FIXME: --block-split is not (yet) part of xz-utils upstream
-	if (0  defined($block_split)) {
-		# We put the block list in front of the parameters to make it
-		# easier to filter it later.
-		unshift @$common, --block-split=$block_split;
+	if (defined($block_list)) {
+		unshift @$common, --block-list=$block_list;
 	}
 	foreach my $preset (@$presets) {
 		push @$possible_args, [@$common, -$preset];

Bug#677250: Acknowledgement (xz files with multiple blocks are not recognized)

2012-06-18 Thread Vincent Ladeuil

The attached patch does a better job than my first shot: it relies on
'xz -lvv --robot' do get as much information as possible from the file
itself.

That reduces the guessing to --extreme and one of (3/4) or (5/6) if
their associated dict size is recognized.

I've disabled the multi-block (--block-split) parameter as xz-utils does
not provide it yet (but I've tested a patch from upstream to that effect
and will follow up when it becomes available).

The attached patch still reduces the number of xz tries and fallback to
the previous guessing is something goes wrong during the file
reading.

I've successfully used it (*with* --block-split) for ~140 multi-block files.


   modified  pristine-xz


=== modified file 'pristine-xz'
--- pristine-xz	2012-06-12 15:45:07 +
+++ pristine-xz	2012-06-18 14:35:58 +
@@ -101,7 +101,147 @@
 	print STDERRpristine-xz [-vdkt] genxz delta file\n;
 }
 
+sub assign_fields {
+my ($hash, $labels, $fields) = @_ ;
+@$hash{@$labels} = @$fields[1..scalar(@$labels)] ;
+}
+
+sub scan_xz_lvv_robot {
+my ($filename) = @_ ;
+# We need at least version 5.0 to get a proper '-lvv --robot' implemented
+my $cmd = xz -lvv --robot $filename ;
+my $ret = open (my $in, $cmd |) || die $cmd failed: $!;
+my %xz = (file = {}, stream = {}, blocks = [],
+  summary = {}, totals = {}) ;
+my (%file, %stream, @blocks, %summary, %totals) ;
+my @file_labels = qw{nb_streams nb_blocks compressed uncompressed ratio
+ checks padding_size} ;
+my @stream_labels =
+qw{stream_num nb_blocks compressed_offset uncompressed_offset
+   compressed_size uncompressed_size ratio check_name padding_size} ;
+my @block_labels =
+qw{stream_num block_in_stream block_in_file compressed_offset
+   uncompressed_offset compressed_size uncompressed_size ratio
+   check_name check_value header_size size_present_flags
+   actual_compressed_size uncompress_memory filter_chain} ;
+my @summary_labels = qw{uncompressed_memory size_in_blocks} ;
+my @totals_labels =
+qw{nb_streams nb_blocks compressed_size uncompressed_size ratio
+   check_names padding_size nb_files uncompressed_memory
+   size_in_blocks} ;
+
+while (my $line = $in) {
+chomp $line ;
+my @fields = split(/\t/, $line) ;
+if ($fields[0] eq 'name') {
+next ;
+}
+if ($fields[0] eq 'file') {
+assign_fields($xz{file}, \@file_labels, \@fields) ;
+next ;
+}
+if ($fields[0] eq 'stream') {
+assign_fields($xz{stream}, \@stream_labels, \@fields) ;
+next ;
+}
+if ($fields[0] eq 'block') {
+my %block ;
+assign_fields(\%block, \@block_labels, \@fields) ;
+push @{$xz{blocks}}, \%block ;
+next ;
+}
+if ($fields[0] eq 'summary') {
+assign_fields($xz{summary}, \@summary_labels, \@fields) ;
+next ;
+}
+if ($fields[0] eq 'totals') {
+assign_fields($xz{totals}, \@totals_labels, \@fields) ;
+next ;
+}
+}
+close $in ;
+return \%xz ;
+}
+
+sub predict_xz_args {
+my ($xz) = @_ ;
+my $presets = undef ;
+my $block_split = undef ;
+my $blocks = $xz-{blocks} ;
+if (scalar(@$blocks)) {
+# There is at least one block. We assume the same compression
+# level for all blocks
+my $block = $blocks-[0] ;
+my @filters = split(/,/, $block-{filter_chain}) ;
+if (scalar(@filters) != 1 || $filters[0] !~ /^--lzma2=/) {
+die Only LZMA2 is supported ;
+}
+# Deduce the presets from the dict size
+if ($filters[0] =~ /--lzma2=dict=(.*)/) {
+my $dict_size = $1 ;
+my %lzma2_presets_from_dict_size_of =
+(
+ '256KiB' = ['0'],
+ '1Mib'   = ['1'],
+ '2MiB'   = ['2'],
+ '4MiB'   = ['4', '3'],
+ # Put 6 before 5 as it's the default and is more likely to
+ # be right
+ '8MiB'   = ['6', '5'],
+ '16MiB'  = ['7'],
+ '32MiB'  = ['8'],
+ '64MiB'  = ['9'],
+) ;
+$presets = $lzma2_presets_from_dict_size_of{$dict_size} ;
+die Unkown dict size: $dict_size\n if (!defined($presets)) ;
+}
+if (scalar(@$blocks)  1) {
+# Gather the block uncompressed sizes
+$block_split = join(',', map {$_-{uncompressed_size}} @$blocks) ;
+}
+}
+# FIXME: none is missing
+my %check_kwd_of = 
+(CRC32 = 'crc32',
+ CRC64 = 'crc64',
+ 'SHA-256' = 'sha256',
+) ;
+my $check_name = $xz-{stream}-{check_name} ;
+my

Bug#677250: Acknowledgement (xz files with multiple blocks are not recognized)

2012-06-19 Thread Vincent Ladeuil

 Joey Hess jo...@debian.org writes:

 +if (scalar(@filters) != 1 || $filters[0] !~ /^--lzma2=/) {
 +die Only LZMA2 is supported ;

 +die Unkown dict size: $dict_size\n if 
(!defined($presets)) ;

 +my $check_kwd = $check_kwd_of{$check_name} ;
 +die Unknown xz check: $check_name\n if (!defined($check_kwd)) ;

 Could it just fall back to the old guessing behavior instead of dying?

It does, in reproducexz:

eval {
$possible_args = readxz($orig);
} ;
# If we get an error we fallback to guessing, otherwise, we should
# succeed with one of the proposed combinations
if (! $@) {

 +foreach my $program (@supported_xz_programs) {
 +# try to guess the xz arguments that are needed
 foreach my $args (predictxzargs($possible_levels, $program)) {
 -testvariant($orig, $tmpin, $program, @$args)
 - return $program, @$args;
 +testvariant($orig, $tmpin, $program, @$args)
 + return $program, @$args;

 Your editor is replacing tabs with spaces..

I can fix that.


   modified  pristine-xz


=== modified file 'pristine-xz'
--- pristine-xz	2012-06-12 15:45:07 +
+++ pristine-xz	2012-06-19 07:47:17 +
@@ -101,6 +101,133 @@
 	print STDERRpristine-xz [-vdkt] genxz delta file\n;
 }
 
+sub assign_fields {
+	my ($hash, $labels, $fields) = @_ ;
+	@$hash{@$labels} = @$fields[1..scalar(@$labels)] ;
+}
+
+sub scan_xz_lvv_robot {
+	my ($filename) = @_ ;
+	# We need at least version 5.0 to get a proper '-lvv --robot'
+	# implemented
+	my $cmd = xz -lvv --robot $filename ;
+	my $ret = open (my $in, $cmd |) || die $cmd failed: $!;
+	my %xz = (file = {}, stream = {}, blocks = [],
+		  summary = {}, totals = {}) ;
+	my (%file, %stream, @blocks, %summary, %totals) ;
+	my @file_labels = qw{nb_streams nb_blocks compressed uncompressed
+			 ratio checks padding_size} ;
+	my @stream_labels =
+		qw{stream_num nb_blocks compressed_offset uncompressed_offset
+		   compressed_size uncompressed_size ratio check_name
+		   padding_size};
+	my @block_labels = 
+		qw{stream_num block_in_stream block_in_file compressed_offset
+		   uncompressed_offset compressed_size uncompressed_size ratio
+		   check_name check_value header_size size_present_flags
+		   actual_compressed_size uncompress_memory filter_chain} ;
+	my @summary_labels = qw{uncompressed_memory size_in_blocks} ;
+	my @totals_labels =
+		qw{nb_streams nb_blocks compressed_size uncompressed_size ratio
+		   check_names padding_size nb_files uncompressed_memory
+		   size_in_blocks} ;
+
+	while (my $line = $in) {
+		chomp $line ;
+		my @fields = split(/\t/, $line) ;
+		if ($fields[0] eq 'name') {
+			next ;
+		}
+		if ($fields[0] eq 'file') {
+			assign_fields($xz{file}, \@file_labels, \@fields) ;
+			next ;
+		}
+		if ($fields[0] eq 'stream') {
+			assign_fields($xz{stream}, \@stream_labels, \@fields) ;
+			next ;
+		}
+		if ($fields[0] eq 'block') {
+			my %block ;
+			assign_fields(\%block, \@block_labels, \@fields) ;
+			push @{$xz{blocks}}, \%block ;
+			next ;
+		}
+		if ($fields[0] eq 'summary') {
+			assign_fields($xz{summary}, \@summary_labels, \@fields);
+			next ;
+		}
+		if ($fields[0] eq 'totals') {
+			assign_fields($xz{totals}, \@totals_labels, \@fields) ;
+			next ;
+		}
+	}
+	close $in ;
+	return \%xz ;
+}
+
+sub predict_xz_args {
+	my ($xz) = @_ ;
+	my $presets = undef ;
+	my $block_split = undef ;
+	my $blocks = $xz-{blocks} ;
+	if (scalar(@$blocks)) {
+		# There is at least one block. We assume the same compression
+		# level for all blocks
+		my $block = $blocks-[0] ;
+		my @filters = split(/,/, $block-{filter_chain}) ;
+		if (scalar(@filters) != 1 || $filters[0] !~ /^--lzma2=/) {
+			die Only LZMA2 is supported ;
+		}
+		# Deduce the presets from the dict size
+		if ($filters[0] =~ /--lzma2=dict=(.*)/) {
+			my $dict_size = $1 ;
+			my %lzma2_presets_from_dict_size_of =
+('256KiB' = ['0'],
+ '1Mib'   = ['1'],
+ '2MiB'   = ['2'],
+ '4MiB'   = ['4', '3'],
+ # Put 6 before 5 as it's the default and is
+ # more likely to be right
+ '8MiB'   = ['6', '5'],
+ '16MiB'  = ['7'],
+ '32MiB'  = ['8'],
+ '64MiB'  = ['9'],
+) ;
+			$presets = $lzma2_presets_from_dict_size_of{$dict_size};
+			die Unkown dict size: $dict_size\n
+if (!defined($presets)) ;
+		}
+		if (scalar(@$blocks)  1) {
+			# Gather the block uncompressed sizes
+			$block_split = join(',', map {$_-{uncompressed_size}}
+	@$blocks) ;
+		}
+	}
+	# FIXME: none is missing
+	my %check_kwd_of = 
+		(CRC32 = 'crc32',
+		 CRC64 = 'crc64',
+		 'SHA-256' = 'sha256',
+		) ;
+	my $check_name = $xz-{stream}-{check_name} ;
+	my $check_kwd = $check_kwd_of{$check_name} ;
+	die Unknown xz check: $check_name\n if

Bug#677250: Acknowledgement (xz files with multiple blocks are not recognized)

2012-07-12 Thread Vincent Ladeuil

 Joey Hess jo...@debian.org writes:

 Vincent Ladeuil wrote:
 But, the option name is now --block-list instead of --block-split.

 Won't use of this option cause the paranoia check to fail when
 reproducing the xz file, unless it's whitelisted there?

Ouch. Dang, I thought my tests covered that, obviously not :-/

Fixed (and also added --check={crc64,none} while I was there).

 (Also, changelog,

Done in the attached patch ?

 git commit etc would be useful.)

Not sure what you mean here, I'm using bzr (based on
bzr+ssh://bazaar.launchpad.net/+branch/pristine-tar/) not git :-}

https://code.launchpad.net/~vcs-imports/pristine-tar/trunk is an import
from git://git.kitenet.net/pristine-tar/ though so there should be a
way...

In the mean time, I've pushed my branch at
bzr+ssh://bazaar.launchpad.net/~vila/pristine-tar/677250-parse-xz or
https://code.launchpad.net/~vila/pristine-tar/677250-parse-xz/ for human
readers.

I'm also quite a noob at packaging but I'm eager to learn...

Vincent

P.S.: I'll be in vacations for a week starting tonight.


   modified  debian/changelog
   modified  pristine-xz


=== modified file 'debian/changelog'
--- debian/changelog	2012-06-19 13:28:41 +
+++ debian/changelog	2012-07-12 06:52:07 +
@@ -1,9 +1,8 @@
 pristine-tar (1.26) UNRELEASED; urgency=low
 
   * pristine-xz: Use xz --robot to extract information from xz files,
-avoiding the need to do more expensive guessing. Will also later
-allow supporting files needing --block-split (once upstream xz
-provides that option; a patch has been submitted).
+avoiding the need to do more expensive guessing. Support --block-list
+when available to reproduce multi blocks files.
 Closes: #677250 Thanks, Vincent Ladeuil
 
  -- Joey Hess jo...@debian.org  Tue, 19 Jun 2012 09:24:30 -0400

=== modified file 'pristine-xz'
--- pristine-xz	2012-06-19 13:28:41 +
+++ pristine-xz	2012-07-12 07:33:57 +
@@ -168,7 +168,7 @@
 sub predict_xz_args {
 	my ($xz) = @_;
 	my $presets = undef;
-	my $block_split = undef;
+	my $block_list = undef;
 	my $blocks = $xz-{blocks};
 	if (scalar(@$blocks)) {
 		# There is at least one block. We assume the same compression
@@ -199,13 +199,13 @@
 		}
 		if (scalar(@$blocks)  1) {
 			# Gather the block uncompressed sizes
-			$block_split = join(',', map {$_-{uncompressed_size}}
+			$block_list = join(',', map {$_-{uncompressed_size}}
 	@$blocks);
 		}
 	}
-	# FIXME: none is missing
 	my %check_kwd_of = 
-		(CRC32 = 'crc32',
+		(None = 'none',
+		 CRC32 = 'crc32',
 		 CRC64 = 'crc64',
 		 'SHA-256' = 'sha256',
 		);
@@ -215,11 +215,8 @@
 
 	my $possible_args = [];
 	my $common = [--check=$check_kwd, -z];
-	# FIXME: --block-split is not (yet) part of xz-utils upstream
-	if (0  defined($block_split)) {
-		# We put the block list in front of the parameters to make it
-		# easier to filter it later.
-		unshift @$common, --block-split=$block_split;
+	if (defined($block_list)) {
+		unshift @$common, --block-list=$block_list;
 	}
 	foreach my $preset (@$presets) {
 		push @$possible_args, [@$common, -$preset];
@@ -350,8 +347,11 @@
 
 		next if $param=~/^(-[0-9]e?)$/;
 		next if $param eq '-z';
+		next if $param eq '--check=none';
 		next if $param eq '--check=crc32';
+		next if $param eq '--check=crc64';
 		next if $param eq '--check=sha256';
+		next if $param=~/^(--block-list=[0-9,]+)$/;
 		die paranoia check failed on params from delta (@params);
 	}
 	@params=split(' ', $delta-{params});

Bug#677250: Acknowledgement (xz files with multiple blocks are not recognized)

2012-08-10 Thread Vincent Ladeuil

I finally managed to get this fix deployed on
http://package-import.ubuntu.com/status/ and wanted to report success :)

The ~240 packages affected have been unblocked leading to ~3000 releases
imported successfully. Not all of those releases involved the offended
xz files but all packages involved such a file at least once in their
recent history. This nevertheless represents a significant coverage.

As far as *this* bug is concerned though, there is nothing left to fix ;)

Thanks for your help,

   Vincent


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#677241: xz files built with -0[e] and/or --check=sha256 not recognized

2012-06-12 Thread Vincent Ladeuil

Package: pristine-tar
Version: 1.24


Relevant .xz files:

https://launchpad.net/debian/sid/+source/haskell-ncurses/0.2.1-1/+files/haskell-ncurses_0.2.1.orig.tar.xz

https://launchpad.net/ubuntu/quantal/+source/evilvte/0.5.0-1/+files/evilvte_0.5.0.orig.tar.xz

https://launchpad.net/ubuntu/quantal/+source/evilvte/0.5.0-1/+files/evilvte_0.5.0-1.debian.tar.xz

https://launchpad.net/debian/sid/+source/evilvte/0.5.1-1/+files/evilvte_0.5.1.orig.tar.xz

https://launchpad.net/debian/sid/+source/evilvte/0.5.1-1/+files/evilvte_0.5.1-1.debian.tar.xz


The following patch fixes the issue:

   modified  pristine-xz


=== modified file 'pristine-xz'
--- pristine-xz 2012-04-01 16:21:10 +
+++ pristine-xz 2012-06-12 15:09:13 +
@@ -115,7 +115,7 @@
#
# So far in the wild only these levels have been seen.
# (Note that level 9 can use a lot of memory.)
-   my $possible_levels = [6, 9, 6e, 9e];
+   my $possible_levels = [6, 9, 0, 6e, 9e, 0e];
 
return ($possible_levels);
 }
@@ -127,7 +127,8 @@
foreach my $level (@$possible_levels) {
push @args, [-z, -$level];
push @args, [-z, -$level, --check=crc32];
-   }
+   push @args, [-z, -$level, --check=sha256];
+}
return @args;
 }
 
@@ -190,6 +191,7 @@
next if $param=~/^(-[0-9]e?)$/;
next if $param eq '-z';
next if $param eq '--check=crc32';
+   next if $param eq '--check=sha256';
die paranoia check failed on params from delta (@params);
}
@params=split(' ', $delta-{params});




-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#677250: xz files with multiple blocks are not recognized

2012-06-12 Thread Vincent Ladeuil

Package: pristine-tar
Version: 1.24

examples:

Many ( 100) kde 4.8.2 packages including:

https://launchpad.net/ubuntu/precise/+source/marble/4:4.8.2-0ubuntu1/+files/marble_4.8.2.orig.tar.xz

https://launchpad.net/ubuntu/precise/+source/kdesdk/4:4.8.2-0ubuntu1/+files/kdesdk_4.8.2.orig.tar.xz

https://launchpad.net/ubuntu/precise/+source/kde-workspace/4:4.8.2-0ubuntu1/+files/kde-workspace_4.8.2.orig.tar.xz


The same multiple blocks files can be found at
ftp://ftp.kde.org/pub/kde/stable/4.8.4/src/ (examples includes kdesk,
kalzium, oxygen-icons, etc).

The following script is a first shot at parsing the .xz files and
extracting the relevant parameters (compression level, check and
blocks).

It may be used to enhance the xz file recognition but doesn't provide a
solution to rebuild multiple blocks xz files.

#!/usr/bin/perl

# Dotted numbers in comments refer to http://tukaani.org/xz/xz-file-format.txt

use warnings;
use strict;
use constant DEBUG = 1 ;

use Fcntl qw{:seek} ;

use Data::Dumper ;

sub checked_read {
# $buf should be a reference so we can still use 'read' and its magic
# trick
my ($in, $buf, $nb_to_read, $msg) = @_ ;
my $nb_read = read($in, $$buf, $nb_to_read) ;
die $msg if ($nb_read != $nb_to_read) ;
}

# see 1.2 with slight adaptation since we read from a file handle instead of
# using a char buffer
sub read_multibyte_integer {
my ($in, $size_max) = @_ ;
return undef, 0 if ($size_max == 0) ;

if ($size_max  9) {
$size_max = 9 ;
}
my $buf ;
my $i = 1 ;
checked_read($in, \$buf, 1, Corrupted xz file\n) ;
$buf = unpack('C', $buf) ;
warn sprintf(buf: %d / %d, $buf, $buf  0x7f) if (DEBUG = 9) ;
my $num = $buf  0x7f ;

while ($buf  0x80) {
return (undef, 0) if ($i = $size_max) ;

checked_read($in, \$buf, 1, Corrupted xz file\n) ;
$buf = unpack('C', $buf) ;
warn sprintf(buf: %d / %d, $buf, $buf  0x7f) if (DEBUG = 9) ;
$i++ ;
return (undef, 0) if ($buf == 0x00) ;

$num |= ($buf  0x7f)  (($i - 1) * 7) ;
}
return ($num, $i) ;
}

sub decode_check_byte {
my ($byte) = @_ ;
my $check_key ;
# We use the xz --check values as values below, the Ids are used as keys.
my %known_check_of =
(
 0x00 = 'none',
 0x01 = 'crc32',
 0x04 = 'crc64',
 0x0A = 'sha256',
) ;
$check_key = $known_check_of{$byte} ;
 if (!defined($check_key)) {
 die sprintf([%02X] is an unknown xz check\n, $byte) ;
 }
return $check_key ;
}

sub decode_stream_flags {
my @flags = @_ ;
die Unknown flags used in first format byte\n if ($flags[0] != 0) ;
my $check = decode_check_byte($flags[1]) ;
return [$check] ;
}

sub parse_stream_header {
my ($in) = @_ ;
my ($buf) ;
checked_read($in, \$buf, 6, Invalid xz file\n) ;
my $magic = unpack('H12', $buf) ;
die Not an xz file\n if ($magic ne 'fd377a585a00') ;
checked_read($in, \$buf, 2, Corrupted xz file\n) ;
my @flags = unpack('CC', $buf) ;
my $flags = decode_stream_flags(@flags) ;
my $check = $flags-[0] ;
checked_read($in, \$buf, 4, Corrupted xz file\n) ;
my $CRC32 = unpack('V', $buf) ;
return [$check, $CRC32] ;
}



my %lzma2_preset_from_dict_size_of =
(
 0x0004 = ['0'],  # 256 KiB
 0x0010 = ['1'],  #   1 MiB
 0x0020 = ['2'],  #   2 MiB
 0x0040 = ['3', '4'], #   4 MiB
 0x0080 = ['5', '6'], #   8 MiB
 0x0100 = ['7'],  #  16 MiB
 0x0200 = ['8'],  #  32 MiB
 0x0400 = ['9'],  #  64 MiB
) ;

sub decode_lzma2_props {
my ($props) = @_ ;
my $bits = unpack('C', $props)  0x3f ;
# see 5.3.1
my $dict_size ;
if ($bits == 40) {
$dict_size = 0x ;
} else {
$dict_size = 2 | ($bits  1) ;
$dict_size = $bits / 2 + 11 ;
}
my $presets = $lzma2_preset_from_dict_size_of{$dict_size} ;
die Unkown dict size: $dict_size\n if (!defined($presets)) ;
return {id = 'lzma2', presets = $presets} ;
}

my %props_decoder_of =
(
 0x21 = \decode_lzma2_props,
) ;

sub decode_filter_props {
my ($id, $props) = @_ ;
my $decoder = $props_decoder_of{$id} ;
die $id is not a known filter\n if (!defined($decoder)) ;

return $decoder($props) ;
}

sub parse_block_header {
my ($in, $check) = @_ ;
my ($buf) ;
checked_read($in, \$buf, 1, Corrupted xz file\n) ;
my $size = unpack('C', $buf) ;
$size = ($size + 1) * 4 ; # see 3.1.1
checked_read($in, \$buf, 1, Corrupted xz file\n) ;
my $flags = unpack('C', $buf) ;
my $nb_filters = ($flags  0x03) + 1 ; # 2 bits used to encode 1-4 values
my $must_be_zero = $flags  0x3C ;
die Reversed block flags non-zero\n if ($must_be_zero != 0) ;
my $compressed_size_present = $flags  0x40 ;
die Not implemented\n if ($compressed_size_present != 0) ;
my $uncompressed_size_present = $flags  0x80 ;
die Not implemented\n if

Bug#787057: bzr: Racy tests

2015-05-28 Thread Vincent Ladeuil

Package: bzr
Version: 2.6.0+bzr6595-6ubuntu1
Severity: normal

Dear Maintainer,

bzr FTBFS as two tests became racy.

The attached debdiff disable these tests for both build and autopkgtest (a
bug has been filed upstream to fix them).


-- System Information:
Debian Release: jessie/sid
  APT prefers vivid-updates
  APT policy: (500, 'vivid-updates'), (500, 'vivid-security'), (500, 'vivid')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.19.0-18-generic (SMP w/8 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages bzr depends on:
ii  python-bzrlib  2.6.0+bzr6595-6ubuntu1
pn  python:any none

Versions of packages bzr recommends:
ii  python-gpgme  0.3-1

Versions of packages bzr suggests:
pn  bzr-doc  none
ii  bzrtools 2.6.0-2
ii  python-bzrlib.tests  2.6.0+bzr6595-6ubuntu1

-- no debconf information
diff -Nru bzr-2.6.0+bzr6595/debian/changelog bzr-2.6.0+bzr6595/debian/changelog
--- bzr-2.6.0+bzr6595/debian/changelog	2014-12-02 22:47:33.0 +0100
+++ bzr-2.6.0+bzr6595/debian/changelog	2015-05-28 10:04:35.0 +0200
@@ -1,3 +1,10 @@
+bzr (2.6.0+bzr6595-6ubuntu3) UNRELEASED; urgency=medium
+
+   * Disable the failing TestBadStatusServer tests for build and autopkgtest,
+ they became racy.
+
+ -- Vincent Ladeuil v.ladeuil...@free.fr  Thu, 28 May 2015 10:03:49 +0200
+
 bzr (2.6.0+bzr6595-6ubuntu1) vivid; urgency=medium
 
   * Merge with Debian unstable.  Remaining Ubuntu changes:
diff -Nru bzr-2.6.0+bzr6595/debian/rules bzr-2.6.0+bzr6595/debian/rules
--- bzr-2.6.0+bzr6595/debian/rules	2014-04-27 23:43:49.0 +0200
+++ bzr-2.6.0+bzr6595/debian/rules	2015-05-28 10:03:41.0 +0200
@@ -24,7 +24,7 @@
 	BZR_PLUGIN_PATH=-site:-user \
 	BZR_DISABLE_PLUGINS=launchpad \
 	PYTHONPATH=$(wildcard $(CURDIR)/build/lib.*-$(PYVERSION)) \
-	$(CURDIR)/build/scripts-$(PYVERSION)/bzr selftest -v --parallel=fork
+	$(CURDIR)/build/scripts-$(PYVERSION)/bzr selftest -v --parallel=fork -x bzrlib.tests.test_http.TestBadStatusServer.test_http_get -x bzrlib.tests.test_http.TestBadStatusServer.test_http_has
 endif
 
 override_dh_auto_clean:
diff -Nru bzr-2.6.0+bzr6595/debian/tests/testsuite bzr-2.6.0+bzr6595/debian/tests/testsuite
--- bzr-2.6.0+bzr6595/debian/tests/testsuite	2014-04-27 23:43:49.0 +0200
+++ bzr-2.6.0+bzr6595/debian/tests/testsuite	2015-05-28 09:36:32.0 +0200
@@ -1,2 +1,2 @@
 #!/bin/sh
-bzr selftest -v --parallel=fork --no-plugins
+bzr selftest -v --parallel=fork --no-plugins -x bzrlib.tests.test_http.TestBadStatusServer.test_http_get -x bzrlib.tests.test_http.TestBadStatusServer.test_http_has

Bug#814539: [Pkg-bazaar-maint] Bug#814539: bzrtools: Uninstallable with current sid bzr and python-bzrlib (2.7.0-2)

2016-02-12 Thread Vincent Ladeuil

>>>>> Agustin Martin <agmar...@debian.org> writes:

> Package: bzrtools
> Version: 2.6.0-2
> Severity: serious
> Justification: uninstallable in sid

> Dear Maintainer,

> Seems that bzrtools needs upgrading for newer bzr package (2.7.0-2),

Fix pushed at
https://anonscm.debian.org/bzr/pkg-bazaar/bzrtools/unstable/ as revno
761

patch attached.

Tested locally with adt-run .// -U --- lxc -es adt-sid

Please upload ;)

Thanks in advance,

   Vincent

Using parent branch bzr+ssh://bzr.debian.org/bzr/pkg-bazaar/bzrtools/unstable/
=== modified file 'debian/changelog'
--- debian/changelog	2014-06-02 02:45:06 +
+++ debian/changelog	2016-02-12 20:12:51 +
@@ -1,3 +1,10 @@
+bzrtools (2.6.0-3) unstable; urgency=medium
+
+  * Drop checking compatibility via deps, this has failed for wrong
+reasons. Rely on Dep8 tests instead (Closes: #814539).
+
+ -- Vincent Ladeuil <v.ladeuil...@free.fr>  Fri, 12 Feb 2016 20:57:32 +0100
+
 bzrtools (2.6.0-2) unstable; urgency=medium
 
   * debian/tests/control: Add Restrictions: allow-stderr

=== modified file 'debian/control'
--- debian/control	2014-01-13 18:41:04 +
+++ debian/control	2016-02-12 19:54:50 +
@@ -6,8 +6,7 @@
Andrew Starr-Bochicchio <a...@debian.org>
 Build-Depends: debhelper (>> 7.0.50~), python (>= 2.6.6-3)
 Build-Depends-Indep: bzr,
- python-bzrlib (<< 2.7.0),
- python-bzrlib (>= 2.6~),
+ python-bzrlib,
  python-bzrlib.tests,
  python-subunit,
  rsync
@@ -20,8 +19,7 @@
 
 Package: bzrtools
 Architecture: all
-Depends: bzr (<< 2.7.0),
- bzr (>= 2.6~),
+Depends: bzr,
  patch,
  ${misc:Depends},
  ${python:Depends}

HPSS calls: 5 (0 vfs) SmartSSHClientMedium(bzr+ssh://bzr.debian.org/)

Bug#814539: [Pkg-bazaar-maint] Bug#814539: bzrtools: Uninstallable with current sid bzr and python-bzrlib (2.7.0-2)

2016-02-12 Thread Vincent Ladeuil

>>>>> Agustin Martin <agmar...@debian.org> writes:

> Package: bzrtools
> Version: 2.6.0-2
> Severity: serious
> Justification: uninstallable in sid

> Dear Maintainer,

> Seems that bzrtools needs upgrading for newer bzr package (2.7.0-2),

Fix pushed at
https://anonscm.debian.org/bzr/pkg-bazaar/bzrtools/unstable/ as revno
761

patch attached.

Tested locally with adt-run -U, tests pass.

Please upload ;)

Thanks in advance,

   Vincent

Using parent branch bzr+ssh://bzr.debian.org/bzr/pkg-bazaar/bzrtools/unstable/
=== modified file 'debian/changelog'
--- debian/changelog	2014-06-02 02:45:06 +
+++ debian/changelog	2016-02-12 20:12:51 +
@@ -1,3 +1,10 @@
+bzrtools (2.6.0-3) unstable; urgency=medium
+
+  * Drop checking compatibility via deps, this has failed for wrong
+reasons. Rely on Dep8 tests instead (Closes: #814539).
+
+ -- Vincent Ladeuil <v.ladeuil...@free.fr>  Fri, 12 Feb 2016 20:57:32 +0100
+
 bzrtools (2.6.0-2) unstable; urgency=medium
 
   * debian/tests/control: Add Restrictions: allow-stderr

=== modified file 'debian/control'
--- debian/control	2014-01-13 18:41:04 +
+++ debian/control	2016-02-12 19:54:50 +
@@ -6,8 +6,7 @@
Andrew Starr-Bochicchio <a...@debian.org>
 Build-Depends: debhelper (>> 7.0.50~), python (>= 2.6.6-3)
 Build-Depends-Indep: bzr,
- python-bzrlib (<< 2.7.0),
- python-bzrlib (>= 2.6~),
+ python-bzrlib,
  python-bzrlib.tests,
  python-subunit,
  rsync
@@ -20,8 +19,7 @@
 
 Package: bzrtools
 Architecture: all
-Depends: bzr (<< 2.7.0),
- bzr (>= 2.6~),
+Depends: bzr,
  patch,
  ${misc:Depends},
  ${python:Depends}

HPSS calls: 5 (0 vfs) SmartSSHClientMedium(bzr+ssh://bzr.debian.org/)

Bug#837268: Please upload bzr 2.7.0+bzr6619-2

2016-09-11 Thread Vincent Ladeuil

The attached patch fixes:

- the test failure caused by diff not exiting '2' on binary files
  anymore (the test was too eager, no other part of bzr relies on this
  behavior), 

- http basic auth was broken when the base64 encoding for the auth
  header become too long.

The former being encountered while fixing the later.

Tested locally with

autopkgtest .// -U -- lxd autopkgtest/debian/sid/amd64


Pushed to
https://anonscm.debian.org/bzr/pkg-bazaar/bzr/2.7/

Thanks in advance,

   Vincent

=== modified file 'debian/changelog'
--- debian/changelog	2016-07-02 16:18:18 +
+++ debian/changelog	2016-09-11 16:43:13 +
@@ -1,3 +1,14 @@
+bzr (2.7.0+bzr6619-2) unstable; urgency=medium
+
+  * Add 18_diff_binaries: With diffutils > 3.5, diff stop exiting with '2'
+on binary files. LP: #1622039
+Closes: #837268
+
+  * Add 19_fix_long_creds: Fix http Basic auth with credentials longer
+than ~57 characters. LP: #1606203
+
+ -- Vincent Ladeuil <v.ladeuil...@free.fr>  Fri, 09 Sep 2016 16:00:49 +0200
+
 bzr (2.7.0+bzr6619-1) unstable; urgency=medium
 
   * Bump standards version to 3.9.8 (no changes).

=== added file 'debian/patches/18_diff_binaries'
--- debian/patches/18_diff_binaries	1970-01-01 00:00:00 +
+++ debian/patches/18_diff_binaries	2016-09-11 16:39:56 +
@@ -0,0 +1,17 @@
+Description: diff doesn't exit with 2 anymore on binary files.
+Author: Vincent Ladeuil <v.ladeuil...@free.fr>
+
+
+=== modified file 'bzrlib/tests/test_diff.py'
+--- unstable.orig/bzrlib/tests/test_diff.py	2016-02-01 18:06:32 +
 unstable/bzrlib/tests/test_diff.py	2016-09-11 16:32:54 +
+@@ -313,8 +313,6 @@
+ pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE,
+  stdin=subprocess.PIPE)
+ out, err = pipe.communicate()
+-# Diff returns '2' on Binary files.
+-self.assertEqual(2, pipe.returncode)
+ # We should output whatever diff tells us, plus a trailing newline
+ self.assertEqual(out.splitlines(True) + ['\n'], lines)
+ 
+

=== added file 'debian/patches/19_fix_long_creds'
--- debian/patches/19_fix_long_creds	1970-01-01 00:00:00 +
+++ debian/patches/19_fix_long_creds	2016-09-09 14:03:49 +
@@ -0,0 +1,49 @@
+Description: http Basic auth was broken
+ When a long (>57) user/pass combination was used, a spurious '\n' ended up in the header value, crashing httplib.
+.
+Author: Vincent Ladeuil <v.ladeuil...@free.fr>
+
+
+
+=== modified file 'bzrlib/tests/test_http.py'
+--- unstable.orig/bzrlib/tests/test_http.py	2016-02-01 18:06:32 +
 unstable/bzrlib/tests/test_http.py	2016-09-09 12:57:44 +
+@@ -260,6 +260,16 @@
+ self.assertEqual('basic', scheme)
+ self.assertEqual('realm="Thou should not pass"', remainder)
+ 
++def test_build_basic_header_with_long_creds(self):
++handler = _urllib2_wrappers.BasicAuthHandler()
++user = 'user' * 10  # length 40
++password = 'password' * 5  # length 40
++header = handler.build_auth_header(
++dict(user=user, password=password), None)
++# https://bugs.launchpad.net/bzr/+bug/1606203 was caused by incorrectly
++# creating a header value with an embedded '\n'
++self.assertFalse('\n' in header)
++
+ def test_basic_extract_realm(self):
+ scheme, remainder = self.parse_header(
+ 'Basic realm="Thou should not pass"',
+
+=== modified file 'bzrlib/transport/http/_urllib2_wrappers.py'
+--- unstable.orig/bzrlib/transport/http/_urllib2_wrappers.py	2016-01-31 12:55:31 +
 unstable/bzrlib/transport/http/_urllib2_wrappers.py	2016-09-09 12:58:12 +
+@@ -48,6 +48,7 @@
+ # actual code more or less do that, tests should be written to
+ # ensure that.
+ 
++import base64
+ import errno
+ import httplib
+ import os
+@@ -1491,7 +1492,7 @@
+ 
+ def build_auth_header(self, auth, request):
+ raw = '%s:%s' % (auth['user'], auth['password'])
+-auth_header = 'Basic ' + raw.encode('base64').strip()
++auth_header = 'Basic ' + base64.b64encode(raw)
+ return auth_header
+ 
+ def extract_realm(self, header_value):
+

=== modified file 'debian/patches/series'
--- debian/patches/series	2016-07-02 16:15:17 +
+++ debian/patches/series	2016-09-11 16:38:05 +
@@ -5,3 +5,5 @@
 15_autodoc_source_epoch
 16_paramiko_compat
 17_reproducible_makefile
+18_diff_binaries
+19_fix_long_creds

Bug#850790: cloud.debian.org: Distribute Debian cloud images to LXD

2017-01-14 Thread Vincent Ladeuil

> Eirik Schwenke  writes:

> On Tue, 10 Jan 2017 03:19:45 -0500 Jeremy Bicha  wrote:
>> Package: cloud.debian.org
>> 
>> I have set up LXD to test Ubuntu images. For more info, see Stephane's
>> tutorials from last year. [1]
>> 
>> The standard Ubuntu images are the Ubuntu cloud images. I tried
>> testing a Debian image, but it was too basic to be useful without more
>> work. I expected it to have openssh-server and cloud-init installed.

+1

> Hi,

> as a Debian/LXD user I Don *not* expect a default image to listen on the
> network - especially with Lxd that has an easy to use built-in 
console/shell
> access.

As a tester, I prefer an ssh access out of the box that I can
parametrize with cloud-init.

I prefer to install packages over ssh as in several edge cases,
cloud-init failures to install packages are harder to diagnose than the
ones over ssh.

> That way I can for example install and expose only opensmtpd as a public
> facing mail server, and not worry that any other component (eg openssl)
> allows remote login etc.

We can all have our cake here, cloud-init can be configured to /remove/
packages (via a provided command).

Worst case scenario you can uninstall ssh and cloud-init when you
install and configure opensmtpd.

> For certain deployments it might make sense to expose ssh on a "container"
> or single-service/single-purpose image/vm - but not in general, IMNHO.

In my case, it makes complete sense to be able to configure a container
in a fully automated way but that requires cloud-init and ssh.

And for the record, I did attempt several times to install cloud-init
and ssh on the actual lxd images (last attempt was several weeks ago) to
bootstrap and never achieved full automation :-/

Having cloud-init and ssh in lxd images will make my life better :-)

   Vincent

Bug#917558: matrix-synapse-ldap3 installs a python2.7 module (not a py3 one)

2018-12-28 Thread Vincent Ladeuil

Package: matrix-synapse
Version: 0.34.0-2
Severity: normal

Dear Maintainer,

I'm a freedombox user using a matrix-synapse server.
The freedombox setup relies on ldap for user credentials.

Upgrading from 0.33.9-2 to 0.34.0-2 broke my setup as the
ldap_auth_provider module wasn't available anymore (so the server
crashed at startup).

I worked around it by doing:


apt-get install python3-ldap3
apt-get install matrix-synapse-ldap3
cp /usr/lib/python2.7/dist-packages/ldap_auth_provider.py 
/usr/lib/python3/dist-packages/

This sounds like a fallout of 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=897222

Thanks in advance,

   Vincent


-- System Information:
Debian Release: buster/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: armhf (armv7l)

Kernel: Linux 4.9.124-imx6-sr (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), 
LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages matrix-synapse depends on:
ii  adduser3.118
ii  debconf [debconf-2.0]  1.5.69
ii  libjs-jquery   3.2.1-1
ii  lsb-base   10.2018112800
ii  python33.7.1-3
ii  python3-attr   18.2.0-1
ii  python3-bcrypt 3.1.5-1
ii  python3-bleach 2.1.3-1
ii  python3-canonicaljson  1.1.4-1
ii  python3-daemonize  2.4.7-2
ii  python3-frozendict 1.2-1
ii  python3-jinja2 2.10-1
ii  python3-jsonschema 2.6.0-4
ii  python3-msgpack0.5.6-1+b1
ii  python3-nacl   1.3.0-2
ii  python3-netaddr0.7.19-1
ii  python3-openssl18.0.0-1
ii  python3-phonenumbers   8.9.10-1
ii  python3-pil5.3.0-1
ii  python3-prometheus-client  0.3.0-1
ii  python3-psutil 5.4.8-1
ii  python3-psycopg2   2.7.6.1-3
ii  python3-pyasn1 0.4.2-3
ii  python3-pyasn1-modules 0.2.1-0.2
ii  python3-pymacaroons0.13.0-1
ii  python3-pysaml24.5.0-4
ii  python3-service-identity   16.0.0-2
ii  python3-signedjson 1.0.0+git20151019-1
ii  python3-six1.12.0-1
ii  python3-sortedcontainers   2.0.4-1
ii  python3-systemd234-2+b1
ii  python3-treq   18.6.0-0.1
ii  python3-twisted18.9.0-3
ii  python3-unpaddedbase64 1.1.0-3
ii  python3-yaml   3.13-1

Versions of packages matrix-synapse recommends:
ii  python3-lxml  4.2.5-1

Versions of packages matrix-synapse suggests:
ii  python3-bleach  2.1.3-1
ii  python3-jinja2  2.10-1

-- Configuration Files:
/etc/matrix-synapse/homeserver.yaml changed:
tls_certificate_path: "/etc/matrix-synapse/homeserver.tls.crt"
tls_private_key_path: "/etc/matrix-synapse/homeserver.tls.key"
tls_dh_params_path: "/etc/matrix-synapse/homeserver.tls.dh"
no_tls: False
tls_fingerprints: []
pid_file: "/var/run/matrix-synapse.pid"
web_client: False
soft_file_limit: 0
listeners:
  # Main HTTPS listener
  # For when matrix traffic is sent directly to synapse.
  -
# The port to listen for HTTPS requests on.
port: 8448
# Local addresses to listen on.
# On Linux and Mac OS, `::` will listen on all IPv4 and IPv6
# addresses by default. For most other OSes, this will only listen
# on IPv6.
bind_addresses:
  - '::1'
  - '127.0.0.1'
  # - '::'
  # - '0.0.0.0'
# This is a 'http' listener, allows us to specify 'resources'.
type: http
tls: true
# Use the X-Forwarded-For (XFF) header as the client IP and not the
# actual client IP.
x_forwarded: false
# List of HTTP resources to serve on this listener.
resources:
  -
# List of resources to host on this listener.
names:
  - client # The client-server APIs, both v1 and v2
  - webclient  # The bundled webclient.
# Should synapse compress HTTP responses to clients that support it?
# This should be disabled if running synapse behind a load balancer
# that can do automatic compression.
compress: true
  - names: [federation]  # Federation APIs
compress: false
# optional list of additional endpoints which can be loaded via
# dynamic modules
# additional_resources:
#   "/_matrix/my/custom/endpoint":
# module: my_module.CustomRequestHandler
# config: {}
  # Unsecure HTTP listener,
  # For when matrix traffic passes through loadbalancer that unwraps TLS.
  - port: 8008
tls: false
bind_addresses:
  - '::1'
  - '127.0.0.1'
  # - '::'
  # - '0.0.0.0'
type: http
x_forwarded: false
resources:
  - names: [client, webclient]
compress: true
  - names: [federation]
compress: false
  # Turn on the twisted ssh manhole service on localhost on the given
  # port.
  # - port: 9000
  #   bind_addresses:
  # - '::1'
  # - '127.0.0.1'
  #   type: manhole

Bug#677250: Acknowledgement (xz files with multiple blocks are not recognized)

Bug#677250: Acknowledgement (xz files with multiple blocks are not recognized)

Bug#677250: Acknowledgement (xz files with multiple blocks are not recognized)

Bug#677250: Acknowledgement (xz files with multiple blocks are not recognized)

Bug#677250: Acknowledgement (xz files with multiple blocks are not recognized)

Bug#677241: xz files built with -0[e] and/or --check=sha256 not recognized

Bug#677250: xz files with multiple blocks are not recognized

Bug#787057: bzr: Racy tests

Bug#814539: [Pkg-bazaar-maint] Bug#814539: bzrtools: Uninstallable with current sid bzr and python-bzrlib (2.7.0-2)

Bug#814539: [Pkg-bazaar-maint] Bug#814539: bzrtools: Uninstallable with current sid bzr and python-bzrlib (2.7.0-2)

Bug#837268: Please upload bzr 2.7.0+bzr6619-2

Bug#850790: cloud.debian.org: Distribute Debian cloud images to LXD

Bug#917558: matrix-synapse-ldap3 installs a python2.7 module (not a py3 one)

13 matches

Site Navigation

Mail list logo

Footer information