Bug#917558: matrix-synapse-ldap3 installs a python2.7 module (not a py3 one)
Package: matrix-synapse Version: 0.34.0-2 Severity: normal Dear Maintainer, I'm a freedombox user using a matrix-synapse server. The freedombox setup relies on ldap for user credentials. Upgrading from 0.33.9-2 to 0.34.0-2 broke my setup as the ldap_auth_provider module wasn't available anymore (so the server crashed at startup). I worked around it by doing: apt-get install python3-ldap3 apt-get install matrix-synapse-ldap3 cp /usr/lib/python2.7/dist-packages/ldap_auth_provider.py /usr/lib/python3/dist-packages/ This sounds like a fallout of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=897222 Thanks in advance, Vincent -- System Information: Debian Release: buster/sid APT prefers testing APT policy: (500, 'testing') Architecture: armhf (armv7l) Kernel: Linux 4.9.124-imx6-sr (SMP w/4 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages matrix-synapse depends on: ii adduser3.118 ii debconf [debconf-2.0] 1.5.69 ii libjs-jquery 3.2.1-1 ii lsb-base 10.2018112800 ii python33.7.1-3 ii python3-attr 18.2.0-1 ii python3-bcrypt 3.1.5-1 ii python3-bleach 2.1.3-1 ii python3-canonicaljson 1.1.4-1 ii python3-daemonize 2.4.7-2 ii python3-frozendict 1.2-1 ii python3-jinja2 2.10-1 ii python3-jsonschema 2.6.0-4 ii python3-msgpack0.5.6-1+b1 ii python3-nacl 1.3.0-2 ii python3-netaddr0.7.19-1 ii python3-openssl18.0.0-1 ii python3-phonenumbers 8.9.10-1 ii python3-pil5.3.0-1 ii python3-prometheus-client 0.3.0-1 ii python3-psutil 5.4.8-1 ii python3-psycopg2 2.7.6.1-3 ii python3-pyasn1 0.4.2-3 ii python3-pyasn1-modules 0.2.1-0.2 ii python3-pymacaroons0.13.0-1 ii python3-pysaml24.5.0-4 ii python3-service-identity 16.0.0-2 ii python3-signedjson 1.0.0+git20151019-1 ii python3-six1.12.0-1 ii python3-sortedcontainers 2.0.4-1 ii python3-systemd234-2+b1 ii python3-treq 18.6.0-0.1 ii python3-twisted18.9.0-3 ii python3-unpaddedbase64 1.1.0-3 ii python3-yaml 3.13-1 Versions of packages matrix-synapse recommends: ii python3-lxml 4.2.5-1 Versions of packages matrix-synapse suggests: ii python3-bleach 2.1.3-1 ii python3-jinja2 2.10-1 -- Configuration Files: /etc/matrix-synapse/homeserver.yaml changed: tls_certificate_path: "/etc/matrix-synapse/homeserver.tls.crt" tls_private_key_path: "/etc/matrix-synapse/homeserver.tls.key" tls_dh_params_path: "/etc/matrix-synapse/homeserver.tls.dh" no_tls: False tls_fingerprints: [] pid_file: "/var/run/matrix-synapse.pid" web_client: False soft_file_limit: 0 listeners: # Main HTTPS listener # For when matrix traffic is sent directly to synapse. - # The port to listen for HTTPS requests on. port: 8448 # Local addresses to listen on. # On Linux and Mac OS, `::` will listen on all IPv4 and IPv6 # addresses by default. For most other OSes, this will only listen # on IPv6. bind_addresses: - '::1' - '127.0.0.1' # - '::' # - '0.0.0.0' # This is a 'http' listener, allows us to specify 'resources'. type: http tls: true # Use the X-Forwarded-For (XFF) header as the client IP and not the # actual client IP. x_forwarded: false # List of HTTP resources to serve on this listener. resources: - # List of resources to host on this listener. names: - client # The client-server APIs, both v1 and v2 - webclient # The bundled webclient. # Should synapse compress HTTP responses to clients that support it? # This should be disabled if running synapse behind a load balancer # that can do automatic compression. compress: true - names: [federation] # Federation APIs compress: false # optional list of additional endpoints which can be loaded via # dynamic modules # additional_resources: # "/_matrix/my/custom/endpoint": # module: my_module.CustomRequestHandler # config: {} # Unsecure HTTP listener, # For when matrix traffic passes through loadbalancer that unwraps TLS. - port: 8008 tls: false bind_addresses: - '::1' - '127.0.0.1' # - '::' # - '0.0.0.0' type: http x_forwarded: false resources: - names: [client, webclient] compress: true - names: [federation] compress: false # Turn on the twisted ssh manhole service on localhost on the given # port. # - port: 9000 # bind_addresses: # - '::1' # - '127.0.0.1' # type: manhole database
Bug#850790: cloud.debian.org: Distribute Debian cloud images to LXD
> Eirik Schwenke writes: > On Tue, 10 Jan 2017 03:19:45 -0500 Jeremy Bicha wrote: >> Package: cloud.debian.org >> >> I have set up LXD to test Ubuntu images. For more info, see Stephane's >> tutorials from last year. [1] >> >> The standard Ubuntu images are the Ubuntu cloud images. I tried >> testing a Debian image, but it was too basic to be useful without more >> work. I expected it to have openssh-server and cloud-init installed. +1 > Hi, > as a Debian/LXD user I Don *not* expect a default image to listen on the > network - especially with Lxd that has an easy to use built-in console/shell > access. As a tester, I prefer an ssh access out of the box that I can parametrize with cloud-init. I prefer to install packages over ssh as in several edge cases, cloud-init failures to install packages are harder to diagnose than the ones over ssh. > That way I can for example install and expose only opensmtpd as a public > facing mail server, and not worry that any other component (eg openssl) > allows remote login etc. We can all have our cake here, cloud-init can be configured to /remove/ packages (via a provided command). Worst case scenario you can uninstall ssh and cloud-init when you install and configure opensmtpd. > For certain deployments it might make sense to expose ssh on a "container" > or single-service/single-purpose image/vm - but not in general, IMNHO. In my case, it makes complete sense to be able to configure a container in a fully automated way but that requires cloud-init and ssh. And for the record, I did attempt several times to install cloud-init and ssh on the actual lxd images (last attempt was several weeks ago) to bootstrap and never achieved full automation :-/ Having cloud-init and ssh in lxd images will make my life better :-) Vincent
Bug#837268: Please upload bzr 2.7.0+bzr6619-2
The attached patch fixes: - the test failure caused by diff not exiting '2' on binary files anymore (the test was too eager, no other part of bzr relies on this behavior), - http basic auth was broken when the base64 encoding for the auth header become too long. The former being encountered while fixing the later. Tested locally with autopkgtest .// -U -- lxd autopkgtest/debian/sid/amd64 Pushed to https://anonscm.debian.org/bzr/pkg-bazaar/bzr/2.7/ Thanks in advance, Vincent === modified file 'debian/changelog' --- debian/changelog 2016-07-02 16:18:18 + +++ debian/changelog 2016-09-11 16:43:13 + @@ -1,3 +1,14 @@ +bzr (2.7.0+bzr6619-2) unstable; urgency=medium + + * Add 18_diff_binaries: With diffutils > 3.5, diff stop exiting with '2' +on binary files. LP: #1622039 +Closes: #837268 + + * Add 19_fix_long_creds: Fix http Basic auth with credentials longer +than ~57 characters. LP: #1606203 + + -- Vincent Ladeuil Fri, 09 Sep 2016 16:00:49 +0200 + bzr (2.7.0+bzr6619-1) unstable; urgency=medium * Bump standards version to 3.9.8 (no changes). === added file 'debian/patches/18_diff_binaries' --- debian/patches/18_diff_binaries 1970-01-01 00:00:00 + +++ debian/patches/18_diff_binaries 2016-09-11 16:39:56 + @@ -0,0 +1,17 @@ +Description: diff doesn't exit with 2 anymore on binary files. +Author: Vincent Ladeuil + + +=== modified file 'bzrlib/tests/test_diff.py' +--- unstable.orig/bzrlib/tests/test_diff.py 2016-02-01 18:06:32 + unstable/bzrlib/tests/test_diff.py 2016-09-11 16:32:54 + +@@ -313,8 +313,6 @@ + pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE, + stdin=subprocess.PIPE) + out, err = pipe.communicate() +-# Diff returns '2' on Binary files. +-self.assertEqual(2, pipe.returncode) + # We should output whatever diff tells us, plus a trailing newline + self.assertEqual(out.splitlines(True) + ['\n'], lines) + + === added file 'debian/patches/19_fix_long_creds' --- debian/patches/19_fix_long_creds 1970-01-01 00:00:00 + +++ debian/patches/19_fix_long_creds 2016-09-09 14:03:49 + @@ -0,0 +1,49 @@ +Description: http Basic auth was broken + When a long (>57) user/pass combination was used, a spurious '\n' ended up in the header value, crashing httplib. +. +Author: Vincent Ladeuil + + + +=== modified file 'bzrlib/tests/test_http.py' +--- unstable.orig/bzrlib/tests/test_http.py 2016-02-01 18:06:32 + unstable/bzrlib/tests/test_http.py 2016-09-09 12:57:44 + +@@ -260,6 +260,16 @@ + self.assertEqual('basic', scheme) + self.assertEqual('realm="Thou should not pass"', remainder) + ++def test_build_basic_header_with_long_creds(self): ++handler = _urllib2_wrappers.BasicAuthHandler() ++user = 'user' * 10 # length 40 ++password = 'password' * 5 # length 40 ++header = handler.build_auth_header( ++dict(user=user, password=password), None) ++# https://bugs.launchpad.net/bzr/+bug/1606203 was caused by incorrectly ++# creating a header value with an embedded '\n' ++self.assertFalse('\n' in header) ++ + def test_basic_extract_realm(self): + scheme, remainder = self.parse_header( + 'Basic realm="Thou should not pass"', + +=== modified file 'bzrlib/transport/http/_urllib2_wrappers.py' +--- unstable.orig/bzrlib/transport/http/_urllib2_wrappers.py 2016-01-31 12:55:31 + unstable/bzrlib/transport/http/_urllib2_wrappers.py 2016-09-09 12:58:12 + +@@ -48,6 +48,7 @@ + # actual code more or less do that, tests should be written to + # ensure that. + ++import base64 + import errno + import httplib + import os +@@ -1491,7 +1492,7 @@ + + def build_auth_header(self, auth, request): + raw = '%s:%s' % (auth['user'], auth['password']) +-auth_header = 'Basic ' + raw.encode('base64').strip() ++auth_header = 'Basic ' + base64.b64encode(raw) + return auth_header + + def extract_realm(self, header_value): + === modified file 'debian/patches/series' --- debian/patches/series 2016-07-02 16:15:17 + +++ debian/patches/series 2016-09-11 16:38:05 + @@ -5,3 +5,5 @@ 15_autodoc_source_epoch 16_paramiko_compat 17_reproducible_makefile +18_diff_binaries +19_fix_long_creds
Bug#814539: [Pkg-bazaar-maint] Bug#814539: bzrtools: Uninstallable with current sid bzr and python-bzrlib (2.7.0-2)
>>>>> Agustin Martin writes: > Package: bzrtools > Version: 2.6.0-2 > Severity: serious > Justification: uninstallable in sid > Dear Maintainer, > Seems that bzrtools needs upgrading for newer bzr package (2.7.0-2), Fix pushed at https://anonscm.debian.org/bzr/pkg-bazaar/bzrtools/unstable/ as revno 761 patch attached. Tested locally with adt-run .// -U --- lxc -es adt-sid Please upload ;) Thanks in advance, Vincent Using parent branch bzr+ssh://bzr.debian.org/bzr/pkg-bazaar/bzrtools/unstable/ === modified file 'debian/changelog' --- debian/changelog 2014-06-02 02:45:06 + +++ debian/changelog 2016-02-12 20:12:51 + @@ -1,3 +1,10 @@ +bzrtools (2.6.0-3) unstable; urgency=medium + + * Drop checking compatibility via deps, this has failed for wrong +reasons. Rely on Dep8 tests instead (Closes: #814539). + + -- Vincent Ladeuil Fri, 12 Feb 2016 20:57:32 +0100 + bzrtools (2.6.0-2) unstable; urgency=medium * debian/tests/control: Add Restrictions: allow-stderr === modified file 'debian/control' --- debian/control 2014-01-13 18:41:04 + +++ debian/control 2016-02-12 19:54:50 + @@ -6,8 +6,7 @@ Andrew Starr-Bochicchio Build-Depends: debhelper (>> 7.0.50~), python (>= 2.6.6-3) Build-Depends-Indep: bzr, - python-bzrlib (<< 2.7.0), - python-bzrlib (>= 2.6~), + python-bzrlib, python-bzrlib.tests, python-subunit, rsync @@ -20,8 +19,7 @@ Package: bzrtools Architecture: all -Depends: bzr (<< 2.7.0), - bzr (>= 2.6~), +Depends: bzr, patch, ${misc:Depends}, ${python:Depends} HPSS calls: 5 (0 vfs) SmartSSHClientMedium(bzr+ssh://bzr.debian.org/)
Bug#814539: [Pkg-bazaar-maint] Bug#814539: bzrtools: Uninstallable with current sid bzr and python-bzrlib (2.7.0-2)
>>>>> Agustin Martin writes: > Package: bzrtools > Version: 2.6.0-2 > Severity: serious > Justification: uninstallable in sid > Dear Maintainer, > Seems that bzrtools needs upgrading for newer bzr package (2.7.0-2), Fix pushed at https://anonscm.debian.org/bzr/pkg-bazaar/bzrtools/unstable/ as revno 761 patch attached. Tested locally with adt-run -U, tests pass. Please upload ;) Thanks in advance, Vincent Using parent branch bzr+ssh://bzr.debian.org/bzr/pkg-bazaar/bzrtools/unstable/ === modified file 'debian/changelog' --- debian/changelog 2014-06-02 02:45:06 + +++ debian/changelog 2016-02-12 20:12:51 + @@ -1,3 +1,10 @@ +bzrtools (2.6.0-3) unstable; urgency=medium + + * Drop checking compatibility via deps, this has failed for wrong +reasons. Rely on Dep8 tests instead (Closes: #814539). + + -- Vincent Ladeuil Fri, 12 Feb 2016 20:57:32 +0100 + bzrtools (2.6.0-2) unstable; urgency=medium * debian/tests/control: Add Restrictions: allow-stderr === modified file 'debian/control' --- debian/control 2014-01-13 18:41:04 + +++ debian/control 2016-02-12 19:54:50 + @@ -6,8 +6,7 @@ Andrew Starr-Bochicchio Build-Depends: debhelper (>> 7.0.50~), python (>= 2.6.6-3) Build-Depends-Indep: bzr, - python-bzrlib (<< 2.7.0), - python-bzrlib (>= 2.6~), + python-bzrlib, python-bzrlib.tests, python-subunit, rsync @@ -20,8 +19,7 @@ Package: bzrtools Architecture: all -Depends: bzr (<< 2.7.0), - bzr (>= 2.6~), +Depends: bzr, patch, ${misc:Depends}, ${python:Depends} HPSS calls: 5 (0 vfs) SmartSSHClientMedium(bzr+ssh://bzr.debian.org/)
Bug#787057: bzr: Racy tests
Package: bzr Version: 2.6.0+bzr6595-6ubuntu1 Severity: normal Dear Maintainer, bzr FTBFS as two tests became racy. The attached debdiff disable these tests for both build and autopkgtest (a bug has been filed upstream to fix them). -- System Information: Debian Release: jessie/sid APT prefers vivid-updates APT policy: (500, 'vivid-updates'), (500, 'vivid-security'), (500, 'vivid') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 3.19.0-18-generic (SMP w/8 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages bzr depends on: ii python-bzrlib 2.6.0+bzr6595-6ubuntu1 pn python:any Versions of packages bzr recommends: ii python-gpgme 0.3-1 Versions of packages bzr suggests: pn bzr-doc ii bzrtools 2.6.0-2 ii python-bzrlib.tests 2.6.0+bzr6595-6ubuntu1 -- no debconf information diff -Nru bzr-2.6.0+bzr6595/debian/changelog bzr-2.6.0+bzr6595/debian/changelog --- bzr-2.6.0+bzr6595/debian/changelog 2014-12-02 22:47:33.0 +0100 +++ bzr-2.6.0+bzr6595/debian/changelog 2015-05-28 10:04:35.0 +0200 @@ -1,3 +1,10 @@ +bzr (2.6.0+bzr6595-6ubuntu3) UNRELEASED; urgency=medium + + * Disable the failing TestBadStatusServer tests for build and autopkgtest, + they became racy. + + -- Vincent Ladeuil Thu, 28 May 2015 10:03:49 +0200 + bzr (2.6.0+bzr6595-6ubuntu1) vivid; urgency=medium * Merge with Debian unstable. Remaining Ubuntu changes: diff -Nru bzr-2.6.0+bzr6595/debian/rules bzr-2.6.0+bzr6595/debian/rules --- bzr-2.6.0+bzr6595/debian/rules 2014-04-27 23:43:49.0 +0200 +++ bzr-2.6.0+bzr6595/debian/rules 2015-05-28 10:03:41.0 +0200 @@ -24,7 +24,7 @@ BZR_PLUGIN_PATH=-site:-user \ BZR_DISABLE_PLUGINS=launchpad \ PYTHONPATH=$(wildcard $(CURDIR)/build/lib.*-$(PYVERSION)) \ - $(CURDIR)/build/scripts-$(PYVERSION)/bzr selftest -v --parallel=fork + $(CURDIR)/build/scripts-$(PYVERSION)/bzr selftest -v --parallel=fork -x bzrlib.tests.test_http.TestBadStatusServer.test_http_get -x bzrlib.tests.test_http.TestBadStatusServer.test_http_has endif override_dh_auto_clean: diff -Nru bzr-2.6.0+bzr6595/debian/tests/testsuite bzr-2.6.0+bzr6595/debian/tests/testsuite --- bzr-2.6.0+bzr6595/debian/tests/testsuite 2014-04-27 23:43:49.0 +0200 +++ bzr-2.6.0+bzr6595/debian/tests/testsuite 2015-05-28 09:36:32.0 +0200 @@ -1,2 +1,2 @@ #!/bin/sh -bzr selftest -v --parallel=fork --no-plugins +bzr selftest -v --parallel=fork --no-plugins -x bzrlib.tests.test_http.TestBadStatusServer.test_http_get -x bzrlib.tests.test_http.TestBadStatusServer.test_http_has
Bug#677250: Acknowledgement (xz files with multiple blocks are not recognized)
I finally managed to get this fix deployed on http://package-import.ubuntu.com/status/ and wanted to report success :) The ~240 packages affected have been unblocked leading to ~3000 releases imported successfully. Not all of those releases involved the offended xz files but all packages involved such a file at least once in their recent history. This nevertheless represents a significant coverage. As far as *this* bug is concerned though, there is nothing left to fix ;) Thanks for your help, Vincent -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#677250: Acknowledgement (xz files with multiple blocks are not recognized)
>>>>> Joey Hess writes: > Vincent Ladeuil wrote: >> But, the option name is now --block-list instead of --block-split. > Won't use of this option cause the paranoia check to fail when > reproducing the xz file, unless it's whitelisted there? Ouch. Dang, I thought my tests covered that, obviously not :-/ Fixed (and also added --check={crc64,none} while I was there). > (Also, changelog, Done in the attached patch ? > git commit etc would be useful.) Not sure what you mean here, I'm using bzr (based on bzr+ssh://bazaar.launchpad.net/+branch/pristine-tar/) not git :-} https://code.launchpad.net/~vcs-imports/pristine-tar/trunk is an import from git://git.kitenet.net/pristine-tar/ though so there should be a way... In the mean time, I've pushed my branch at bzr+ssh://bazaar.launchpad.net/~vila/pristine-tar/677250-parse-xz or https://code.launchpad.net/~vila/pristine-tar/677250-parse-xz/ for human readers. I'm also quite a noob at packaging but I'm eager to learn... Vincent P.S.: I'll be in vacations for a week starting tonight. modified debian/changelog modified pristine-xz === modified file 'debian/changelog' --- debian/changelog 2012-06-19 13:28:41 + +++ debian/changelog 2012-07-12 06:52:07 + @@ -1,9 +1,8 @@ pristine-tar (1.26) UNRELEASED; urgency=low * pristine-xz: Use xz --robot to extract information from xz files, -avoiding the need to do more expensive guessing. Will also later -allow supporting files needing --block-split (once upstream xz -provides that option; a patch has been submitted). +avoiding the need to do more expensive guessing. Support --block-list +when available to reproduce multi blocks files. Closes: #677250 Thanks, Vincent Ladeuil -- Joey Hess Tue, 19 Jun 2012 09:24:30 -0400 === modified file 'pristine-xz' --- pristine-xz 2012-06-19 13:28:41 + +++ pristine-xz 2012-07-12 07:33:57 + @@ -168,7 +168,7 @@ sub predict_xz_args { my ($xz) = @_; my $presets = undef; - my $block_split = undef; + my $block_list = undef; my $blocks = $xz->{blocks}; if (scalar(@$blocks)) { # There is at least one block. We assume the same compression @@ -199,13 +199,13 @@ } if (scalar(@$blocks) > 1) { # Gather the block uncompressed sizes - $block_split = join(',', map {$_->{uncompressed_size}} + $block_list = join(',', map {$_->{uncompressed_size}} @$blocks); } } - # FIXME: none is missing my %check_kwd_of = - (CRC32 => 'crc32', + (None => 'none', + CRC32 => 'crc32', CRC64 => 'crc64', 'SHA-256' => 'sha256', ); @@ -215,11 +215,8 @@ my $possible_args = []; my $common = ["--check=$check_kwd", "-z"]; - # FIXME: --block-split is not (yet) part of xz-utils upstream - if (0 && defined($block_split)) { - # We put the block list in front of the parameters to make it - # easier to filter it later. - unshift @$common, "--block-split=$block_split"; + if (defined($block_list)) { + unshift @$common, "--block-list=$block_list"; } foreach my $preset (@$presets) { push @$possible_args, [@$common, "-$preset"]; @@ -350,8 +347,11 @@ next if $param=~/^(-[0-9]e?)$/; next if $param eq '-z'; + next if $param eq '--check=none'; next if $param eq '--check=crc32'; + next if $param eq '--check=crc64'; next if $param eq '--check=sha256'; + next if $param=~/^(--block-list=[0-9,]+)$/; die "paranoia check failed on params from delta (@params)"; } @params=split(' ', $delta->{params});
Bug#677250: Acknowledgement (xz files with multiple blocks are not recognized)
> Joey Hess writes: > Ok, I've applied this in git. Thanks. > I'll release it later, so send more patches if needed. Sorry for the delay, I was waiting for xz-utils maintainer to commit the patch I was testing. This has now occurred and the code will be available as part of xz-utils 5.1.2alpha (whenever it's released, hopefully soon according to the xz maintainer). But, the option name is now --block-list instead of --block-split. Therefore, the attached additional patch change the option name and activates it. If the option is not available (because the installed xz is too old), it will fail and fallback to the usual guessing. pristine-xz will ultimately fails but without a newer xz there is nothing we can do anyway. I've tested the attached patch against a newer xz and it works as expected. Vincent P.S.: I think I've got the indent style right this time ;) modified pristine-xz === modified file 'pristine-xz' --- pristine-xz 2012-06-19 13:28:41 + +++ pristine-xz 2012-07-11 08:32:02 + @@ -168,7 +168,7 @@ sub predict_xz_args { my ($xz) = @_; my $presets = undef; - my $block_split = undef; + my $block_list = undef; my $blocks = $xz->{blocks}; if (scalar(@$blocks)) { # There is at least one block. We assume the same compression @@ -199,7 +199,7 @@ } if (scalar(@$blocks) > 1) { # Gather the block uncompressed sizes - $block_split = join(',', map {$_->{uncompressed_size}} + $block_list = join(',', map {$_->{uncompressed_size}} @$blocks); } } @@ -215,11 +215,8 @@ my $possible_args = []; my $common = ["--check=$check_kwd", "-z"]; - # FIXME: --block-split is not (yet) part of xz-utils upstream - if (0 && defined($block_split)) { - # We put the block list in front of the parameters to make it - # easier to filter it later. - unshift @$common, "--block-split=$block_split"; + if (defined($block_list)) { + unshift @$common, "--block-list=$block_list"; } foreach my $preset (@$presets) { push @$possible_args, [@$common, "-$preset"];
Bug#677250: Acknowledgement (xz files with multiple blocks are not recognized)
> Joey Hess writes: >> +if (scalar(@filters) != 1 || $filters[0] !~ /^--lzma2=/) { >> +die "Only LZMA2 is supported" ; >> +die "Unkown dict size: $dict_size\n" if (!defined($presets)) ; >> +my $check_kwd = $check_kwd_of{$check_name} ; >> +die "Unknown xz check: $check_name\n" if (!defined($check_kwd)) ; > Could it just fall back to the old guessing behavior instead of dying? It does, in reproducexz: eval { $possible_args = readxz($orig); } ; # If we get an error we fallback to guessing, otherwise, we should # succeed with one of the proposed combinations if (! $@) { >> +foreach my $program (@supported_xz_programs) { >> +# try to guess the xz arguments that are needed >> foreach my $args (predictxzargs($possible_levels, $program)) { >> -testvariant($orig, $tmpin, $program, @$args) >> -&& return $program, @$args; >> +testvariant($orig, $tmpin, $program, @$args) >> +&& return $program, @$args; > Your editor is replacing tabs with spaces.. I can fix that. modified pristine-xz === modified file 'pristine-xz' --- pristine-xz 2012-06-12 15:45:07 + +++ pristine-xz 2012-06-19 07:47:17 + @@ -101,6 +101,133 @@ print STDERR " pristine-xz [-vdkt] genxz delta file\n"; } +sub assign_fields { + my ($hash, $labels, $fields) = @_ ; + @$hash{@$labels} = @$fields[1..scalar(@$labels)] ; +} + +sub scan_xz_lvv_robot { + my ($filename) = @_ ; + # We need at least version 5.0 to get a proper '-lvv --robot' + # implemented + my $cmd = "xz -lvv --robot $filename" ; + my $ret = open (my $in, "$cmd |") || die "$cmd failed: $!"; + my %xz = (file => {}, stream => {}, blocks => [], + summary => {}, totals => {}) ; + my (%file, %stream, @blocks, %summary, %totals) ; + my @file_labels = qw{nb_streams nb_blocks compressed uncompressed + ratio checks padding_size} ; + my @stream_labels = + qw{stream_num nb_blocks compressed_offset uncompressed_offset + compressed_size uncompressed_size ratio check_name + padding_size}; + my @block_labels = + qw{stream_num block_in_stream block_in_file compressed_offset + uncompressed_offset compressed_size uncompressed_size ratio + check_name check_value header_size size_present_flags + actual_compressed_size uncompress_memory filter_chain} ; + my @summary_labels = qw{uncompressed_memory size_in_blocks} ; + my @totals_labels = + qw{nb_streams nb_blocks compressed_size uncompressed_size ratio + check_names padding_size nb_files uncompressed_memory + size_in_blocks} ; + + while (my $line = <$in>) { + chomp $line ; + my @fields = split(/\t/, $line) ; + if ($fields[0] eq 'name') { + next ; + } + if ($fields[0] eq 'file') { + assign_fields($xz{file}, \@file_labels, \@fields) ; + next ; + } + if ($fields[0] eq 'stream') { + assign_fields($xz{stream}, \@stream_labels, \@fields) ; + next ; + } + if ($fields[0] eq 'block') { + my %block ; + assign_fields(\%block, \@block_labels, \@fields) ; + push @{$xz{blocks}}, \%block ; + next ; + } + if ($fields[0] eq 'summary') { + assign_fields($xz{summary}, \@summary_labels, \@fields); + next ; + } + if ($fields[0] eq 'totals') { + assign_fields($xz{totals}, \@totals_labels, \@fields) ; + next ; + } + } + close $in ; + return \%xz ; +} + +sub predict_xz_args { + my ($xz) = @_ ; + my $presets = undef ; + my $block_split = undef ; + my $blocks = $xz->{blocks} ; + if (scalar(@$blocks)) { + # There is at least one block. We assume the same compression + # level for all blocks + my $block = $blocks->[0] ; + my @filters = split(/,/, $block->{filter_chain}) ; + if (scalar(@filters) != 1 || $filters[0] !~ /^--lzma2=/) { + die "Only LZMA2 is supported" ; + } + # Deduce the presets from the dict size + if ($filters[0] =~ /--lzma2=dict=(.*)/) { + my $dict_size = $1 ; + my %lzma2_presets_from_dict_size_of = +('256KiB' => ['0'], + '1Mib' => ['1'], + '2MiB' => ['2'], + '4MiB' => ['4', '3'], + # Put 6 before 5 as it's the default and is + # more likely to be right + '8MiB' => ['6', '5'], + '16MiB' => ['7'], + '32MiB' => ['8'], + '64MiB' => ['9'], +) ; + $presets = $lzma2_presets_from_dict_size_of{$dict_size}; + die "Unkown dict size: $dict_size\n" +if (!defined($presets)) ; + } + if (scalar(@$blocks) > 1) { + # Gather the block uncompressed sizes + $block_split = join(',', map {$_->{uncompressed_size}} + @$blocks) ; + } + } + # FIXME: none is missing + my %check_kwd_of = + (CRC32 => 'crc32', + CRC64 => 'crc64', + 'SHA-256' => 'sha256', + ) ; + my $check_name = $xz->{stream}->{check_name} ; + my $check_kwd = $check_kwd_of{$check
Bug#677250: Acknowledgement (xz files with multiple blocks are not recognized)
The attached patch does a better job than my first shot: it relies on 'xz -lvv --robot' do get as much information as possible from the file itself. That reduces the guessing to --extreme and one of (3/4) or (5/6) if their associated dict size is recognized. I've disabled the multi-block (--block-split) parameter as xz-utils does not provide it yet (but I've tested a patch from upstream to that effect and will follow up when it becomes available). The attached patch still reduces the number of xz tries and fallback to the previous guessing is something goes wrong during the file reading. I've successfully used it (*with* --block-split) for ~140 multi-block files. modified pristine-xz === modified file 'pristine-xz' --- pristine-xz 2012-06-12 15:45:07 + +++ pristine-xz 2012-06-18 14:35:58 + @@ -101,7 +101,147 @@ print STDERR " pristine-xz [-vdkt] genxz delta file\n"; } +sub assign_fields { +my ($hash, $labels, $fields) = @_ ; +@$hash{@$labels} = @$fields[1..scalar(@$labels)] ; +} + +sub scan_xz_lvv_robot { +my ($filename) = @_ ; +# We need at least version 5.0 to get a proper '-lvv --robot' implemented +my $cmd = "xz -lvv --robot $filename" ; +my $ret = open (my $in, "$cmd |") || die "$cmd failed: $!"; +my %xz = (file => {}, stream => {}, blocks => [], + summary => {}, totals => {}) ; +my (%file, %stream, @blocks, %summary, %totals) ; +my @file_labels = qw{nb_streams nb_blocks compressed uncompressed ratio + checks padding_size} ; +my @stream_labels = +qw{stream_num nb_blocks compressed_offset uncompressed_offset + compressed_size uncompressed_size ratio check_name padding_size} ; +my @block_labels = +qw{stream_num block_in_stream block_in_file compressed_offset + uncompressed_offset compressed_size uncompressed_size ratio + check_name check_value header_size size_present_flags + actual_compressed_size uncompress_memory filter_chain} ; +my @summary_labels = qw{uncompressed_memory size_in_blocks} ; +my @totals_labels = +qw{nb_streams nb_blocks compressed_size uncompressed_size ratio + check_names padding_size nb_files uncompressed_memory + size_in_blocks} ; + +while (my $line = <$in>) { +chomp $line ; +my @fields = split(/\t/, $line) ; +if ($fields[0] eq 'name') { +next ; +} +if ($fields[0] eq 'file') { +assign_fields($xz{file}, \@file_labels, \@fields) ; +next ; +} +if ($fields[0] eq 'stream') { +assign_fields($xz{stream}, \@stream_labels, \@fields) ; +next ; +} +if ($fields[0] eq 'block') { +my %block ; +assign_fields(\%block, \@block_labels, \@fields) ; +push @{$xz{blocks}}, \%block ; +next ; +} +if ($fields[0] eq 'summary') { +assign_fields($xz{summary}, \@summary_labels, \@fields) ; +next ; +} +if ($fields[0] eq 'totals') { +assign_fields($xz{totals}, \@totals_labels, \@fields) ; +next ; +} +} +close $in ; +return \%xz ; +} + +sub predict_xz_args { +my ($xz) = @_ ; +my $presets = undef ; +my $block_split = undef ; +my $blocks = $xz->{blocks} ; +if (scalar(@$blocks)) { +# There is at least one block. We assume the same compression +# level for all blocks +my $block = $blocks->[0] ; +my @filters = split(/,/, $block->{filter_chain}) ; +if (scalar(@filters) != 1 || $filters[0] !~ /^--lzma2=/) { +die "Only LZMA2 is supported" ; +} +# Deduce the presets from the dict size +if ($filters[0] =~ /--lzma2=dict=(.*)/) { +my $dict_size = $1 ; +my %lzma2_presets_from_dict_size_of = +( + '256KiB' => ['0'], + '1Mib' => ['1'], + '2MiB' => ['2'], + '4MiB' => ['4', '3'], + # Put 6 before 5 as it's the default and is more likely to + # be right + '8MiB' => ['6', '5'], + '16MiB' => ['7'], + '32MiB' => ['8'], + '64MiB' => ['9'], +) ; +$presets = $lzma2_presets_from_dict_size_of{$dict_size} ; +die "Unkown dict size: $dict_size\n" if (!defined($presets)) ; +} +if (scalar(@$blocks) > 1) { +# Gather the block uncompressed sizes +$block_split = join(',', map {$_->{uncompressed_size}} @$blocks) ; +} +} +# FIXME: none is missing +my %check_kwd_of = +(CRC32 => 'crc32', + CRC64 => 'crc64', + 'SHA-256' => 'sha256', +) ; +my $check_name = $xz->{st
Bug#677250: xz files with multiple blocks are not recognized
Package: pristine-tar Version: 1.24 examples: Many (> 100) kde 4.8.2 packages including: https://launchpad.net/ubuntu/precise/+source/marble/4:4.8.2-0ubuntu1/+files/marble_4.8.2.orig.tar.xz https://launchpad.net/ubuntu/precise/+source/kdesdk/4:4.8.2-0ubuntu1/+files/kdesdk_4.8.2.orig.tar.xz https://launchpad.net/ubuntu/precise/+source/kde-workspace/4:4.8.2-0ubuntu1/+files/kde-workspace_4.8.2.orig.tar.xz The same multiple blocks files can be found at ftp://ftp.kde.org/pub/kde/stable/4.8.4/src/ (examples includes kdesk, kalzium, oxygen-icons, etc). The following script is a first shot at parsing the .xz files and extracting the relevant parameters (compression level, check and blocks). It may be used to enhance the xz file recognition but doesn't provide a solution to rebuild multiple blocks xz files. #!/usr/bin/perl # Dotted numbers in comments refer to http://tukaani.org/xz/xz-file-format.txt use warnings; use strict; use constant DEBUG => 1 ; use Fcntl qw{:seek} ; use Data::Dumper ; sub checked_read { # $buf should be a reference so we can still use 'read' and its magic # trick my ($in, $buf, $nb_to_read, $msg) = @_ ; my $nb_read = read($in, $$buf, $nb_to_read) ; die $msg if ($nb_read != $nb_to_read) ; } # see 1.2 with slight adaptation since we read from a file handle instead of # using a char buffer sub read_multibyte_integer { my ($in, $size_max) = @_ ; return undef, 0 if ($size_max == 0) ; if ($size_max > 9) { $size_max = 9 ; } my $buf ; my $i = 1 ; checked_read($in, \$buf, 1, "Corrupted xz file\n") ; $buf = unpack('C', $buf) ; warn sprintf("buf: %d / %d", $buf, $buf & 0x7f) if (DEBUG >= 9) ; my $num = $buf & 0x7f ; while ($buf & 0x80) { return (undef, 0) if ($i >= $size_max) ; checked_read($in, \$buf, 1, "Corrupted xz file\n") ; $buf = unpack('C', $buf) ; warn sprintf("buf: %d / %d", $buf, $buf & 0x7f) if (DEBUG >= 9) ; $i++ ; return (undef, 0) if ($buf == 0x00) ; $num |= ($buf & 0x7f) << (($i - 1) * 7) ; } return ($num, $i) ; } sub decode_check_byte { my ($byte) = @_ ; my $check_key ; # We use the xz --check values as values below, the Ids are used as keys. my %known_check_of = ( 0x00 => 'none', 0x01 => 'crc32', 0x04 => 'crc64', 0x0A => 'sha256', ) ; $check_key = $known_check_of{$byte} ; if (!defined($check_key)) { die sprintf("[%02X] is an unknown xz check\n", $byte) ; } return $check_key ; } sub decode_stream_flags { my @flags = @_ ; die "Unknown flags used in first format byte\n" if ($flags[0] != 0) ; my $check = decode_check_byte($flags[1]) ; return [$check] ; } sub parse_stream_header { my ($in) = @_ ; my ($buf) ; checked_read($in, \$buf, 6, "Invalid xz file\n") ; my $magic = unpack('H12', $buf) ; die "Not an xz file\n" if ($magic ne 'fd377a585a00') ; checked_read($in, \$buf, 2, "Corrupted xz file\n") ; my @flags = unpack('CC', $buf) ; my $flags = decode_stream_flags(@flags) ; my $check = $flags->[0] ; checked_read($in, \$buf, 4, "Corrupted xz file\n") ; my $CRC32 = unpack('V', $buf) ; return [$check, $CRC32] ; } my %lzma2_preset_from_dict_size_of = ( 0x0004 => ['0'], # 256 KiB 0x0010 => ['1'], # 1 MiB 0x0020 => ['2'], # 2 MiB 0x0040 => ['3', '4'], # 4 MiB 0x0080 => ['5', '6'], # 8 MiB 0x0100 => ['7'], # 16 MiB 0x0200 => ['8'], # 32 MiB 0x0400 => ['9'], # 64 MiB ) ; sub decode_lzma2_props { my ($props) = @_ ; my $bits = unpack('C', $props) & 0x3f ; # see 5.3.1 my $dict_size ; if ($bits == 40) { $dict_size = 0x ; } else { $dict_size = 2 | ($bits & 1) ; $dict_size <<= $bits / 2 + 11 ; } my $presets = $lzma2_preset_from_dict_size_of{$dict_size} ; die "Unkown dict size: $dict_size\n" if (!defined($presets)) ; return {id => 'lzma2', presets => $presets} ; } my %props_decoder_of = ( 0x21 => \&decode_lzma2_props, ) ; sub decode_filter_props { my ($id, $props) = @_ ; my $decoder = $props_decoder_of{$id} ; die "$id is not a known filter\n" if (!defined($decoder)) ; return &$decoder($props) ; } sub parse_block_header { my ($in, $check) = @_ ; my ($buf) ; checked_read($in, \$buf, 1, "Corrupted xz file\n") ; my $size = unpack('C', $buf) ; $size = ($size + 1) * 4 ; # see 3.1.1 checked_read($in, \$buf, 1, "Corrupted xz file\n") ; my $flags = unpack('C', $buf) ; my $nb_filters = ($flags & 0x03) + 1 ; # 2 bits used to encode 1-4 values my $must_be_zero = $flags & 0x3C ; die "Reversed block flags non-zero\n" if ($must_be_zero != 0) ; my $compressed_size_present = $flags & 0x40 ; die "Not implemented\n" if ($compressed_size_present != 0) ; my $unco
Bug#677241: xz files built with -0[e] and/or --check=sha256 not recognized
Package: pristine-tar Version: 1.24 Relevant .xz files: https://launchpad.net/debian/sid/+source/haskell-ncurses/0.2.1-1/+files/haskell-ncurses_0.2.1.orig.tar.xz https://launchpad.net/ubuntu/quantal/+source/evilvte/0.5.0-1/+files/evilvte_0.5.0.orig.tar.xz https://launchpad.net/ubuntu/quantal/+source/evilvte/0.5.0-1/+files/evilvte_0.5.0-1.debian.tar.xz https://launchpad.net/debian/sid/+source/evilvte/0.5.1-1/+files/evilvte_0.5.1.orig.tar.xz https://launchpad.net/debian/sid/+source/evilvte/0.5.1-1/+files/evilvte_0.5.1-1.debian.tar.xz The following patch fixes the issue: modified pristine-xz === modified file 'pristine-xz' --- pristine-xz 2012-04-01 16:21:10 + +++ pristine-xz 2012-06-12 15:09:13 + @@ -115,7 +115,7 @@ # # So far in the wild only these levels have been seen. # (Note that level 9 can use a lot of memory.) - my $possible_levels = ["6", "9", "6e", "9e"]; + my $possible_levels = ["6", "9", "0", "6e", "9e", "0e"]; return ($possible_levels); } @@ -127,7 +127,8 @@ foreach my $level (@$possible_levels) { push @args, ["-z", "-$level"]; push @args, ["-z", "-$level", "--check=crc32"]; - } + push @args, ["-z", "-$level", "--check=sha256"]; +} return @args; } @@ -190,6 +191,7 @@ next if $param=~/^(-[0-9]e?)$/; next if $param eq '-z'; next if $param eq '--check=crc32'; + next if $param eq '--check=sha256'; die "paranoia check failed on params from delta (@params)"; } @params=split(' ', $delta->{params}); -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org