Bug#805488: [Patch] Fix (including for a lot of other failing tarballs)

2017-07-12 Thread Tomasz Buchert 
On 10/07/17 11:45, Lennart Sorensen wrote:
> On Fri, Jul 07, 2017 at 10:10:04PM +0200, Tomasz Buchert wrote:
> > Wow, this is nice. Would you mind adding some tests to cover this?
> > Ideally, we would have coverage for all rsyncable "dialects" we
> > support.  Can you commit to collab-maint?
>
> Something like this perhaps:
>
> diff --git a/test/test_roundtrip.sh b/test/test_roundtrip.sh
> index 5465ed4..b82620d 100644
> --- a/test/test_roundtrip.sh
> +++ b/test/test_roundtrip.sh
> @@ -19,6 +19,18 @@ test_bz2() {
>assertWorksWithTarball $SAMPLES/tarballs/foo-1.0.tar.bz2
>  }
>
> +test_gz_135_rsyncable() {
> +  assertWorksWithTarball 
> $SAMPLES/tarballs/libinotify-kqueue-1.3.5rsyncable_20120419.orig.tar.gz
> +}
> +
> +test_gz_14_rsyncable() {
> +  assertWorksWithTarball 
> $SAMPLES/tarballs/libinotify-kqueue-1.4rsyncable_20120419.orig.tar.gz
> +}
> +
> +test_gz_16_rsyncable() {
> +  assertWorksWithTarball 
> $SAMPLES/tarballs/libinotify-kqueue-1.6rsyncable_20120419.orig.tar.gz
> +}
> +
>  test_gz() {
>assertWorksWithTarball $SAMPLES/tarballs/foo-1.0.tar.gz
>  }
>
> I tried with a smaller tar file first, but you need to be at least 4 or
> 8KB depending on gzip version to trigger the rsyncable window size thing
> so I used the other sample which resulted in different files for each
> gzip version.
>
> --
> Len Sorensen

Thanks a lot! I've imported this and prepared a new release [1].
I will upload it soon.

Cheers,
Tomasz

[1] 
https://anonscm.debian.org/cgit/collab-maint/pristine-tar.git/commit/?id=383071b4978c1d548d4b6d22fa19db57d59ca046


signature.asc
Description: PGP signature

Bug#805488: [Patch] Fix (including for a lot of other failing tarballs)

2017-07-10 Thread Lennart Sorensen 
On Fri, Jul 07, 2017 at 10:10:04PM +0200, Tomasz Buchert wrote:
> Wow, this is nice. Would you mind adding some tests to cover this?
> Ideally, we would have coverage for all rsyncable "dialects" we
> support.  Can you commit to collab-maint?

Something like this perhaps:

diff --git a/test/test_roundtrip.sh b/test/test_roundtrip.sh
index 5465ed4..b82620d 100644
--- a/test/test_roundtrip.sh
+++ b/test/test_roundtrip.sh
@@ -19,6 +19,18 @@ test_bz2() {
   assertWorksWithTarball $SAMPLES/tarballs/foo-1.0.tar.bz2
 }
 
+test_gz_135_rsyncable() {
+  assertWorksWithTarball 
$SAMPLES/tarballs/libinotify-kqueue-1.3.5rsyncable_20120419.orig.tar.gz
+}
+
+test_gz_14_rsyncable() {
+  assertWorksWithTarball 
$SAMPLES/tarballs/libinotify-kqueue-1.4rsyncable_20120419.orig.tar.gz
+}
+
+test_gz_16_rsyncable() {
+  assertWorksWithTarball 
$SAMPLES/tarballs/libinotify-kqueue-1.6rsyncable_20120419.orig.tar.gz
+}
+
 test_gz() {
   assertWorksWithTarball $SAMPLES/tarballs/foo-1.0.tar.gz
 }

I tried with a smaller tar file first, but you need to be at least 4 or
8KB depending on gzip version to trigger the rsyncable window size thing
so I used the other sample which resulted in different files for each
gzip version.

-- 
Len Sorensen


libinotify-kqueue-1.3.5rsyncable_20120419.orig.tar.gz
Description: application/gzip


libinotify-kqueue-1.4rsyncable_20120419.orig.tar.gz
Description: application/gzip


libinotify-kqueue-1.6rsyncable_20120419.orig.tar.gz
Description: application/gzip

Bug#805488: [Patch] Fix (including for a lot of other failing tarballs)

2017-07-07 Thread Lennart Sorensen 
On Fri, Jul 07, 2017 at 10:10:04PM +0200, Tomasz Buchert wrote:
> Wow, this is nice. Would you mind adding some tests to cover this?
> Ideally, we would have coverage for all rsyncable "dialects" we
> support.

I can try to add some test coverage.  I will have to check how to do that.

>Can you commit to collab-maint?

I highly doubt it.  I am not a DD or DM (I keep meaning to become one,
but never seem to get around to it).

-- 
Len Sorensen

Bug#805488: [Patch] Fix (including for a lot of other failing tarballs)

2017-07-07 Thread Tomasz Buchert 
On 07/07/17 10:34, Lennart Sorensen wrote:
> I managed to fix almost half the failures in knownproblems by making
> --gnu always be tried rather than only when GZIP_OS_UNIX is found.
> Doing the same for --rsyncable and --new-rsyncable probably makes sense
> too.
>
> The --new-rsyncable was written for gzip 1.4, while gzip 1.6 does things
> a bit different (it's a bit of a hybrid between the original rsyncable
> and the 1.4 rsyncable).  I have added a new --16-rsyncable option to
> zgz that matches the new behaviour, which fixes this bug.
>
> I have tested both gzip 1.4 and gzip 1.6 with --rsyncable and pristine-tar
> now handles both.
>
> --
> Len Sorensen

Wow, this is nice. Would you mind adding some tests to cover this?
Ideally, we would have coverage for all rsyncable "dialects" we
support.  Can you commit to collab-maint?

Thanks!
Tomasz


signature.asc
Description: PGP signature

Bug#805488: [Patch] Fix (including for a lot of other failing tarballs)

2017-07-07 Thread Lennart Sorensen 
I managed to fix almost half the failures in knownproblems by making
--gnu always be tried rather than only when GZIP_OS_UNIX is found.
Doing the same for --rsyncable and --new-rsyncable probably makes sense
too.

The --new-rsyncable was written for gzip 1.4, while gzip 1.6 does things
a bit different (it's a bit of a hybrid between the original rsyncable
and the 1.4 rsyncable).  I have added a new --16-rsyncable option to
zgz that matches the new behaviour, which fixes this bug.

I have tested both gzip 1.4 and gzip 1.6 with --rsyncable and pristine-tar
now handles both.

-- 
Len Sorensen
>From dd0d04fa27d77c1a808e43a313788b63200e54b5 Mon Sep 17 00:00:00 2001
From: Len Sorensen 
Date: Thu, 6 Jul 2017 16:52:41 -0400
Subject: [PATCH] Try --gnu all the time and fix current rsyncable

The rsyncable code was not quite in sync with what gzip --rsyncable
actually does on debian.  Resolving this fixes bug 805488.

Only trying --gnu when GZIP_OS_UNIX is found causes a lot of possible
matches to be missed.  Making --gnu always be attemped no matter what
fixes the following knownproblem cases:

abind_1.1.0.orig.tar.gz
arj_3.10.22.orig.tar.gz
commons-configuration_1.6.orig.tar.gz
dot2tex_2.8.5.orig.tar.gz
evolver_2.30c.orig.tar.gz
faad2_2.6.1.orig.tar.gz
intercal_0.28.orig.tar.gz
libapache-session-perl_1.87.orig.tar.gz
libdatetime-event-sunrise-perl_0.0501.orig.tar.gz
libfreezethaw-perl_0.45.orig.tar.gz
libhtml-linkextractor-perl_0.130.orig.tar.gz
libhtml-prototype-perl_1.48.orig.tar.gz
libhttp-browserdetect-perl_0.98.orig.tar.gz
libinline-perl_0.45.orig.tar.gz
libmail-sendeasy-perl_1.2.orig.tar.gz
libmail-sender-perl_0.8.13.orig.tar.gz
libmail-sender-perl_0.8.16.orig.tar.gz
libmarc-lint-perl_1.43.orig.tar.gz
libmp3-tag-perl_0.9714.orig.tar.gz
libnet-daemon-perl_0.43.orig.tar.gz
libnet-ifconfig-wrapper-perl_0.09.orig.tar.gz
libpdf-api2-perl_0.72.003.orig.tar.gz
libpdf-api2-perl_0.73.orig.tar.gz
libpdf-reuse-perl_0.35.orig.tar.gz
libphp-adodb_5.07.orig.tar.gz
libplrpc-perl_0.2020.orig.tar.gz
libpod-simple-wiki-perl_0.08.orig.tar.gz
libreadonly-perl_1.03.orig.tar.gz
libspreadsheet-writeexcel-perl_2.25.orig.tar.gz
libterm-readline-perl-perl_1.0302.orig.tar.gz
libxml-smart-perl_1.6.9.orig.tar.gz
pyparsing_1.5.1.orig.tar.gz
python-kinterbasdb_3.2.orig.tar.gz
rmetrics_260.72.orig.tar.gz
scalpel_1.60.orig.tar.gz
ttf-umefont_401.orig.tar.gz
ttf-umefont_402.orig.tar.gz
unace_1.2b.orig.tar.gz
unrar-nonfree_3.8.5.orig.tar.gz
---
 pristine-gz| 13 +
 zgz/gzip/deflate.c | 27 ---
 zgz/gzip/gzip.c|  4 ++--
 zgz/gzip/gzip.h|  2 +-
 zgz/zgz.c  | 15 ++-
 5 files changed, 38 insertions(+), 23 deletions(-)

diff --git a/pristine-gz b/pristine-gz
index 2f67160..d110eaf 100755
--- a/pristine-gz
+++ b/pristine-gz
@@ -188,13 +188,10 @@ sub reproducegz {
 
   my @try;
 
-  if ($os == $fconstants{GZIP_OS_UNIX}) {
-# for 98% of the cases the simple heuristic above works
-# and it was produced by gnu gzip.
-push @try, [ '--gnu', @args ];
-push @try, [ '--gnu', @args, '--rsyncable' ];
-push @try, [ '--gnu', @args, '--new-rsyncable' ];
-  }
+  push @try, [ '--gnu', @args ];
+  push @try, [ '--gnu', @args, '--rsyncable' ];
+  push @try, [ '--gnu', @args, '--new-rsyncable' ];
+  push @try, [ '--gnu', @args, '--16-rsyncable' ];
 
   if ($name =~ /\//) {
 push @args, "--original-name", $name;
@@ -301,7 +298,7 @@ sub gengz {
   my @params = split(' ', $delta->{params});
   while (@params) {
 $_ = shift @params;
-next if /^(--gnu|--rsyncable|--new-rsyncable|-[nmM1-9])$/;
+next if /^(--gnu|--rsyncable|--new-rsyncable|--16-rsyncable|-[nmM1-9])$/;
 if (/^(--original-name|--quirk|--osflag)$/) {
   shift @params;
   next;
diff --git a/zgz/gzip/deflate.c b/zgz/gzip/deflate.c
index 780e500..016b757 100644
--- a/zgz/gzip/deflate.c
+++ b/zgz/gzip/deflate.c
@@ -107,11 +107,19 @@ int RSYNC_WIN = 4096;
 /* Whether to enable compatability with Debian's old rsyncable patch. */
 int debian_rsyncable = 1;
 
-void disable_debian_rsyncable () {
+void set_14_rsyncable () {
 	debian_rsyncable = 0;
 	RSYNC_WIN = 8192;
 }
 
+/* gzip 1.6 uses a different rsyncable that is a hybrid between the
+ * original and the gzip 1.4 version
+ */
+void set_16_rsyncable () {
+	debian_rsyncable = 2;
+	RSYNC_WIN = 8192;
+}
+
 #define RSYNC_SUM_MATCH_DEBIAN(sum) ((sum) % RSYNC_WIN == 0)
 #define RSYNC_SUM_MATCH(sum) (((sum) & (RSYNC_WIN - 1)) == 0)
 /* Whether window sum matches magic value */
@@ -532,9 +540,9 @@ static void rsync_roll(unsigned start, unsigned num)
 	rsync_sum += (ulg)window[i];
 	/* Old character out */
 	rsync_sum -= (ulg)window[i - RSYNC_WIN];
-	if (debian_rsyncable && rsync_chunk_end == 0xUL && RSYNC_SUM_MATCH_DEBIAN(rsync_sum))
+	if (debian_rsyncable == 1 && rsync_chunk_end == 0xUL && RSYNC_SUM_MATCH_DEBIAN(rsync_sum))
 	rsync_chunk_end = i;
-	if (! debian_rsyncable && rsync_chunk_end == 0xUL &&