Bug#873615: Fails to detect changes in tarballs that share the same top-level directory

[Carlos Maddela]

On 2/9/19 1:21 am, Axel Beckert wrote:
> 
> So I can't really reproduce this.
> 
> Any idea what I might do different than you? Did you use any
> additional commandline options when you ran into this?

Without the -m option, tardiff only details the changes to the source
tree (i.e. new or removed files and directories), but not changes to
actual source files. So to reproduce the bug in a way that shows it
clearly, you would have to use tarballs that only have changes to source
files and to use the -m option with tardiff. This is what I did:

$ mkdir -p foo-1.0/src
$ echo README > foo-1.0/README
$ echo 1.0 > foo-1.0/VERSION
$ touch foo-1.0/src/main.c
$ tar cvf foo-1.0.tar foo-1.0
foo-1.0/
foo-1.0/README
foo-1.0/VERSION
foo-1.0/src/
foo-1.0/src/main.c
$ cp -a foo-1.0 foo-1.1
$ echo 1.1 > foo-1.1/VERSION
$ tar cvf foo-1.1.tar foo-1.1
foo-1.1/
foo-1.1/README
foo-1.1/VERSION
foo-1.1/src/
foo-1.1/src/main.c
$ tardiff -m foo-1.0.tar foo-1.1.tar
/ VERSION
$ rm *.tar
$ mv foo-1.0 foo
$ tar cvf foo-1.0.tar foo
foo/
foo/README
foo/VERSION
foo/src/
foo/src/main.c
$ rm -rf foo
$ mv foo-1.1 foo
$ tar cvf foo-1.1.tar foo
foo/
foo/README
foo/VERSION
foo/src/
foo/src/main.c
$ tardiff -m foo-1.0.tar foo-1.1.tar
$ exit
exit

In the first case, when the top-level directories are different, it
correctly detects that the VERSION file has changed, whereas when the
top-level directories are the same, it fails to detect the change.
The reason being, the VERSION file from the second tarball will have
clobbered that of the first.

> 
> P.S.: I've hence uploaded a new tardiff package without your patch,
> but once I'm able to reproduce this, I'd include your patch and do
> another upload.

Cheers for that.


Carlos

Bug#873615: Fails to detect changes in tarballs that share the same top-level directory

[Axel Beckert]

Control: tag -1 + unreproducible moreinfo

Hi Carlos,

sorry for the late reply. Seems to have slipped through the cracks.

Carlos Maddela wrote:
> Perhaps I'm not using this utility in the way it was intended to be
> used, but it falsely reports tarballs to be identical if the top-level
> directories are identical.

Hmmm, I just tried to reproduce this:

$ als foo-1.tar
drwxr-xr-x abe/abe   0 2019-09-01 17:07 foo/
drwxr-xr-x abe/abe   0 2019-09-01 17:07 foo/bar/
-rw-r--r-- abe/abe   4 2019-09-01 17:07 foo/bar/foo
$ als foo-2.tar
drwxr-xr-x abe/abe   0 2019-09-01 17:07 foo/
drwxr-xr-x abe/abe   0 2019-09-01 17:07 foo/fnord/
-rw-r--r-- abe/abe   4 2019-09-01 17:07 foo/fnord/bar
$ tardiff foo-1.tar foo-2.tar
- bar
- bar/foo
+ fnord
+ fnord/bar

I also checked if compressed tar balls make a difference:

$ gzip -9vn *.tar
foo-1.tar:   98.7% -- replaced with foo-1.tar.gz
foo-2.tar:   98.7% -- replaced with foo-2.tar.gz
$ tardiff foo-1.tar.gz foo-2.tar.gz
- bar
- bar/foo
+ fnord
+ fnord/bar

I also tried two debian.tar.xz files:

$ tardiff /var/cache/pbuilder/result/zsh_5.{6.2-3,7.1-1}.debian.tar.xz
- compat
- patches/cherry-pick-551ff842-43464-another-attachtty-fix.patch
- patches/cherrypick_decc78c72__svn:_Allow_hyphens_in_command_name_aliases.patch
+ patches/example-shebang.patch
+ pkg-zsh-workflow-new-upstream-release.md
- pkg-zsh-workflow.md
+ pkg-zsh-workflow_outdated.md

So I can't really reproduce this.

Any idea what I might do different than you? Did you use any
additional commandline options when you ran into this?

P.S.: I've hence uploaded a new tardiff package without your patch,
but once I'm able to reproduce this, I'd include your patch and do
another upload.

Regards, Axel
-- 
 ,''`.  |  Axel Beckert , https://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5
  `-|  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE

Bug#873615: Fails to detect changes in tarballs that share the same top-level directory

[Carlos Maddela]

Package: tardiff
Version: 0.1-5
Followup-For: Bug #873615

Amended patch with correct indentation.

-- System Information:
Debian Release: buster/sid
  APT prefers unstable
  APT policy: (990, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.12.0-1-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), 
LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages tardiff depends on:
ii  libtext-diff-perl  1.44-1
ii  perl   5.26.0-5

tardiff recommends no packages.

tardiff suggests no packages.

-- no debconf information
>From b2bf0819142a1391020e43a8182bf0b483a70d99 Mon Sep 17 00:00:00 2001
From: Carlos Maddela 
Date: Tue, 29 Aug 2017 23:24:57 +1000
Subject: [PATCH] Handle tarballs with the same top-level directory.

---
 tardiff | 12 
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/tardiff b/tardiff
index 2577b8f..c23e683 100755
--- a/tardiff
+++ b/tardiff
@@ -67,6 +67,10 @@ sub arguments{
 
 sub untar{
my $tarball = shift(@_);
+   my $prefix = shift(@_);
+
+   my $tardir = "$tempdir/$prefix";
+   mkdir $tardir or die "Couldn't create $tardir";
 
my $flag = "";
if($tarball =~ /\.gz$/){
@@ -75,7 +79,7 @@ sub untar{
$flag = "-j";
}
 
-   open(TARLIST, '-|', qw(tar -C), $tempdir, $flag, qw(-xvf), $tarball)
+   open(TARLIST, '-|', qw(tar -C), $tardir, $flag, qw(-xvf), $tarball)
or die "Can't call tar as expected: $!";
local $/ = undef; # slurp mode
my $list =  or die "Couldn't read from tar";
@@ -174,8 +178,8 @@ sub autofile{
 sub tardiff{
my $error = 0;
 
-   my $filelist1 = untar($tarball1) or die "Error: Could not unpack 
$tarball1.";
-   my $filelist2 = untar($tarball2) or die "Error: Could not unpack 
$tarball2.";
+   my $filelist1 = untar($tarball1, 1) or die "Error: Could not unpack 
$tarball1.";
+   my $filelist2 = untar($tarball2, 2) or die "Error: Could not unpack 
$tarball2.";
 
my %files;
 
@@ -189,7 +193,7 @@ sub tardiff{
next if $opt_autoskip and autofile($file);
my $modified = 0;
if($opt_modified){
-   $modified = comparefile($base1, $base2, $file);
+   $modified = comparefile("1/$base1", "2/$base2", 
$file);
if($modified){
if($opt_stats){
print "/ $file $modified\n";
-- 
2.14.1

Bug#873615: Fails to detect changes in tarballs that share the same top-level directory

[Carlos Maddela]

Package: tardiff
Version: 0.1-5
Severity: normal
Tags: patch

Hi,

Perhaps I'm not using this utility in the way it was intended to be
used, but it falsely reports tarballs to be identical if the top-level
directories are identical. This is because the tarballs get extracted
to the same location, and therefore files are always compared to
themselves. This is the case with Debian tarballs, since the top-level
directory will always be debian/.

I have attached a patch which fixes this.

I also detected a minor fault in its error messages. 'Too much arguments'
should be changed to "Too many arguments'.

Best regards,

Carlos

-- System Information:
Debian Release: buster/sid
  APT prefers unstable
  APT policy: (990, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.12.0-1-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), 
LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages tardiff depends on:
ii  libtext-diff-perl  1.44-1
ii  perl   5.26.0-5

tardiff recommends no packages.

tardiff suggests no packages.

-- no debconf information
>From 7ff1a7d462dbd48e634aba4c1181b187310b667b Mon Sep 17 00:00:00 2001
From: Carlos Maddela 
Date: Tue, 29 Aug 2017 23:24:57 +1000
Subject: [PATCH] Handle tarballs with the same top-level directory.

---
 tardiff | 12 
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/tardiff b/tardiff
index 2577b8f..000e9b1 100755
--- a/tardiff
+++ b/tardiff
@@ -67,6 +67,10 @@ sub arguments{
 
 sub untar{
my $tarball = shift(@_);
+my $prefix = shift(@_);
+
+my $tardir = "$tempdir/$prefix";
+mkdir $tardir or die "Couldn't create $tardir";
 
my $flag = "";
if($tarball =~ /\.gz$/){
@@ -75,7 +79,7 @@ sub untar{
$flag = "-j";
}
 
-   open(TARLIST, '-|', qw(tar -C), $tempdir, $flag, qw(-xvf), $tarball)
+   open(TARLIST, '-|', qw(tar -C), $tardir, $flag, qw(-xvf), $tarball)
or die "Can't call tar as expected: $!";
local $/ = undef; # slurp mode
my $list =  or die "Couldn't read from tar";
@@ -174,8 +178,8 @@ sub autofile{
 sub tardiff{
my $error = 0;
 
-   my $filelist1 = untar($tarball1) or die "Error: Could not unpack 
$tarball1.";
-   my $filelist2 = untar($tarball2) or die "Error: Could not unpack 
$tarball2.";
+   my $filelist1 = untar($tarball1, 1) or die "Error: Could not unpack 
$tarball1.";
+   my $filelist2 = untar($tarball2, 2) or die "Error: Could not unpack 
$tarball2.";
 
my %files;
 
@@ -189,7 +193,7 @@ sub tardiff{
next if $opt_autoskip and autofile($file);
my $modified = 0;
if($opt_modified){
-   $modified = comparefile($base1, $base2, $file);
+   $modified = comparefile("1/$base1", "2/$base2", 
$file);
if($modified){
if($opt_stats){
print "/ $file $modified\n";
-- 
2.14.1