[please keep me in CC as I'm not subscribed, thank you!] Hi,
I want to create bit-by-bit reproducible sparse tar archives of the same sparse
input file and find myself unable to do so using gnu tar. I create the sparse
test input like this:
$ echo start > sparsefile
$ truncate -s 4M sparsefile
$ echo middle >> sparsefile
$ truncate -s 8388604 sparsefile
$ echo end >> sparsefile
$ stat -c '%b*%B-%s' -- sparsefile
24*512-8388608
$ sha1sum sparsefile
e4d6196039c65beb8ba41b5b45f404549e41303c sparsefile
The output of stat shows that indeed the file I created contains holes as the
number of allocated blocks (24) is much smaller than the size of the file
(8MiB) when multiplied by the block size (512).
I try to pack up the file `sparsefile` in a tarball using this invocation of
gnu tar:
$ SOURCE_DATE_EPOCH=0 tar --mtime=@0
--pax-option='delete=atime,delete=ctime' --numeric-owner --owner=0 --group=0
--format=pax --sparse -cf sparse-archive1.tar sparsefile
I then repeat the process, only changing the output filename:
$ SOURCE_DATE_EPOCH=0 tar --mtime=@0
--pax-option='delete=atime,delete=ctime' --numeric-owner --owner=0 --group=0
--format=pax --sparse -cf sparse-archive2.tar sparsefile
My expectation is, that `sparse-archive1.tar` and `sparse-archive2.tar` are
identical as they contain the same file, I have $SOURCE_DATE_EPOCH set, I
delete atime and ctime from the pax header, I set the mtime to 0 and force the
owner to be root. But despite all of that, `sparse-archive1.tar` and
`sparse-archive2.tar` differ like this when looking at them using xxd:
@@ -59,23 +59,23 @@
000003a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000003b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000003c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000003d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000003e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000003f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000400: 2e2f 474e 5553 7061 7273 6546 696c 652e ./GNUSparseFile.
-00000410: 3138 3636 3433 372f 7370 6172 7365 6669 1866437/sparsefi
+00000410: 3138 3636 3432 342f 7370 6172 7365 6669 1866424/sparsefi
00000420: 6c65 0000 0000 0000 0000 0000 0000 0000 le..............
00000430: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000440: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000450: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000460: 0000 0000 3030 3030 3636 3400 3030 3030 ....0000664.0000
00000470: 3030 3000 3030 3030 3030 3000 3030 3030 000.0000000.0000
00000480: 3030 3331 3030 3000 3030 3030 3030 3030 0031000.00000000
-00000490: 3030 3000 3031 3334 3236 0020 3000 0000 000.013426. 0...
+00000490: 3030 3000 3031 3334 3232 0020 3000 0000 000.013422. 0...
000004a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000004b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000004c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000004d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000004e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000004f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000500: 0075 7374 6172 0030 3000 0000 0000 0000 .ustar.00.......
The checksum differs because the stored file contains the tar pid 1866437 for
`sparse-archive1.tar` and 1866424 for `sparse-archive2.tar`. Is there a way to
convince GNU tar to produce bit-by-bit identical sparse output for sparse input
files?
I read https://www.gnu.org/software/tar/manual/html_node/PAX-0.html to find out
that the sparse file is stored using the pattern %d/GNUSparseFile.%p/%f (where
%p is the PID) so that tar implementations that are not aware of gnu extensions
store the sparse file into separate directories. But why does this feature
require embedding the PID? Is this an attempt to support that a tar
implementation not aware of gnu extensions can extract different tarballs with
the same path of a sparse file into the same location multiple times without
overwriting a previous version?
I question that having this feature weighs higher than being able to produce
bit-by-bit identical output given the same input. I wonder if it would be
possible to either:
1. not include the PID in the output by default or
2. to add a command line flag which to suppress inclusion of the PID
If you do not like option 1, option 2 would allow me to convince tar that I
either:
* know that I will always extract the tarball with an implementation aware of
gnu extensions or
* that I don't care that extracting the same sparse file multiple times with a
tar implementation not aware of gnu extensions into the same output location
will overwrite a previously extracted file
Workarounds I found so far:
1. unshare the pid namespace
-----------------------------
If I run tar prefixed with
$ unshare --map-auto --map-root-user --fork --pid -- tar ...
Then tar will run inside a new pid namespace and will always run with a pid of
1 in that namespace. To create a new pid namespace without being root, I also
need to unshare the user namespace, so I am likely no longer allowed to write
into the current directory as the unshared user. To work around that I can
let tar write the tarball to stdout and then redirect that into a file. Full
commandline:
$ unshare --map-auto --map-root-user --fork --pid -- tar --clamp-mtime
--mtime=@0 --pax-option='delete=atime,delete=ctime' --numeric-owner --owner=0
--group=0 --format=pax --sparse -cf - sparsefile > sparse-archive.tar
2. use PAX format 0.0
----------------------
The initial format does not embed the PID into the tarball, so I get bit-by-bit
reproducible output if I run tar like this:
$ tar --sparse-version 0.1 --pax-option delete=GNU.sparse.map --clamp-mtime
--mtime=@0 --pax-option='delete=atime,delete=ctime' --numeric-owner --owner=0
--group=0 -- format=pax --sparse -cf sparse-archive.tar sparsefile
Using format 0.0 is probably not recommended for reasons explained in the
documentation I linked above.
3. don't use gnu tar
---------------------
bsdtar from libarchive does not include the PID into the output and uses a
static value of 0 instead:
$ bsdtar --clamp-mtime --mtime=@0 --numeric-owner --uid 0 --gid 0
--read-sparse -cf sparse-archive.tar sparsefile
But this is what the bsdtar man page calls "restricted pax format". If I run
bsdtar with --format pax then the resulting tarball becomes unreproducible
again because in contrast to gnu tar, bsdtar does not have a way to remove the
atime and ctime pax options from the header. Even when setting the atime using
the touch utility, I cannot control the ctime.
In summary, by default gnu tar is unable to produce bit-by-bit identical sparse
tarballs from the same sparse input file. I wonder whether the feature: "tar
implementations which do not understand gnu extensions extract sparse files
into differently named directories when extracting a tarball into the same
location multiple times" is worth dropping reproducibility. If you find this
feature useful, I wonder if tar could add a command line flag which would allow
me to decide that I rather prefer bit-by-bit identical output than cater for
tar implementations that do not understand gnu extensions.
To find out more about bit-by-bit reproducible builds, visit
https://reproducible-builds.org/docs/definition/
Thanks!
cheers, josch
signature.asc
Description: signature
