Re: [Reproducible-builds] Create tar archive with deterministic file order
Steven Chamberlain: > This isn't really limited to data.tar but potentially any tarball. > Linux packaging ships an equivalent linux-source-3.16.tar.xz built > from debian/rules.real in a probably non-reproducible way: > > cd '$(BUILD_DIR)'; tar -caf 'linux-source-$(UPSTREAMVERSION).tar.xz' > 'linux-source-$(UPSTREAMVERSION)' > > so I could file a bug requesting to patch that too. It's on the latest one I've sent for linux: https://bugs.debian.org/cgi-bin/bugreport.cgi?msg=34;filename=linux_3.16.7-ckt2-1_reproducible2.patch;att=1;bug=769844 > Plenty more packages ship tarballs though: > > https://packages.debian.org/search?searchon=contents&keywords=.tgz > https://packages.debian.org/search?searchon=contents&keywords=.tar.gz > https://packages.debian.org/search?searchon=contents&keywords=.tar.bz2 > https://packages.debian.org/search?searchon=contents&keywords=.tar.xz > > so I wonder if GNU tar itself could make this any easier. Please ask upstream if you feel like it! :) -- Lunar.''`. lu...@debian.org: :Ⓐ : # apt-get install anarchism `. `'` `- signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Create tar archive with deterministic file order
Jérémy Bobbio wrote: > Suggested solution: > > find foo -print0 | LC_ALL=C sort -z | > tar --null -T - --no-recursion -Jcvf foo.tar.xz Seems ideal to me. > > (I thought this problem would be quite common, so I could add it to the > > Wiki FAQ). > > It's there already: > https://wiki.debian.org/ReproducibleBuilds/FileOrderInTarballs Oh, brilliant, thanks! It was a few days ago that I looked through the Wiki pages for this but I see it there now. This isn't really limited to data.tar but potentially any tarball. Linux packaging ships an equivalent linux-source-3.16.tar.xz built from debian/rules.real in a probably non-reproducible way: cd '$(BUILD_DIR)'; tar -caf 'linux-source-$(UPSTREAMVERSION).tar.xz' 'linux-source-$(UPSTREAMVERSION)' so I could file a bug requesting to patch that too. Plenty more packages ship tarballs though: https://packages.debian.org/search?searchon=contents&keywords=.tgz https://packages.debian.org/search?searchon=contents&keywords=.tar.gz https://packages.debian.org/search?searchon=contents&keywords=.tar.bz2 https://packages.debian.org/search?searchon=contents&keywords=.tar.xz so I wonder if GNU tar itself could make this any easier. Regards, -- Steven Chamberlain ste...@pyro.eu.org ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Create tar archive with deterministic file order
Steven Chamberlain: > Finally I ended up with this: > > $ find foo/ -type f -print0 > filelist > $ tar -Jcvf foo.tar.xz -T filelist --null > > Does that seem like the neatest way, or do you have better suggestions? You are missing the actual sorting on this one. ;) Suggested solution: find foo -print0 | LC_ALL=C sort -z | tar --null -T - --no-recursion -Jcvf foo.tar.xz > (I thought this problem would be quite common, so I could add it to the > Wiki FAQ). It's there already: https://wiki.debian.org/ReproducibleBuilds/FileOrderInTarballs -- Lunar.''`. lu...@debian.org: :Ⓐ : # apt-get install anarchism `. `'` `- signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Create tar archive with deterministic file order
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi Steven, On 02/14/2015 09:44 PM, Steven Chamberlain wrote: > What would be a preferred way to build a tar archive with deterministic > file order? > > Firstly let me point out that: > > $ find foo/ -type f -print0 | sort -z | xargs -0 tar -cvf foo.tar > > is risky as the list of filenames could overflow what xargs can supply > to a single invocation of tar. In that case it would invoke tar again > and clobber the previous output. [...] > Finally I ended up with this: > > $ find foo/ -type f -print0 > filelist > $ tar -Jcvf foo.tar.xz -T filelist --null > > Does that seem like the neatest way, or do you have better suggestions? In the wiki [1] we currently have the following solution, which is similar to yours: # find src -print0 | LC_ALL=C sort -z | tar --null -T - --no-recursion -cf archive.tar Setting LC_ALL is required as there could be a different sorting order depending on the locale. Regards, Reiner [1] https://wiki.debian.org/ReproducibleBuilds/FileOrderInTarballs -BEGIN PGP SIGNATURE- iQIcBAEBCgAGBQJU38K/AAoJEMzwSSjbDuqn9JQQAJIiKGCMtkFxP3S0YHSfXACf 6DbWaNaJkQaGECby+lMr/4/JKEgLRqqCrS93/kjlaC1kOUAyBFBYWuarA9EMhseB AQupv86bvzzbL3vKxXCtF5ORZjo/9peHnxXt5bTn5QyyJrJEVbs6VRztjhTHVGSA 2iQm28AqeB/UPmEVt5U3x5QxbZkpNmtARDPSdcaRitxlxT5wb3HSNNMleGO/dLHk IfBpxJmNefx5FHG/FmD7okZFoYhC8cnes3eqr+tWgcONJ6pVBAg59kt6DmScRi6O bfKF2I9diuf/3tmlULUART7KW5kb/h9tUV1l4umaUK7aFifO2wHQU9SKsDNvNtpd kHwQFPy9lzKumTYIMLtILBvE8iDUkTQ/z1lBBVlz+ZwkQxE0dhSTQ6c5eb1u+kCg ljR0D3Z+JCCGB1dVS8moXYQcY9Ijxtd4ib5DQSBz5azh2GvPykA3LGVD+Zl1gGp6 pEyM7opavgWq/xQnk78Uux5PUiKkv4BZ0SBny8xd9226TEeXbtbS2+VE6EE3R7EY cae0pVytS4MQHKEo/iJ3vfAPntG6aBu9fj2uF+QNnEAlh3UO6G+vYZ0JMfqg4/sY a55hysAktptFNSl2+YAVJe6gAAXWARHlV84BWwgCHBaxFa7Nz7h2qT3n4SvIv7eX DiN8o4GcSwP/dLvRRIt1 =tY8p -END PGP SIGNATURE- ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
