Bug#900593: diffoscope: GNU tar xattrs result in: libarchive: Ignoring malformed pax extended attribute
forwarded 900593 https://salsa.debian.org/reproducible-builds/diffoscope/issues/5 thanks I've forwarded this upstream here: https://salsa.debian.org/reproducible-builds/diffoscope/issues/5 Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#900593: diffoscope: GNU tar xattrs result in: libarchive: Ignoring malformed pax extended attribute
On Mon, 11 Jun 2018 at 01:29, Chris Lamb wrote:
>
> Chris Lamb wrote:
>
> > Glad I could resolve your use-case. However, I assume there is still a
> > diffoscope bug here?
> >
> > ie. We certainly shouldn't be emitting "Ignoring malformed pax
> > extended attribute" to stderr and — as a possible improvement — we
> > should be emitting that as part of the "proper" output?
>
> Gentle ping on this Tianon? :-)
Apologies! Yes, IMO there's definitely a diffoscope bug here. The
_ideal_ case would be that it shows me useful data about the PaX
headers that don't match (so that I can narrow down why my tarballs
aren't reproducible).
For some slight context, I reach for diffoscope when I have two binary
objects that I expect to be the same and aren't because it's really
good at taking all that binary noise and converting it to something I
can make some sense out of with an eventual fallback to something like
"diff <(hexdump) <(hexdump)" if it can't find any useful signal in the
noise (or doesn't have the necessary tools installed).
In the case of these PaX headers, we get this misleading warning a few
times, then the "file(1)" diff output ("-POSIX tar archive" / "+POSIX
tar archive (GNU)") -- not even a note about how diffoscope couldn't
find any more meaningful differences between the two objects, so I
think this is at least a wishlist bug for diffoscope and possibly a
minor/normal issue depending on how important you think tar files
containing capabilities / extended attributes are. :)
♥,
- Tianon
4096R / B42F 6819 007F 00F8 8E36 4FD4 036A 9C25 BF35 7DD4
Bug#900593: diffoscope: GNU tar xattrs result in: libarchive: Ignoring malformed pax extended attribute
Chris Lamb wrote: > Glad I could resolve your use-case. However, I assume there is still a > diffoscope bug here? > > ie. We certainly shouldn't be emitting "Ignoring malformed pax > extended attribute" to stderr and — as a possible improvement — we > should be emitting that as part of the "proper" output? Gentle ping on this Tianon? :-) Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#900593: diffoscope: GNU tar xattrs result in: libarchive: Ignoring malformed pax extended attribute
Hi Tianon, > > Oh, neat! Hm, can you try: > > > > --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0 > > Hey, that's snazzy! That gives me a reproducible result. :) "It's so simple!" >.> Glad I could resolve your use-case. However, I assume there is still a diffoscope bug here? ie. We certainly shouldn't be emitting "Ignoring malformed pax extended attribute" to stderr and — as a possible improvement — we should be emitting that as part of the "proper" output? Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#900593: diffoscope: GNU tar xattrs result in: libarchive: Ignoring malformed pax extended attribute
On Fri, 1 Jun 2018 at 14:16, Chris Lamb wrote: > Oh, neat! Hm, can you try: > > --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0 Hey, that's snazzy! That gives me a reproducible result. :) Without: $ sudo tar --create --directory wtf --numeric-owner --transform 's,^./,,' --sort name --xattrs . | md5sum 7ce7cc6c2ad85b1f6fbe2e061188c7a0 - $ sudo tar --create --directory wtf --numeric-owner --transform 's,^./,,' --sort name --xattrs . | md5sum 48d6da7b220e95ce913b0ab69e7edcd3 - With: $ sudo tar --create --directory wtf --numeric-owner --transform 's,^./,,' --sort name --xattrs --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0 . | md5sum b45e8c1739b34f844731e94afc2ccaac - $ sudo tar --create --directory wtf --numeric-owner --transform 's,^./,,' --sort name --xattrs --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0 . | md5sum b45e8c1739b34f844731e94afc2ccaac - And after running "sudo setcap cap_net_raw+ep wtf/bin/ping" (to make sure there are some caps that actually need to be represented): $ sudo tar --create --directory wtf --numeric-owner --transform 's,^./,,' --sort name --xattrs --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0 . | md5sum ce0aa6ac1b6fc9df117696cc17cb07b3 - $ sudo tar --create --directory wtf --numeric-owner --transform 's,^./,,' --sort name --xattrs --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0 . | md5sum ce0aa6ac1b6fc9df117696cc17cb07b3 - ♥, - Tianon 4096R / B42F 6819 007F 00F8 8E36 4FD4 036A 9C25 BF35 7DD4
Bug#900593: diffoscope: GNU tar xattrs result in: libarchive: Ignoring malformed pax extended attribute
Hi Tianon, > It gets worse -- apparently any combination of "--xattrs" (with or > without "--xattrs-include" values) makes GNU tar's output completely > unreproducible (running it on the same directory twice will provide a > different result). Oh, neat! Hm, can you try: --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0 Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#900593: diffoscope: GNU tar xattrs result in: libarchive: Ignoring malformed pax extended attribute
On Fri, 1 Jun 2018 12:25:15 -0700 Tianon Gravi wrote: > The context for this bug is in [1]. The basic gist is that I need to > preserve extended attributes to get the proper capabilities on > "/bin/ping" inside a rootfs tarball, so I updated my GNU tar invocation > to include "--xattrs", "--acls", and "--selinux" (just for good > measure). > > [1]: https://github.com/debuerreotype/debuerreotype/pull/38 It gets worse -- apparently any combination of "--xattrs" (with or without "--xattrs-include" values) makes GNU tar's output completely unreproducible (running it on the same directory twice will provide a different result). I've noted this in [2], and provided a full `hexdump` in [3]. [2]: https://github.com/debuerreotype/debuerreotype/pull/38#issuecomment-393992413 [3]: https://gist.github.com/tianon/5bbfe30a4e889dcece42b8d210f8c8f9 (I poked lamby on IRC about where to note this for the reproducible builds project as a whole, and he told me to just append it to this bug for now.) ♥, - Tianon 4096R / B42F 6819 007F 00F8 8E36 4FD4 036A 9C25 BF35 7DD4
Bug#900593: diffoscope: GNU tar xattrs result in: libarchive: Ignoring malformed pax extended attribute
Package: diffoscope Version: 95 Severity: normal First, thanks for diffoscope; it's the best. :) The context for this bug is in [1]. The basic gist is that I need to preserve extended attributes to get the proper capabilities on "/bin/ping" inside a rootfs tarball, so I updated my GNU tar invocation to include "--xattrs", "--acls", and "--selinux" (just for good measure). [1]: https://github.com/debuerreotype/debuerreotype/pull/38 After doing so, I did a quick "diffoscope" of the before/after (which normally is indispensibly helpful in ensuring that my PR does exactly what I expect it to and not a single thing more). I was expecting to see something showing me that I've got some new extended attributes on at least "/bin/ping" (and possibly other files), but instead got the following: | $ diffoscope travis/20170101/amd64/jessie/rootfs.tar.xz output/20170101/amd64/jessie/rootfs.tar.xz | 2018-06-01 17:22:24 W: libarchive: Ignoring malformed pax extended attribute | 2018-06-01 17:22:24 W: libarchive: Ignoring malformed pax extended attribute | 2018-06-01 17:22:24 W: libarchive: Ignoring malformed pax extended attribute | 2018-06-01 17:22:25 W: libarchive: Ignoring malformed pax extended attribute | 2018-06-01 17:22:25 W: libarchive: Ignoring malformed pax extended attribute | 2018-06-01 17:22:26 W: libarchive: Ignoring malformed pax extended attribute | 2018-06-01 17:22:26 W: libarchive: Ignoring malformed pax extended attribute | 2018-06-01 17:22:26 W: libarchive: Ignoring malformed pax extended attribute | 2018-06-01 17:22:26 W: libarchive: Ignoring malformed pax extended attribute | || 100% Time: 0:00:10 | --- travis/20170101/amd64/jessie/rootfs.tar.xz | +++ output/20170101/amd64/jessie/rootfs.tar.xz | ├── rootfs.tar | │ ├── filetype from file(1) | │ │ @@ -1 +1 @@ | │ │ -POSIX tar archive (GNU) | │ │ +POSIX tar archive I've uploaded the two .tar.xz files in question to [2] in case you'd like to reproduce. :) [2]: https://people.debian.org/~tianon/diffoscope-gnu-tar-xattrs/ If there's something obvious I've missed, I apologize! I searched the bugs and the list as much as I could, and couldn't find anything similar (which was surprising -- I would've thought someone else would've run into this long before me). O:> ♥, - Tianon 4096R / B42F 6819 007F 00F8 8E36 4FD4 036A 9C25 BF35 7DD4 -- System Information: Debian Release: buster/sid APT prefers testing APT policy: (500, 'testing') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.15.0-2-amd64 (SMP w/12 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages diffoscope depends on: ii libpython3.6-stdlib3.6.5-9 ii python33.6.5-3 ii python3-distro 1.0.1-2 ii python3-distutils 3.6.5-3 ii python3-libarchive-c 2.1-3.1 ii python3-magic 2:0.4.15-1 ii python3-pkg-resources 39.1.0-1 Versions of packages diffoscope recommends: ii abootimg 0.6-1+b2 ii acl 2.2.52-3+b1 ii apktool 2.3.3-1 ii binutils-multiarch 2.30-20 ii bzip21.0.6-8.1 ii caca-utils 0.99.beta19-2+b3 ii colord 1.3.3-2 ii db-util 5.3.1 ii default-jdk-headless 2:1.10-65 ii device-tree-compiler 1.4.6-1 ii docx2txt 1.4-1 ii e2fsprogs1.44.2-1 ii enjarify 1:1.0.3-4 ii fontforge-extras 0.3-4 ii fp-utils 3.0.4+dfsg-19 ii fp-utils-3.0.4 [fp-utils]3.0.4+dfsg-19 ii genisoimage 9:1.1.11-3+b2 ii gettext 0.19.8.1-6+b1 ii ghc 8.0.2-11 ii ghostscript 9.22~dfsg-2.1 ii giflib-tools 5.1.4-2 ii gnumeric 1.12.39-1 ii gnupg2.2.5-1 ii imagemagick 8:6.9.9.34+dfsg-3+b1 ii imagemagick-6.q16 [imagemagick] 8:6.9.9.34+dfsg-3+b1 ii jsbeautifier 1.6.4-7 ii libarchive-tools 3.2.2-3.1 ii llvm 1:4.0-40 ii mono-utils 4.6.2.7+dfsg-1 ii odt2txt 0.5-1+b2 ii oggvideotools0.9.1-4 ii openssh-client 1:7.7p1-2 ii pdftk2.02-4+b2 ii pgpdump 0.31-0.2 ii poppler-utils0.63.0-2 ii procyon-decompiler 0.5.32-4 ii python3-argcomplete 1.8.1-1 ii python3-binwalk 2.1.1-16 ii python3-debian 0.1.32 ii python3-defusedxml
