[Reproducible-builds] Bug#808197: diffoscope: readelf --debug-dump can be brutal
Source: diffoscope Version: 43 Severity: wishlist Dear Maintainer, When comparing the debug symbol files for Firefox, which are rather massive, the output for readelf --debug-dump is 1.7GB compressed. Diff can't handle with that much data without OOM'ing. However, if each debug section was compared individually, that give more chance for diffoscope to work. (--debug-dump takes values allowing to treat each debug section independently) -- System Information: Debian Release: stretch/sid APT prefers unstable APT policy: (500, 'unstable'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.2.0-1-amd64 (SMP w/4 CPU cores) Locale: LANG=ja_JP.UTF-8, LC_CTYPE=ja_JP.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#808199: diffoscope: Weird line numbering in diff output
Source: diffoscope
Version: 43
Severity: normal
Dear Maintainer,
* What led up to the situation?
Comparing
https://queue.taskcluster.net/v1/task/FEXzanIjQKW_dBmv3ZmBJQ/runs/0/artifacts/public/build/target.tar.bz2
with
https://queue.taskcluster.net/v1/task/HqZ2DuHYRJ6kR2_P8Rc7Ig/runs/0/artifacts/public/build/target.tar.bz2
* What exactly did you do (or not do) that was effective (or
ineffective)?
`diffoscope --html out.html target.tar.bz2 target.tar.bz2.1`
* What was the outcome of this action?
The HTML output line numbering on the diff is weird. Specifically, it looks
like (simplified):
1 firefox/1 firefox/
2 firefox/libnssdbm3.chk 2 firefox/browser/
3 firefox/libfreebl3.chk 3 firefox/browser/omni.ja
4 firefox/libnssdbm3.so 4 firefox/browser/extensions/
5 firefox/crashreporter 5
firefox/browser/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}.xpi
6 firefox/libnss3.so 6
7 firefox/firefox-bin 7
... ...
Lines 1 to 5 are yellow, 6 and 7 red.
The weird thing about the numbering is that there are numbers on the right hand
side for lines that were removed. Interestingly, there is no number for line
72, which is also a removed line.
The same problem shows up for added lines.
* What outcome did you expect instead?
No numbers for added/removed lines in the part where there were no lines
originally
-- System Information:
Debian Release: stretch/sid
APT prefers unstable
APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 4.2.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=ja_JP.UTF-8, LC_CTYPE=ja_JP.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#808199: diffoscope: Weird line numbering in diff output
Attached here is two files that can be compared directly, exposing the same problem. That's essentially the output of tar --full-time -tvf on both tarballs from the original bug report. The following patch fixes the numbering issue: diff --git a/diffoscope/presenters/html.py b/diffoscope/presenters/html.py index f425889..315a639 100644 --- a/diffoscope/presenters/html.py +++ b/diffoscope/presenters/html.py @@ -346,7 +346,7 @@ def output_line(print_func, s1, s2): print_func(u'' % type_name) try: -if s1 is not None: +if s1: print_func(u'%d ' % line1) print_func(u'') print_func(convert(s1, ponct=1, tag='del')) @@ -355,7 +355,7 @@ def output_line(print_func, s1, s2): s1 = "" print_func(u'') -if s2 is not None: +if s2: print_func(u'%d ' % line2) print_func(u'') print_func(convert(s2, ponct=1, tag='ins')) @@ -369,12 +369,12 @@ def output_line(print_func, s1, s2): m = orig1 and re.match(r"^\[ (\d+) lines removed \]$", orig1) if m: line1 += int(m.group(1)) -elif orig1 is not None: +elif orig1: line1 += 1 m = orig2 and re.match(r"^\[ (\d+) lines removed \]$", orig2) if m: line2 += int(m.group(1)) -elif orig2 is not None: +elif orig2: line2 += 1 However, there's another issue, that the diff is actually incomplete: the right hand side stops 3 lines too early. (The patch is not actually changing anything here, I just figured once the numbers looked normal, but didn't match) Mike drwxr-xr-x 0/0 0 2015-12-16 18:43:24 firefox/ -rw-r--r-- 0/0 899 2015-12-16 18:43:17 firefox/libnssdbm3.chk -rw-r--r-- 0/0 899 2015-12-16 18:43:17 firefox/libfreebl3.chk -rwxr-xr-x 0/0 141782 2015-12-16 18:43:17 firefox/libnssdbm3.so -rwxr-xr-x 0/0 98107 2015-12-16 18:43:17 firefox/crashreporter -rwxr-xr-x 0/0 1152003 2015-12-16 18:43:17 firefox/libnss3.so -rwxr-xr-x 0/0 170163 2015-12-16 18:43:17 firefox/firefox-bin -rw-r--r-- 0/01936 2015-12-16 18:43:24 firefox/precomplete -rw-r--r-- 0/04003 2015-09-11 09:29:56 firefox/crashreporter.ini -rwxr-xr-x 0/0 288136 2015-12-16 18:43:13 firefox/libnspr4.so -rw-r--r-- 0/0 691 2015-12-16 18:17:17 firefox/application.ini -rwxr-xr-x 0/0 178734 2015-12-16 18:43:17 firefox/libnssutil3.so -rwxr-xr-x 0/0 415676 2015-12-16 18:43:17 firefox/plugin-container -rw-r--r-- 0/0 140 2015-12-16 18:37:34 firefox/dependentlibs.list -rw-r--r-- 0/0 132 2015-12-16 18:38:17 firefox/update-settings.ini -rw-r--r-- 0/0 825 2015-09-11 09:29:59 firefox/Throbber-small.gif -rwxr-xr-x 0/08915 2015-09-11 09:29:34 firefox/run-mozilla.sh -rwxr-xr-x 0/0 17250 2015-12-16 18:43:13 firefox/libplds4.so drwxr-xr-x 0/0 0 2015-12-16 18:43:13 firefox/dictionaries/ -rw-r--r-- 0/03074 2015-09-11 09:30:02 firefox/dictionaries/en-US.aff -rw-r--r-- 0/0 599769 2015-12-04 07:02:07 firefox/dictionaries/en-US.dic -rwxr-xr-x 0/0 200987 2015-12-16 18:43:17 firefox/webapprt-stub -rwxr-xr-x 0/099927433 2015-12-16 18:43:16 firefox/libxul.so -rwxr-xr-x 0/0 85802 2015-12-16 18:43:17 firefox/liblgpllibs.so -rwxr-xr-x 0/0 604259 2015-12-16 18:43:17 firefox/libnssckbi.so -rwxr-xr-x 0/0 22824 2015-12-16 18:43:13 firefox/libplc4.so -rw-r--r-- 0/0 9814288 2015-12-16 18:43:09 firefox/omni.ja -rwxr-xr-x 0/0 693226 2015-12-16 18:43:17 firefox/libmozsqlite3.so -rw-r--r-- 0/0 707 2015-12-16 18:43:17 firefox/removed-files -rwxr-xr-x 0/0 141334 2015-12-16 18:43:17 firefox/updater -rw-r--r-- 0/0 899 2015-12-16 18:43:17 firefox/libsoftokn3.chk drwxr-xr-x 0/0 0 2015-12-16 18:43:00 firefox/defaults/ drwxr-xr-x 0/0 0 2015-12-16 18:43:17 firefox/defaults/pref/ -rw-r--r-- 0/0 245 2015-12-16 18:38:37 firefox/defaults/pref/channel-prefs.js -rwxr-xr-x 0/0 504460 2015-12-16 18:43:17 firefox/libfreebl3.so drwxr-xr-x 0/0 0 2015-12-16 18:43:17 firefox/webapprt/ -rw-r--r-- 0/0 75463 2015-12-16 18:43:17 firefox/webapprt/omni.ja -rw-r--r-- 0/0 495 2015-12-16 18:38:30 firefox/webapprt/webapprt.ini -rwxr-xr-x 0/0 13582 2015-12-16 18:43:17 firefox/libmozgtk.so -rw-r--r-- 0/0 681 2015-12-16 18:38:34 firefox/updater.ini -rwxr-xr-x 0/0 170155 2015-12-16 18:43:17 firefox/firefox -rw-r--r-- 0/0 145 2015-12-16 18:38:27 firefox/platform.ini -rwxr-xr-x 0/0 263135 2015-12-16 18:43:17 firefox/libsoftokn3.so -rwxr-xr-x 0/0 257055 2015-12-16 18:43:17 firefox/libssl3.so drwxr-xr-x 0/0 0 2015-12-16 18:43:00 firefox/gmp-clearkey/ drwxr-xr-x 0/0 0 2015-12-16 18:43:17
[Reproducible-builds] Bug#808207: diffoscope: Filter objdump --disassemble output before diffing it
Source: diffoscope
Version: 43
Severity: wishlist
When comparing large ELF binaries, some minor differences can end up hurting
the visibility of more important differences.
Specifically, objdump --disassemble displays symbols+offsets for addresses
it derives from IP-relative addressing, like the following:
9d2be2: 48 8d 05 42 65 24 02lea0x2246542(%rip),%rax#
2c1912b <_fini@@xul45a1+0x1d803>
In the particular case I'm looking at, though, some function ends up pushing
the rest of the .text section, so that the _fini symbol (and many others,
actually) move.
So I end up with a *lot* of differences like:
< 9d2be2: 48 8d 05 42 65 24 02lea0x2246542(%rip),%rax#
2c1912b <_fini@@xul45a1+0x1d803>
---
> 9d2be2: 48 8d 05 42 65 24 02lea0x2246542(%rip),%rax#
> 2c1912b <_fini@@xul45a1+0x1d7e3>
(note: this is a diff I got manually, because it's easier to visualize than a
copy/paste of the HTML output I got from diffoscope)
The code is the same, the address is the same, but the pseudo-symbol doesn't
match and it actually doesn't matter because that actually points to some place
in .rodata, and the .rodata hasn't moved, only _fini and some earlier symbols
have.
In another case, the symbol between angle brackets is an actual symbol (on
non-stripped binaries) but the symbol name is different because GCC decided
to use a different suffix[1]. For example:
< 9d2f35: 48 8d 05 d1 5b 33 02lea0x2335bd1(%rip),%rax#
2d08b0d <__FUNCTION__.10544+0x29d>
---
> 9d2f35: 48 8d 05 d1 5b 33 02lea0x2335bd1(%rip),%rax#
> 2d08b0d <__FUNCTION__.10547+0x29d>
The difference might seem interesting to note, but in fact it's not, because it
will already appear in the `readelf --all` diff:
< 17956: 02d0887021 OBJECT LOCAL DEFAULT 16 __FUNCTION__.10544
---
> 17956: 02d0887021 OBJECT LOCAL DEFAULT 16 __FUNCTION__.10547
Anyways, those symbols between angle brackets are just adding noise that would
be better left out. I'm not sure, though, that there is an option to objdump
that allows to make it not display those symbols (and a quick glance at the
binutils source suggests there isn't). I can only suggest sending the output
of objdump through sed :-/
Something like (awful):
@tool_required('objdump')
@tool_required('sed')
def cmdline(self):
return ['sh', '-c', 'objdump --disassemble --full-contents "%s" | sed
"s/<.*>//"' % self.path]
Mike
1. Example of how this can happen:
$ cat > test.c
[Reproducible-builds] Bug#808003: diffoscope: Comparing directories shouldn't care about file order
Source: diffoscope Version: ada1a1dcdc19217fb611e0a1e57bc3744399aefa Severity: wishlist Dear Maintainer, * What led up to the situation? Comparing two directories * What exactly did you do (or not do) that was effective (or ineffective)? Ran `diffoscope --html output.html a b` where a and b are directories containing more or less the same thing. * What was the outcome of this action? The output contained differences in the output for find due to filesystem-dependent behavior wrt file ordering in readdir(3). * What outcome did you expect instead? The sorted list should be compared instead. -- System Information: Debian Release: stretch/sid APT prefers unstable APT policy: (500, 'unstable'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.2.0-1-amd64 (SMP w/4 CPU cores) Locale: LANG=ja_JP.UTF-8, LC_CTYPE=ja_JP.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#807997: diffoscope: CbfsFile checks make diffoscope extra slow
On Tue, Dec 15, 2015 at 02:43:26PM +0900, Mike Hommey wrote: > Source: diffoscope > Version: ada1a1dcdc19217fb611e0a1e57bc3744399aefa > Severity: important > > Dear Maintainer, > >* What led up to the situation? > > Comparing > http://archive.mozilla.org/pub/firefox/try-builds/m...@glandium.org-93eecb52fcadc562b0f52cb22a8971770446a12e/try-linux64/firefox-45.0a1.en-US.linux-x86_64.tar.bz2 > with > https://queue.taskcluster.net/v1/task/Y1kcxbpQQcOQhIHDp7oRxA/runs/0/artifacts/public/build/target.tar.bz2 > >* What exactly did you do (or not do) that was effective (or > ineffective)? > > Ran `diffoscope --html output.html firefox-45.0a1.en-US.linux-x86_64.tar.bz2 > target.tar.bz2` > >* What was the outcome of this action? > > It took 9 minutes and 15 seconds on my machine. > >* What outcome did you expect instead? > > It taking much less time. In fact, after doing some profiling and seeing that > CbfsFile was a hotspot, applying the following patch made the command take 80 > seconds instead, most of which is spent bunzip2'ing the two archives > sequentially, and objdump --disassemble'ing libxul.so in the two archives in > parallel (so there could be a win with bunzip2'ing the two archives in > parallel, too). > > diff --git a/diffoscope/comparators/__init__.py > b/diffoscope/comparators/__init__.py > index b4615c9..c4cffbb 100644 > --- a/diffoscope/comparators/__init__.py > +++ b/diffoscope/comparators/__init__.py > @@ -129,7 +129,6 @@ FILE_CLASSES = ( > DebDataTarFile, > TextFile, > Bzip2File, > -CbfsFile, > CpioFile, > DebFile, > DexFile, Another way to make it faster here is to put CbfsFile last in this list, although it will still make things slow for large files that contain unknown content. Mike ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#807997: Bug#807997: diffoscope: CbfsFile checks make diffoscope extra slow
On Tue, Dec 15, 2015 at 11:13:54AM +0100, Holger Levsen wrote: > Hi Mike, > > out of curiosity: did you have cbfstool installed? I don't. Mike ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#808121: Bug#808121: diffoscope: HTML output is bloated
On Wed, Dec 16, 2015 at 11:19:02AM +0100, Jérémy Bobbio wrote:
> Close: tag -1 + pending
>
> Mike Hommey:
> > Looking at the HTML in the HTML output, one can see that it is needlessly
> > large.
> >
> > Specifically, there appears to be a lot of e.g.
> > following each other, without even a separation between them. This conflates
> > the amount of memory necessary for browsers to render those pages.
>
> I've commited a fix for this specific issue. The HTML presenter borrowed
> a lot of code from diff2html which was probably not much optimized in
> the first place. I guess the output could be vastly improved, but I'd
> rather focus on other part of the code for now. Patches highly welcome
> in the meantime.
Here's another easy win, attached.
Mike
diff --git a/diffoscope/presenters/html.py b/diffoscope/presenters/html.py
index d843f39..f425889 100644
--- a/diffoscope/presenters/html.py
+++ b/diffoscope/presenters/html.py
@@ -116,8 +116,9 @@ HEADER = """
tr.diffchanged td {
background: #A0
}
-span.diffchanged2 {
- background: #E0C880
+ins, del {
+ background: #E0C880;
+ text-decoration: none
}
span.diffponct {
color: #B08080
@@ -274,15 +275,15 @@ def linediff(s, t):
return ''.join(l1).replace(DIFFOFF + DIFFON, ''),
''.join(l2).replace(DIFFOFF + DIFFON, '')
-def convert(s, ponct=0):
+def convert(s, ponct=0, tag=''):
i = 0
t = StringIO()
for c in s:
# used by diffs
if c == DIFFON:
-t.write('')
+t.write('<%s>' % tag)
elif c == DIFFOFF:
-t.write('')
+t.write('' % tag)
# special highlighted chars
elif c == "\t" and ponct == 1:
@@ -348,7 +349,7 @@ def output_line(print_func, s1, s2):
if s1 is not None:
print_func(u'%d ' % line1)
print_func(u'')
-print_func(convert(s1, ponct=1))
+print_func(convert(s1, ponct=1, tag='del'))
print_func(u'')
else:
s1 = ""
@@ -357,7 +358,7 @@ def output_line(print_func, s1, s2):
if s2 is not None:
print_func(u'%d ' % line2)
print_func(u'')
-print_func(convert(s2, ponct=1))
+print_func(convert(s2, ponct=1, tag='ins'))
print_func(u'')
else:
s2 = ""
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#808121: diffoscope: HTML output is bloated
Source: diffoscope Version: 43 Severity: wishlist Dear Maintainer, Looking at the HTML in the HTML output, one can see that it is needlessly large. Specifically, there appears to be a lot of e.g. following each other, without even a separation between them. This conflates the amount of memory necessary for browsers to render those pages. Mike -- System Information: Debian Release: stretch/sid APT prefers unstable APT policy: (500, 'unstable'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.2.0-1-amd64 (SMP w/4 CPU cores) Locale: LANG=ja_JP.UTF-8, LC_CTYPE=ja_JP.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#808267: diffoscope: Redundant information in ELF comparisons
Source: diffoscope Version: 43 Severity: normal When comparing ELF files, the following commands are used: - readelf --all - readelf --debug-dump - objdump --disassemble --full-contents objdump --disassemble --full-contents is actually redundant in itself. For example, it will dump both an hexdump and a disassembly of the .text section. It's also redundant with the output of readelf --debug-dump because it does an hexdump of the .debug_* sections that readelf --debug-dump does a dwarf dump of. -- System Information: Debian Release: stretch/sid APT prefers unstable APT policy: (500, 'unstable'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.2.0-1-amd64 (SMP w/4 CPU cores) Locale: LANG=ja_JP.UTF-8, LC_CTYPE=ja_JP.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#808267: diffoscope: Redundant information in ELF comparisons
On Fri, Dec 18, 2015 at 10:10:54AM +0900, Mike Hommey wrote: > Source: diffoscope > Version: 43 > Severity: normal > > When comparing ELF files, the following commands are used: > - readelf --all > - readelf --debug-dump > - objdump --disassemble --full-contents > > objdump --disassemble --full-contents is actually redundant in itself. For > example, it will dump both an hexdump and a disassembly of the .text section. > It's also redundant with the output of readelf --debug-dump because it does an > hexdump of the .debug_* sections that readelf --debug-dump does a dwarf dump > of. objdump --disassemble --full-contents also outputs a dump of e.g. .note.gnu.build-id, which is printed out in nicer form in readelf --all. Mike ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#808207: diffoscope: Filter objdump --disassemble output before diffing it
On Mon, Jan 18, 2016 at 07:06:14PM +0100, Jérémy Bobbio wrote:
> Hi Mike!
>
> Mike Hommey:
> > When comparing large ELF binaries, some minor differences can end up hurting
> > the visibility of more important differences.
> >
> > Specifically, objdump --disassemble displays symbols+offsets for addresses
> > it derives from IP-relative addressing, like the following:
> >
> >9d2be2: 48 8d 05 42 65 24 02lea0x2246542(%rip),%rax#
> > 2c1912b <_fini@@xul45a1+0x1d803>
>
> I would be grateful if you could try again using the master branch.
> Dhole made diffoscope compare ELF sections individually and I wonder how
> much it helped with this problem.
I don't think it changed anything.
> If it doesn't, would you be so kind to provide example binaries?
I can't find the differing binary anymore :( But that can be reproduced
with a small testcase:
$ cat < test.c
extern const int qux;
int bar() {
#ifdef FOO
return qux * 2;
#else
return qux;
#endif
}
EOF
$ cat < test2.c
const int qux = 2;
EOF
$ gcc -shared -o /tmp/test.so /tmp/test{,2}.c -fPIC
$ gcc -shared -o /tmp/test2.so /tmp/test{,2}.c -fPIC -DFOO
$ strip test{,2}.so
$ diffoscope --text - /tmp/test*.so
(...)
│ @@ -29,56 +29,57 @@
│ 5a2:48 89 e5mov%rsp,%rbp
│ 5a5:48 c1 fe 03 sar$0x3,%rsi
│ 5a9:48 89 f0mov%rsi,%rax
│ 5ac:48 c1 e8 3f shr$0x3f,%rax
│ 5b0:48 01 c6add%rax,%rsi
│ 5b3:48 d1 fesar%rsi
│ 5b6:74 18 je 5d0 <_init@@Base+0xc8>
│ - 5b8:48 8b 05 01 03 20 00mov0x200301(%rip),%rax#
2008c0 <qux@@Base+0x200254>
│ + 5b8:48 8b 05 01 03 20 00mov0x200301(%rip),%rax#
2008c0 <qux@@Base+0x200250>
│ 5bf:48 85 c0test %rax,%rax
│ 5c2:74 0c je 5d0 <_init@@Base+0xc8>
│ 5c4:5d pop%rbp
│ 5c5:ff e0 jmpq *%rax
│ 5c7:66 0f 1f 84 00 00 00nopw 0x0(%rax,%rax,1)
│ 5ce:00 00
│ 5d0:5d pop%rbp
│ 5d1:c3 retq
│ 5d2:0f 1f 40 00 nopl 0x0(%rax)
│ 5d6:66 2e 0f 1f 84 00 00nopw %cs:0x0(%rax,%rax,1)
│ 5dd:00 00 00
│ 5e0:80 3d 09 03 20 00 00cmpb $0x0,0x200309(%rip)#
2008f0 <_edata@@Base>
│ 5e7:75 27 jne610 <_init@@Base+0x108>
│ - 5e9:48 83 3d d7 02 20 00cmpq $0x0,0x2002d7(%rip)#
2008c8 <qux@@Base+0x20025c>
│ + 5e9:48 83 3d d7 02 20 00cmpq $0x0,0x2002d7(%rip)#
2008c8 <qux@@Base+0x200258>
│ 5f0:00
│ 5f1:55 push %rbp
│ 5f2:48 89 e5mov%rsp,%rbp
│ 5f5:74 0c je 603 <_init@@Base+0xfb>
│ - 5f7:48 8b 3d ea 02 20 00mov0x2002ea(%rip),%rdi#
2008e8 <qux@@Base+0x20027c>
│ + 5f7:48 8b 3d ea 02 20 00mov0x2002ea(%rip),%rdi#
2008e8 <qux@@Base+0x200278>
│ 5fe:e8 45 ff ff ff callq 548 <_init@@Base+0x40>
│ 603:e8 48 ff ff ff callq 550 <_init@@Base+0x48>
│ 608:5d pop%rbp
│ 609:c6 05 e0 02 20 00 01movb $0x1,0x2002e0(%rip)#
2008f0 <_edata@@Base>
│ 610:f3 c3 repz retq
│ 612:0f 1f 40 00 nopl 0x0(%rax)
│ 616:66 2e 0f 1f 84 00 00nopw %cs:0x0(%rax,%rax,1)
│ 61d:00 00 00
│ - 620:48 8d 3d e1 00 20 00lea0x2000e1(%rip),%rdi#
200708 <qux@@Base+0x20009c>
│ + 620:48 8d 3d e1 00 20 00lea0x2000e1(%rip),%rdi#
200708 <qux@@Base+0x200098>
│ 627:48 83 3f 00 cmpq $0x0,(%rdi)
│ 62b:75 0b jne638 <_init@@Base+0x130>
│ 62d:e9 5e ff ff ff jmpq 590 <_init@@Base+0x88>
│ 632:66 0f 1f 44 00 00 nopw 0x0(%rax,%rax,1)
│ - 638:48 8b 05 79 02 20 00mov0x200279(%rip),%rax#
2008b8 <qux@@Base+0x20024c>
│ + 638:48 8b 05 79 02 20 00mov0x200279(%rip),%rax#
2008b8 <qux@@Base+0x200248>
│ 63f:48 85 c0test %rax,%rax
│ 642:74 e9 je 62d <_init@@Base+0x125>
│ 644:55 push %rbp
│ 645:48 89 e5mov%rsp,%rbp
│ 648:ff d0 callq *%rax
│ 64a:5d pop%rbp
│ 64b:e9 40 ff ff ff jmpq 590 <_init@@Base
[Reproducible-builds] Bug#808267: diffoscope: Redundant information in ELF comparisons
On Mon, Jan 18, 2016 at 07:20:38PM +0100, Jérémy Bobbio wrote:
> Hi Mike,
>
> Mike Hommey:
> > When comparing ELF files, the following commands are used:
> > - readelf --all
> > - readelf --debug-dump
> > - objdump --disassemble --full-contents
> >
> > objdump --disassemble --full-contents is actually redundant in itself. For
> > example, it will dump both an hexdump and a disassembly of the .text
> > section.
> > It's also redundant with the output of readelf --debug-dump because it does
> > an
> > hexdump of the .debug_* sections that readelf --debug-dump does a dwarf dump
> > of.
>
> The master branch now compare ELF files section by section. If you could
> test it and see if there's still redundancies, I would be grateful.
It seems better, but there's still redundancy at least between `readelf
--wide --all` and `readelf --wide --hex-dump
{.symtab,.note.gnu-build-id,.rela.dyn,.rela.plt,.dynamic,.dynsym}` and other
sections ; just check the list of sections in the output of readelf
--all.
Reading the code suggests debug sections would also be redunded between
--debug-dump and --hex-dump for the individual sections, but I haven't
tested with binaries with debuginfo.
Mike
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#886736: diffoscope: mach-o disassembly with otool can fail in a way that fools diffoscope into dumping raw data instead
On Wed, Jan 10, 2018 at 10:23:59AM +0900, Mike Hommey wrote: > On Tue, Jan 09, 2018 at 01:07:58PM +, Chris Lamb wrote: > > Hi Mike, > > > > > I only have a very large XUL library... you probably don't want that. > > > > Probably not for the testsuite (!) but if you could make it available it > > would help with a fix anyway... > > The two builds I was comparing: > https://queue.taskcluster.net/v1/task/AKmfRsZPQjqkc7ZoBmS2zw/runs/0/artifacts/public/build/target.dmg > https://queue.taskcluster.net/v1/task/Hs2lbc9dTFi4eXjwwM26NA/runs/0/artifacts/public/build/target.dmg > > The corresponding diff: > https://queue.taskcluster.net/v1/task/QRazQl4PQZeuiU81C9PtlA/runs/0/artifacts/public/diff.html > > Note that the actual diff you get is: > https://queue.taskcluster.net/v1/task/Uhv3G5XUQA6DDFrOhErA9Q/runs/0/artifacts/public/diff.html > > which has more differences. Stripping the binaries first gets the first > diff. I should probably file another bug about that, those differences > show up when comparing nm -a output. BTW, since I'm also looking at where those differences are coming from, the first one in the first diff is a UUID difference that appears in the otool -l output. Mike ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#886736: diffoscope: mach-o disassembly with otool can fail in a way that fools diffoscope into dumping raw data instead
On Tue, Jan 09, 2018 at 01:07:58PM +, Chris Lamb wrote: > Hi Mike, > > > I only have a very large XUL library... you probably don't want that. > > Probably not for the testsuite (!) but if you could make it available it > would help with a fix anyway... The two builds I was comparing: https://queue.taskcluster.net/v1/task/AKmfRsZPQjqkc7ZoBmS2zw/runs/0/artifacts/public/build/target.dmg https://queue.taskcluster.net/v1/task/Hs2lbc9dTFi4eXjwwM26NA/runs/0/artifacts/public/build/target.dmg The corresponding diff: https://queue.taskcluster.net/v1/task/QRazQl4PQZeuiU81C9PtlA/runs/0/artifacts/public/diff.html Note that the actual diff you get is: https://queue.taskcluster.net/v1/task/Uhv3G5XUQA6DDFrOhErA9Q/runs/0/artifacts/public/diff.html which has more differences. Stripping the binaries first gets the first diff. I should probably file another bug about that, those differences show up when comparing nm -a output. Mike ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#886736: diffoscope: mach-o disassembly with otool can fail in a way that fools diffoscope into dumping raw data instead
On Tue, Jan 09, 2018 at 12:50:05PM +, Chris Lamb wrote: > Hi Mike, > > > > In some cases, otool can fail with: > > > > can't create x86_64 llvm disassembler > > Do you happen to have example files you could point to or upload? Not > essential of course, but would be much easier and reliable to write a > fix with an addition to the testsuite armed with them... I only have a very large XUL library... you probably don't want that. Mike ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#886736: diffoscope: mach-o disassembly with otool can fail in a way that fools diffoscope into dumping raw data instead
Package: diffoscope Version: 90 Severity: normal Dear Maintainer, In some cases, otool can fail with: can't create x86_64 llvm disassembler (where x86_64 may be another platform, and where the message is usually preceded by the file path name and "(__TEXT,__text) section", well, in fact, anything that would normally come before the disassembly for the given command line) "Sensibly", it does print that on stdout, and quits with exit code 0. Which means when you're comparing two binaries that have assembly differences, the otool output is identical and non-failing, from diffoscope's perspective, meaning it goes on to the fallback "No file format specific differences found inside, yet data differs", which then goes on to do a diff on a hexdump. When the llvm disassembler fails for some reason, one can use the -Q option to otool to make it use its internal disassembler, which is better than nothing. Mike ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#886736: diffoscope: mach-o disassembly with otool can fail in a way that fools diffoscope into dumping raw data instead
On Wed, Jan 10, 2018 at 10:36:02AM +0530, Chris Lamb wrote: > Mike Hommey wrote: > > > The two builds I was comparing: > > […] > > Thanks for sending this over. For some reason, I completely failed to > realise that I would need access to otool to make use of these, and > this is not in Debian ;) > > Can you run some commands for me? In particular, is it sufficient to > fallback to, say, -tdvVQ if -tdvV does not work? It's enough for me. Although some versions of otool don't support -Q, and I don't know what they do. Note you can get otool and other mac tools for linux from https://github.com/tpoechtrager/cctools-port/ Mike ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
