Bug#855169: diffoscope: use BSD-style stat on FreeBSD
Package: diffoscope
Version: 77
>From GNU stat output diffoscope filters those parts that are allowed
to change. The BSD stat(1) command does not match the output format
and reports undesired differences:
% python3 setup.py test --pytest-args="-rsx -l -vv
tests/test_main.py::test_no_differences_directories"
...
> assert ret == 0
E assert 1 == 0
capsys = <_pytest.capture.CaptureFixture object at 0x806b08c18>
create_dir = .create_dir at 0x808576b70>
err= ''
out= '---
/tmp/pytest-of-emaste/pytest-13/test_no_differences_directorie0/a\n+++
/tmp/pytest-of-emaste/pytest-13/test_no_di...14 21:49:59 2017" "Jan 1
00:00:00 1970" 32768 8 0
/tmp/pytest-of-emaste/pytest-13/test_no_differences_directorie0/b\n'
ret= 1
...
BSD stat accepts a -f argument to specify the output format so the
filtering can be avoided altogether.
Attached patch (also available as
https://github.com/emaste/diffoscope/commit/e709c11212a18105f3b6862d5ed20de61038b71a)
is my approach at addressing this.
commit e709c11212a18105f3b6862d5ed20de61038b71a
Author: Ed Maste <ema...@freebsd.org>
Date: Tue Feb 14 16:42:20 2017 -0500
Assume BSD-style stat(1) on FreeBSD
diff --git a/diffoscope/comparators/directory.py
b/diffoscope/comparators/directory.py
index d30cb64..1327912 100644
--- a/diffoscope/comparators/directory.py
+++ b/diffoscope/comparators/directory.py
@@ -45,25 +45,31 @@ def list_files(path):
return all_files
-class Stat(Command):
-@tool_required('stat')
-def cmdline(self):
-return ['stat', self.path]
-
-FILE_RE = re.compile(r'^\s*File:.*$')
-DEVICE_RE = re.compile(r'Device: [0-9a-f]+h/[0-9]+d\s+')
-INODE_RE = re.compile(r'Inode: [0-9]+\s+')
-ACCESS_TIME_RE = re.compile(r'^Access: [0-9]{4}-[0-9]{2}-[0-9]{2}.*$')
-CHANGE_TIME_RE = re.compile(r'^Change: [0-9]{4}-[0-9]{2}-[0-9]{2}.*$')
-
-def filter(self, line):
-line = line.decode('utf-8')
-line = Stat.FILE_RE.sub('', line)
-line = Stat.DEVICE_RE.sub('', line)
-line = Stat.INODE_RE.sub('', line)
-line = Stat.ACCESS_TIME_RE.sub('', line)
-line = Stat.CHANGE_TIME_RE.sub('', line)
-return line.encode('utf-8')
+if os.uname()[0] == 'FreeBSD':
+class Stat(Command):
+@tool_required('stat')
+def cmdline(self):
+return ['stat', '-t', '%Y-%m-%d %H:%M:%S', '-f', '%Sp %l %Su %Sg
%z %Sm %k %b %#Xf', self.path]
+else:
+class Stat(Command):
+@tool_required('stat')
+def cmdline(self):
+return ['stat', self.path]
+
+FILE_RE = re.compile(r'^\s*File:.*$')
+DEVICE_RE = re.compile(r'Device: [0-9a-f]+h/[0-9]+d\s+')
+INODE_RE = re.compile(r'Inode: [0-9]+\s+')
+ACCESS_TIME_RE = re.compile(r'^Access: [0-9]{4}-[0-9]{2}-[0-9]{2}.*$')
+CHANGE_TIME_RE = re.compile(r'^Change: [0-9]{4}-[0-9]{2}-[0-9]{2}.*$')
+
+def filter(self, line):
+line = line.decode('utf-8')
+line = Stat.FILE_RE.sub('', line)
+line = Stat.DEVICE_RE.sub('', line)
+line = Stat.INODE_RE.sub('', line)
+line = Stat.ACCESS_TIME_RE.sub('', line)
+line = Stat.CHANGE_TIME_RE.sub('', line)
+return line.encode('utf-8')
@tool_required('lsattr')
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#827222: Support a timeout on the individual commands invoked by diffoscope
Package: diffoscope Version: 54 Severity: wishlist I am using diffoscope on FreeBSD and often encounter very long run times. I've since found that the same issue is known upstream: a number of Debian reproducibility bugs are tagged with diffoscope_runs_forever[1] and the Debian CI infrastructure kills it after 60 minutes. I think it would be nice to have a configurable runtime limit built in to diffoscope for the individual commands it invokes (e.g. objdump --disassemble), so that other nonreproducible components in the same package (etc.) can still be reported, and so that diffoscope still obtains a report. E.g. it could report "Excessive runtime for command 'objdump --disassemble'. Falling back to binary comparison." [1] https://tests.reproducible-builds.org/debian/issues/unstable/diffoscope_runs_forever_issue.html ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] reproducible builds of FreeBSD in a chroot on Linux
On 16 June 2015 at 17:50, Holger Levsen hol...@layer-acht.org wrote: So in a while, I expect to have set up https://reproducible.debian.net/freebsd/ as well as https://reproducible.debian.net/netbsd/ - but no promises (yet), but these are my plans ;-) Great, looking forward to it! https://wiki.freebsd.org/ReproducibleBuilds claims there are 3 known issues (for make world AIUI) for HEAD, I would like to build twice and verify myself. I'm interested in fixing the remaining kernel / world issues, with the kernel being my higher priority. For the kernel we have the username, hostname, and build timestamp. The path is included too, but I don't anticipate trying to address it at first; release builds are done in a consistent location anyhow (/usr/src). These are used only as user-facing strings for the kern.version sysctl and reported by uname. An example kern.version string: FreeBSD 10.1-STABLE #28 r280427+86df2de(stable-10): Thu Mar 26 16:07:47 EDT 2015 emaste@feynman:/tank/emaste/obj/tank/emaste/src/git-stable-10/sys/GENERIC From a technical perspective they're trivially eliminated. There may be some 3rd party ports expect the precise format, but probably not very many (and they should be fixed, anyhow). There's a much larger social issue in convincing the FreeBSD developer community to accept their removal, though :-) https://wiki.freebsd.org/PortsReproducibleBuilds says Of the 23599 packages which were built in both runs, 15164 have the same checksum when using the previously mentioned patch, giving 64.25% reproducible packages. - I'm also curious to re-confirm this - and set up a test bed, which can be triggered regularily and easily. Our jenkins set up allows this and I'm interested to do this. I'm pleasantly surprised by the ports results -- 64.25% seems quite good for such a straightforward change. The test there is on the same host though, and so avoids any non-reproducibility from host/user/path leaks. My interest is to help FreeBSD with reproducible builds as I want to see reproducible builds become the norm in the free software world and as I believe FreeBSD is an important part of this world. And also because I'm curious. :) Great! Hopefully we can help lend some weight in convincing upstream projects to accept reproducibility patches (once we get further along in our ports effort). -Ed ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
